system

Abstract

We provide the system. [Solution] An imaging means for capturing images of animal movements, An analysis means for analyzing video data obtained by the imaging means and determining the emotional state of the animal, A monitoring device for monitoring the vital signs of animals, An evaluation method for evaluating the health status of an animal based on the vital signs, A feedback means for providing the results of the analysis means and the evaluation means to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventionally, it has been difficult for animal owners to accurately understand the emotions and health conditions of their pets, resulting in problems such as inability to provide appropriate care and responses. In particular, there has been a lack of means to detect early the poor physical condition or stress state of pets, which has been a factor damaging the health of animals. Also, due to the inability to accurately grasp the emotions of animals, communication with pets has not been smooth, and there has been a problem that owners cannot appropriately respond to the requests and emotions of their pets.

Means for Solving the Problems

[0005] This invention provides an analysis means for determining an animal's emotional state by analyzing video data obtained from an imaging means that captures images of the animal's movements. Furthermore, it evaluates the animal's health state using a monitoring means that monitors the animal's vital signs. By providing feedback of these results to the user and a display means that visually presents the animal's emotional state, pet owners can grasp their pet's emotions and health state in real time and take appropriate action. In addition, by issuing a rapid warning when an abnormality in the health state is detected, early intervention can be encouraged and the animal's health can be protected.

[0006] The term "animal" refers to all living beings that possess emotions and will, including pets and livestock, as they are not humans.

[0007] "Imaging means" refers to a device or system for capturing images of an object and acquiring that data.

[0008] "Analysis means" refers to devices or systems that perform a process of analyzing information based on input data and deriving specific conclusions or results.

[0009] "Vital signs" refer to biological information of an animal, and usually include important indicators that show its life status, such as body temperature and heart rate.

[0010] "Monitoring means" refers to devices or systems used to continuously observe specific conditions or parameters and detect changes therein.

[0011] "Evaluation means" refers to devices or systems used to make decisions based on collected data and information.

[0012] "Feedback means" refers to devices or systems used to communicate the analysis results and evaluations of a system to the user.

[0013] "Display means" refers to devices or systems used to visually present information such as analysis results to the user.

[0014] A "warning device" refers to a device or system that notifies the user of information when it detects an abnormality or a situation requiring attention. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention is a system that monitors the emotional and health status of animals in real time and provides visual or audible feedback to the user. The system mainly consists of three components: a terminal (goggles and a sensor-equipped pendant), a server, and the user.

[0037] First, the goggles used as terminals have a built-in camera to capture images of the animals' movements. While observing the animals, the goggles capture their facial expressions and movements, recording them as video data. In addition, a sensor-equipped pendant is attached to the animals' bodies to acquire vital sign data such as heart rate and body temperature. This data is transmitted to a server via the goggles.

[0038] The server analyzes the received video data using an AI model to determine the animal's emotional state. For example, if a dog is wagging its tail, it infers that the animal is "happy" or "excited." The server also analyzes vital sign data and compares it to reference values ​​to assess the animal's health and determine if there are any abnormalities. For example, if the body temperature is higher than normal, it will be detected as an abnormality.

[0039] These analysis results are fed back from the server to the terminal. The goggles have a function to visually display the received data, showing the user the animal's emotions and intentions in real time. The display may include text information, icons, and graphic elements. In addition, the smartphone app notifies the user of health status reports and warnings based on vital sign information sent from the pendant.

[0040] Based on this feedback, users can understand the emotional and physical state of their animals and take appropriate action. For example, if they know their dog is "happy," they can continue playing with it, or if they receive a warning about an abnormal body temperature, they can immediately rest the animal or consult a veterinarian.

[0041] Therefore, the present invention provides specific embodiments to support animal health management and emotional communication, and to enrich life with pets.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The device (goggles) continuously photographs animals with a camera and collects video data. This makes it possible to record the animals' facial expressions and movements in real time.

[0045] Step 2:

[0046] The device (sensor-equipped pendant) monitors vital signs such as heart rate and body temperature of the animal it is attached to. It measures and stores data at regular intervals.

[0047] Step 3:

[0048] The terminal wirelessly transfers captured video data and acquired vital sign data to the server. Data transfer is continuous, maintaining real-time performance.

[0049] Step 4:

[0050] The server applies an AI model to analyze the received video data, using pattern recognition to analyze the animals' facial expressions and movements. This allows it to infer the animals' emotional state. For example, it can determine emotions such as "happy" from the way the tail wags or the movement of the ears.

[0051] Step 5:

[0052] The server assesses the animal's health status based on vital sign data. It compares these values ​​to baselines to determine if heart rate and body temperature are within the normal range. If an abnormality is detected, it generates an alert.

[0053] Step 6:

[0054] The server compiles the analysis results and sends them back to the terminal as feedback data. This feedback data includes information on emotional state, health status assessment, and warning messages as needed.

[0055] Step 7:

[0056] The device (goggles) receives feedback data from the server and presents it to the user as a visual display. This includes icons representing animal emotions and simple text information.

[0057] Step 8:

[0058] The device (smartphone app) notifies the user of their health status and alert information. This allows the user to immediately understand the situation and take necessary actions.

[0059] Step 9:

[0060] Users take appropriate action towards their animals based on the information provided. They can monitor their pet's emotions and continue playing, encourage rest if there are health issues, or consult a veterinarian.

[0061] (Example 1)

[0062] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0063] In animal-human communication, accurately understanding an animal's emotional state and health condition is difficult. In particular, interpreting an animal's facial expressions and movements and taking appropriate action relies heavily on human senses and experience. In such situations, when immediate action is required regarding an animal's health, judgment errors are likely to occur. Therefore, there is a need for a means to deepen mutual understanding between animals and humans and to more effectively manage animal health by analyzing the animal's emotions and health condition in real time and providing feedback to the user.

[0064] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0065] In this invention, the server includes imaging means for acquiring animal movements as images, analysis means for analyzing the image information using a generation AI model to determine the animal's emotional state, and evaluation means for monitoring the animal's biological information, comparing it with reference values, and detecting abnormalities. This enables real-time determination of the animal's emotional state and analysis of its health status.

[0066] "An imaging device for acquiring motion as an image" refers to a device or mechanism for capturing the motion of an animal and recording it as digital image data.

[0067] "Means of analysis using generative AI models" refers to a method or apparatus that uses a generative AI model, a type of machine learning, to analyze collected image data and infer the emotional state of an animal.

[0068] An "evaluation means for monitoring biological information, comparing it to reference values, and detecting abnormalities" is a device or mechanism for collecting biological information such as an animal's body temperature and heart rate, and determining abnormalities by comparing it to a predetermined normal range.

[0069] "Feedback means" refers to a method or apparatus for providing a user with the results of an analysis regarding an animal's emotions or health status, either visually or audibly.

[0070] "Visual display means" refers to a device or mechanism that communicates an animal's emotional state or health condition to a user in the form of textual information, icons, graphic elements, etc.

[0071] A "warning device" is a method or device for notifying the user when an abnormality in the health condition of an animal is detected.

[0072] This invention is a system that monitors the emotional and health status of animals in real time and provides appropriate feedback to the user. This system primarily consists of three components: a terminal, a server, and the user.

[0073] Device Operation: The device includes goggles and a sensor pendant. The goggles have a built-in high-resolution camera to capture images of the animal's movements, recording its facial expressions and actions as video. The sensor pendant also monitors the animal's biometric information, such as heart rate and body temperature, with high precision. This data is transmitted to the server in real time.

[0074] Server-based analysis: The server analyzes the video data transmitted from the terminal using a generating AI model. This AI model is trained on diverse animal data and can determine emotional states with high accuracy. Furthermore, the server compares biometric information with existing reference values ​​to assess health status and detect any abnormalities.

[0075] Providing Feedback: Analysis results are sent back from the server to the device and visually presented to the user through the goggles' display. The feedback includes text and icons indicating emotional states, as well as warnings, allowing the user to instantly understand the animal's condition. Users can also receive warnings, such as voice notifications, via a smartphone app.

[0076] User Interaction: Users can adjust their response to the animal based on the feedback provided by the system. For example, if the system indicates that "the dog is happy," they can extend playtime, or if it warns that "the dog's body temperature is higher than normal," they can take immediate action, such as letting the dog rest or consulting a veterinarian.

[0077] Examples of specific cases and prompt statements:

[0078] As a concrete example, consider a scenario where a user observes a flock of sheep at a zoo. Using this system, if it is determined that one of the sheep is experiencing stress, measures can be taken to improve its environment.

[0079] Examples of prompt messages are as follows:

[0080] "I want to know how a system works that analyzes my dog's emotional state in real time and provides support for health management as needed."

[0081] Thus, the present invention provides an effective system to enrich the interaction between animals and humans and to support the health and well-being of animals.

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The device acquires the animal's movements and biometric information. The goggles' camera captures the animal's posture and facial expressions, and a sensor-equipped pendant measures heart rate and body temperature. The input for this step is the animal's visual and biometric information, and the output is the captured video data and acquired vital data.

[0085] Step 2:

[0086] The terminal sends acquired video data and vital data to the server. The data is sent to the server in real time using a secure and high-speed communication protocol. In this step, the acquired data is used as input to generate data packets that are transmitted to the server as output.

[0087] Step 3:

[0088] The server analyzes the video data using an AI model to determine the animal's emotional state. Specifically, it extracts features from the video, and the trained AI model infers emotions such as "happy" or "anxious." The input for this step is the video data sent from the terminal, and the output is the inferred emotional state.

[0089] Step 4:

[0090] The server evaluates the animal's health status by comparing vital data to reference values. If the data deviates from the normal range, it is judged as abnormal. The input for this step is vital data from the terminal, and the output is the judged health status and the evaluation result.

[0091] Step 5:

[0092] The server integrates the analyzed emotional and health states to generate feedback information. This feedback includes text indicating the emotional state and health-related warnings. The input for this step is the analysis results obtained in the previous step, and the output is the feedback information provided to the user.

[0093] Step 6:

[0094] The device receives feedback information provided by the server and presents it to the user visually or audibly. This may involve displaying text or icons on the goggles, or providing audio notifications via a smartphone app. The input for this step is the feedback information from the server, and the output is the presentation of this information as a notification to the user.

[0095] Step 7:

[0096] Based on the feedback information provided by the user, decisions are made regarding how to respond to the animal. For example, if there is a warning of a health problem, a decision may be made to consult a veterinarian. The input for this step is notification information from the device, and the output is the user's specific response.

[0097] (Application Example 1)

[0098] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0099] In workplaces where machinery and equipment are in operation, regular maintenance and inspections alone are insufficient to prevent all malfunctions, and unexpected stoppages or breakdowns can significantly impact production. Furthermore, constantly monitoring the status of machinery and equipment is a heavy burden for managers, highlighting the need for more efficient monitoring methods.

[0100] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0101] In this invention, the server includes imaging means for capturing images of the operation of a machine device, analysis means for analyzing the video data obtained by the imaging means and determining the operating state of the machine device, and monitoring means for monitoring the operating pattern and temperature of the machine device. This makes it possible to analyze the state of the machine device in real time and provide accurate and rapid feedback to the administrator.

[0102] "Machinery and equipment" refers to equipment or devices that automatically perform specific tasks in production sites and other similar settings.

[0103] "Imaging means" refers to a device or function that captures the movement of an object using a camera or similar device and acquires video data.

[0104] "Analysis means" refers to a function or device that uses acquired video data and sensor data to determine and analyze the state and movement of an object.

[0105] "Monitoring means" refers to devices or functions that continuously observe sensory data such as the movement patterns and temperature of an object, and collect data as needed.

[0106] "Evaluation means" refers to functions or devices that determine the operating state or abnormalities of an object based on sensing data and evaluate them by comparing them with a standard.

[0107] A "feedback mechanism" is a means of notifying users of the results of analysis and evaluation and providing them with necessary information.

[0108] The server constitutes a system for monitoring the operation of machinery and equipment installed within the factory, thereby enabling efficient and safe monitoring of the operating status of the machinery and equipment.

[0109] This system consists of multiple hardware components, primarily imaging devices, sensors, and analysis devices. The cameras capture images of the machinery's operation, acquiring video data in real time. The acquired video data is processed by an analysis device on a server to determine the machinery's operating status. The analysis device uses a generative AI model and is designed to detect operational anomalies. The analysis results are visually communicated to administrators via a feedback device, enabling them to take necessary actions quickly.

[0110] Sensors continuously record the temperature and vibration of machinery and transmit the data to a server. The server analyzes this data using evaluation equipment and compares it to baseline values ​​to proactively detect risks such as abnormal operation or overheating, and issues warnings. For example, if machinery vibrates beyond its normal operating pattern or if the temperature exceeds the designed tolerance range, the administrator is notified via the warning system.

[0111] As a concrete example, if a robotic arm operating in a factory begins to emit abnormal vibrations, this system will immediately detect the anomaly. A warning will be sent to the manager, allowing the robot to be stopped immediately, enabling investigation and rapid repair of the problem. An example of a prompt message to the generated AI model would be, "What kind of abnormal operation is there in this video frame?", prompting a specific analysis.

[0112] In this way, the integrated operation of the server and related devices ensures stable and efficient operation of the machinery and prevents production line shutdowns due to unexpected failures.

[0113] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0114] Step 1:

[0115] The server uses cameras installed within the factory to capture images of the machinery's operation. It acquires real-time video data as input and sends it to the server for processing. This video data records the machinery's movements and serves as the raw data for analysis.

[0116] Step 2:

[0117] The server processes the acquired video data using an analysis device. Using the video data obtained in step 1 as input, it performs data processing to detect operational anomalies using a generated AI model. The prompt message "What operational anomalies are present in this video frame?" is used to prompt the AI ​​for analysis. The output is a determination of whether the operating status of the machinery is abnormal or normal.

[0118] Step 3:

[0119] The server acquires operating patterns and temperature data from sensors attached to the machinery. It takes real-time sensor data as input and continuously records it through monitoring devices. This allows for the collection of detailed measurements regarding the operating status of the machinery.

[0120] Step 4:

[0121] The server analyzes the collected sensor data using an evaluation device. The operating pattern and temperature data obtained in step 3 are used as input. By comparing this data with reference values, it determines whether the machine is operating within acceptable limits. The output is a result indicating whether it is within normal limits or if there is an abnormality.

[0122] Step 5:

[0123] The server notifies the administrator of the analysis and evaluation results using a feedback device. It receives the judgment results obtained in steps 2 and 4 as input and presents them visually in a human-readable format. The output is displayed in real time through the management screen and warning alerts, allowing the administrator to quickly take necessary actions.

[0124] Step 6:

[0125] The user, or administrator, will check for any abnormalities in the machinery and equipment based on the information provided and take appropriate action. Based on the information obtained as output, they will, for example, issue emergency shutdown or repair instructions to ensure the stable operation of the system.

[0126] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0127] This invention is a system that simultaneously recognizes the emotional states of both the animal and the user (owner) and optimizes communication with the animal based on that recognition. This system mainly consists of three components: a terminal, a server, and the user.

[0128] First, the terminal is equipped with imaging capabilities to capture images of the animal's movements, continuously collecting the animal's facial expressions and actions. This data is transmitted to a server, where the server's analysis capabilities determine the animal's emotional state. Simultaneously, the terminal monitors the animal's vital signs through a sensor-equipped pendant, and the transmitted data is used by the server to assess the animal's health.

[0129] The server not only analyzes animal data but also recognizes the user's emotional state using an emotion engine. This includes methods for analyzing emotions from changes in the user's facial expressions and voice captured through the camera and microphone. The obtained user emotional state is compared with the animal's emotional state, and the correlation between the two is analyzed. Based on this analysis, the server generates advice and suggestions to optimize interaction with the pet and provides them to the user as feedback.

[0130] The device visually and audibly presents feedback received from the server to the user, informing them how to interact with their pet. It also notifies the user via an alert system if any abnormalities are detected in the animal's health. This allows users to respond quickly and use that information to improve their pet's care.

[0131] As a concrete example, consider a scenario where a user is playing with their pet. When the user's smile is detected through the goggles, the emotion engine determines that the user is "having fun." At the same time, the video data of the pet wagging its tail determines that the pet is "happy." In this situation, the server makes a suggestion such as "Please continue having fun," encouraging the user to continue the interaction.

[0132] This system allows users to understand their own emotions and those of their pets, enabling them to interact with their pets optimally based on that understanding. This invention offers a new form of relationship building that goes beyond conventional pet management.

[0133] The following describes the processing flow.

[0134] Step 1:

[0135] The device (goggles) captures the animal's movements with a camera and collects video data in real time. This data includes the animal's facial expressions and movement characteristics.

[0136] Step 2:

[0137] The device (a pendant with a sensor) continuously monitors and collects data on the animal's vital signs, such as heart rate and body temperature. This data is necessary to understand the animal's health status.

[0138] Step 3:

[0139] The server receives video data transmitted from the terminal and analyzes it using an AI model to determine the animal's emotional state. This allows it to determine whether the animal is feeling emotions such as "happy" or "calm."

[0140] Step 4:

[0141] The server analyzes vital sign data and assesses the animal's health status. If the assessment results exceed the standard values, the server determines it to be abnormal and generates information about it.

[0142] Step 5:

[0143] The device uses its camera and microphone to collect facial and audio data from the user in order to recognize the user's emotional state. This information is essential for interpreting the user's reactions and emotions.

[0144] Step 6:

[0145] The server analyzes the user's emotional data using an emotion engine to determine the user's current emotional state. For example, if a smile is detected, it is determined that the user is "having fun."

[0146] Step 7:

[0147] The server compares the emotional states of the animals and the users and analyzes their correlation. Simultaneously, it generates suggestions for optimizing the interaction based on the animals' health status and the users' emotional states.

[0148] Step 8:

[0149] The device displays feedback from the server to the user. This feedback includes information about the animal's emotions and health, as well as advice on what to do. This allows the user to take appropriate action.

[0150] Step 9:

[0151] Users interact with their pets based on information provided through the device's interface. If necessary, they can take appropriate action in response to health alerts. For example, if a health problem is detected, they can allow their pet to rest or consult a veterinarian.

[0152] (Example 2)

[0153] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0154] Conventional animal communication systems were unable to simultaneously analyze the emotional state of the animal and the emotions of the user, making it difficult to optimize interactions based on mutual emotions. Furthermore, there was a lack of systems that could quickly notify the user of abnormalities in the animal's health, making it impossible to provide appropriate feedback to deepen the relationship with the animal.

[0155] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0156] In this invention, the server includes video acquisition means for capturing images of animal movements, data analysis means for analyzing animal data, state evaluation means for evaluating health status based on biological information, user analysis means for analyzing the user's facial expressions and voice, correlation analysis means for comparing the emotional states of the animal and the user, and information presentation means for providing the analysis results to the user. This makes it possible to optimize interaction based on the emotions of both the animal and the user, and to quickly provide health warnings as needed.

[0157] "Image acquisition means" refers to a hardware and software system for capturing images of an animal's movements and facial expressions and acquiring that image data.

[0158] "Data analysis means" refers to a function that includes an algorithm for analyzing acquired video data and determining the emotional state of an animal.

[0159] "Indicator collection means" refers to equipment and systems for monitoring biological information such as the heart rate and body temperature of animals and continuously collecting data.

[0160] A "condition evaluation method" is a technology that performs analysis and makes judgments to evaluate the health status of an animal based on collected biological information.

[0161] "User analysis methods" refer to technologies that collect and analyze a user's facial expressions and voice to determine the user's emotional state.

[0162] A "correlation analysis method" is a method for comparing the emotional state of an animal with the emotional state of a user and analyzing their interrelationships.

[0163] "Information presentation means" refers to methods that utilize display devices and output devices to provide users with analysis results and suggestions visually or audibly.

[0164] The "warning function" is a mechanism that sends a warning to the user when it detects an abnormality in the animal's health condition.

[0165] This invention is a system that simultaneously recognizes the emotional states of both animals and users and optimizes their interactions. This system is primarily operated by a terminal, a server, and the user.

[0166] The terminal uses a camera as a means of acquiring images to capture the animal's movements. This camera continuously records the animal's facial expressions and movements, and acquires the data. Furthermore, the terminal also includes an indicator collection means for collecting biometric information, which is a pendant equipped with a sensor. This pendant monitors the animal's heart rate, body temperature, and other parameters. This data is transmitted from the terminal to a server.

[0167] The server processes the received data using data analysis tools to determine the animal's emotional state. It analyzes the state using specific algorithms based on the animal's facial expressions and movement data. Furthermore, a state evaluation tool is used to assess the animal's health based on biological information.

[0168] In addition, the device uses a camera and microphone as user analysis tools to analyze the user's emotions. This collects changes in the user's facial expressions and tone of voice, which are then analyzed on a server. As a result, the server compares the emotional states of the animal and the user using correlation analysis tools to confirm the relationship between their emotions.

[0169] Ultimately, the server provides feedback to the user through information presentation methods based on these analysis results. For example, if a smile is detected while the user is playing with their pet and the animal is wagging its tail, the server will generate and provide a suggestion such as "Let's continue having fun." In this way, the user can receive information to optimize their interaction with their pet.

[0170] The advantage of this system lies in its ability to comprehensively analyze the emotional states of both the user and the pet, deepening their understanding based on their respective emotions. An example of a prompt might be, "Read the pet's emotions from its behavioral patterns and compare them with the user's facial expressions and voice to provide the best suggestion." This prompt enables the generative AI model to provide an advanced communication tool based on emotional understanding.

[0171] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0172] Step 1:

[0173] The device uses a camera to capture images of animal movements, recording the animal's facial expressions and actions in real time. It acquires camera video data as input and continuously streams that video. The output obtained from this process is video data related to the animal's movements.

[0174] Step 2:

[0175] The device uses a sensor-equipped pendant to collect animal biometric information. Inputs include monitoring vital signs such as the animal's heart rate and body temperature. Outputs provide data related to the animal's health status.

[0176] Step 3:

[0177] The terminal transmits collected video and biometric data to the server. This data is acquired from the terminal as input and rapidly transferred to the server. The output consists of the animals' emotional state and health information, awaiting analysis.

[0178] Step 4:

[0179] The server processes the received video data using data analysis tools. Using the video data as input, it analyzes the animal's facial expressions and movements and executes an algorithm to determine its emotional state. The output is the animal's specific emotional state (e.g., "happy" or "anxious").

[0180] Step 5:

[0181] The server evaluates the health status of animals based on collected biometric data. It uses biometric data as input and performs an evaluation by comparing it to normal health standards. The output is the evaluation result, indicating whether the animal is healthy or not.

[0182] Step 6:

[0183] The device uses a camera and microphone to collect the user's facial expressions and voice. It takes the user's facial expression changes and voice tone as input, and outputs data on the user's emotional state.

[0184] Step 7:

[0185] The server uses correlation analysis to analyze the user's emotional state and compare it to the animal's emotional state. It takes the emotional data of both as input and analyzes their relationship. The output is an evaluation of the emotional correlation between the animal and the user.

[0186] Step 8:

[0187] Based on the analysis results obtained, the server generates suggestions for how the user should interact with animals. All analysis results are used as input, and the information presentation means creates appropriate feedback using a generating AI model. The output is a specific suggestion, such as "Let's continue playing."

[0188] Step 9:

[0189] The terminal presents suggestions from the server to the user visually and audibly. Input is feedback data from the server, which is presented through the user's display or speaker. Output is specific advice or warnings received by the user.

[0190] (Application Example 2)

[0191] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0192] In modern brick-and-mortar stores, improving services for customers who bring their pets is crucial, but it is difficult to understand the emotional state of both the customer and their pet in real time and provide appropriate services based on that understanding. In particular, there is a need to accurately understand the health and emotional state of pets and provide interactions accordingly, but conventional technologies do not adequately address this.

[0193] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0194] In this invention, the server includes means for a camera that captures images of an animal's movements, means for an analysis function that analyzes the video information obtained by the camera to determine the animal's emotional state, and means for an information presentation function that grasps the emotional state of users and pets in the store in real time and provides information to promote interaction between the users and pets. This makes it possible to provide appropriate services in physical stores based on the emotional state of customers and pets.

[0195] A "filming device" is a device used to record the movements of animals as video, and has the function of providing video information.

[0196] The "analysis function" is a function that processes video information obtained from the camera to determine the emotional state of the animal.

[0197] The "monitoring function" is a function that continuously observes an animal's vital information and manages its health status.

[0198] The "evaluation function" is a function that evaluates the health status of an animal based on vital information obtained through the monitoring function.

[0199] A "notification device" is a device that informs users of the analysis results obtained through analysis and evaluation functions.

[0200] The "information presentation function" is a function that analyzes the emotional state of customers and their pets in real time within the store and provides appropriate information to customers based on the results.

[0201] The "alarm function" is a feature that warns the user if an abnormality is detected in the animal's health condition.

[0202] In order to implement this invention, a system is needed that uses various hardware and software to efficiently analyze the emotional states of animals and users, and to optimize interactions in physical stores.

[0203] The server manages the imaging equipment used to capture images of animal movements and analyzes the captured video information. The software used includes OpenCV for video analysis and emotion analysis engines such as Hume AI for emotional state analysis. The server acquires vital information from the animals, which is collected via medical sensors. This vital information is processed using cloud platforms such as Google Cloud.

[0204] The server provides store staff with real-time information based on the analyzed data. The information display function alerts customers about what services are appropriate in the store based on the emotional state of the user and their pet. For example, if a pet is detected as restless, the server will recommend relaxation items to the store staff.

[0205] For example, when a customer visits a store with their dog, the camera may capture the customer's smile and the movement of their pet's tail as they walk through the store. Based on this information, the server can provide feedback to the staff, such as, "That looks like a fun time. Please continue." If any abnormalities are detected in the animal's health or emotional state, a warning function allows for quick countermeasures to be taken.

[0206] By utilizing generative AI models, appropriate prompt messages can be created. Examples of prompt messages include, "Please diagnose this person's emotions using the facial expression data obtained from the camera," and "Identify the dog's emotions from this image and indicate them as either 'happy,' 'sad,' or 'nervous.'" This enables accurate emotion analysis, making in-store services more personalized.

[0207] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0208] Step 1:

[0209] The terminal uses a camera to visualize animal movements and acquire video information. The input is the animal's movements, and the output is video data. This video data is used for subsequent analysis.

[0210] Step 2:

[0211] The server uses OpenCV to analyze the video data sent from the terminal. In this step, the animal's facial expressions and movements are broken down frame by frame, and the emotional state for each frame is determined. The input is the video data obtained in step 1, and the output is data indicating the animal's emotions.

[0212] Step 3:

[0213] The device collects vital information from animals via medical sensors. The input is the animal's biological information, and the output is vital data. This data is monitored in real time.

[0214] Step 4:

[0215] The server processes vital data using the Google Cloud platform. In this step, the health status of the animals is assessed and any abnormalities are detected. The input is the vital data collected in step 3, and the output is health assessment data.

[0216] Step 5:

[0217] The server also analyzes the user's emotional state using an emotion analysis engine such as Hume AI. In this step, the user's facial expressions and voice collected through the camera and microphone are analyzed. The input is the user's facial expressions and voice data, and the output is the user's emotional state.

[0218] Step 6:

[0219] The server integrates the emotional states of animals and users and uses information presentation functions to provide appropriate service strategies to store staff in real time. The input is emotional state data from steps 2 and 5, and the output is service suggestion information. This result is displayed on the staff interface.

[0220] Step 7:

[0221] The server uses an alarm function to warn the user if an anomaly is detected. The input is the health assessment data from step 4, and the output is the warning information. This allows for a rapid response.

[0222] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0223] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0224] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0225] [Second Embodiment]

[0226] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0227] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0228] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0229] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0230] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0231] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0232] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0233] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0234] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0235] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0236] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0237] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0238] This invention is a system that monitors the emotional and health status of animals in real time and provides visual or audible feedback to the user. The system mainly consists of three components: a terminal (goggles and a sensor-equipped pendant), a server, and the user.

[0239] First, the goggles used as terminals have a built-in camera to capture images of the animals' movements. While observing the animals, the goggles capture their facial expressions and movements, recording them as video data. In addition, a sensor-equipped pendant is attached to the animals' bodies to acquire vital sign data such as heart rate and body temperature. This data is transmitted to a server via the goggles.

[0240] The server analyzes the received video data using an AI model to determine the animal's emotional state. For example, if a dog is wagging its tail, it infers that the animal is "happy" or "excited." The server also analyzes vital sign data and compares it to reference values ​​to assess the animal's health and determine if there are any abnormalities. For example, if the body temperature is higher than normal, it will be detected as an abnormality.

[0241] These analysis results are fed back from the server to the terminal. The goggles have a function to visually display the received data, showing the user the animal's emotions and intentions in real time. The display may include text information, icons, and graphic elements. In addition, the smartphone app notifies the user of health status reports and warnings based on vital sign information sent from the pendant.

[0242] Based on this feedback, users can understand the emotional and physical state of their animals and take appropriate action. For example, if they know their dog is "happy," they can continue playing with it, or if they receive a warning about an abnormal body temperature, they can immediately rest the animal or consult a veterinarian.

[0243] Therefore, the present invention provides specific embodiments to support animal health management and emotional communication, and to enrich life with pets.

[0244] The following describes the processing flow.

[0245] Step 1:

[0246] The device (goggles) continuously photographs animals with a camera and collects video data. This makes it possible to record the animals' facial expressions and movements in real time.

[0247] Step 2:

[0248] The device (sensor-equipped pendant) monitors vital signs such as heart rate and body temperature of the animal it is attached to. It measures and stores data at regular intervals.

[0249] Step 3:

[0250] The terminal wirelessly transfers captured video data and acquired vital sign data to the server. Data transfer is continuous, maintaining real-time performance.

[0251] Step 4:

[0252] The server applies an AI model to analyze the received video data, using pattern recognition to analyze the animals' facial expressions and movements. This allows it to infer the animals' emotional state. For example, it can determine emotions such as "happy" from the way the tail wags or the movement of the ears.

[0253] Step 5:

[0254] The server assesses the animal's health status based on vital sign data. It compares these values ​​to baselines to determine if heart rate and body temperature are within the normal range. If an abnormality is detected, it generates an alert.

[0255] Step 6:

[0256] The server compiles the analysis results and sends them back to the terminal as feedback data. This feedback data includes information on emotional state, health status assessment, and warning messages as needed.

[0257] Step 7:

[0258] The device (goggles) receives feedback data from the server and presents it to the user as a visual display. This includes icons representing animal emotions and simple text information.

[0259] Step 8:

[0260] The device (smartphone app) notifies the user of their health status and alert information. This allows the user to immediately understand the situation and take necessary actions.

[0261] Step 9:

[0262] Users take appropriate action towards their animals based on the information provided. They can monitor their pet's emotions and continue playing, encourage rest if there are health issues, or consult a veterinarian.

[0263] (Example 1)

[0264] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0265] In animal-human communication, accurately understanding an animal's emotional state and health condition is difficult. In particular, interpreting an animal's facial expressions and movements and taking appropriate action relies heavily on human senses and experience. In such situations, when immediate action is required regarding an animal's health, judgment errors are likely to occur. Therefore, there is a need for a means to deepen mutual understanding between animals and humans and to more effectively manage animal health by analyzing the animal's emotions and health condition in real time and providing feedback to the user.

[0266] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0267] In this invention, the server includes imaging means for acquiring animal movements as images, analysis means for analyzing the image information using a generation AI model to determine the animal's emotional state, and evaluation means for monitoring the animal's biological information, comparing it with reference values, and detecting abnormalities. This enables real-time determination of the animal's emotional state and analysis of its health status.

[0268] "An imaging device for acquiring motion as an image" refers to a device or mechanism for capturing the motion of an animal and recording it as digital image data.

[0269] "Means of analysis using generative AI models" refers to a method or apparatus that uses a generative AI model, a type of machine learning, to analyze collected image data and infer the emotional state of an animal.

[0270] An "evaluation means for monitoring biological information, comparing it to reference values, and detecting abnormalities" is a device or mechanism for collecting biological information such as an animal's body temperature and heart rate, and determining abnormalities by comparing it to a predetermined normal range.

[0271] "Feedback means" refers to a method or apparatus for providing a user with the results of an analysis regarding an animal's emotions or health status, either visually or audibly.

[0272] "Visual display means" refers to a device or mechanism that communicates an animal's emotional state or health condition to a user in the form of textual information, icons, graphic elements, etc.

[0273] A "warning device" is a method or device for notifying the user when an abnormality in the health condition of an animal is detected.

[0274] This invention is a system that monitors the emotional and health status of animals in real time and provides appropriate feedback to the user. This system primarily consists of three components: a terminal, a server, and the user.

[0275] Device Operation: The device includes goggles and a sensor pendant. The goggles have a built-in high-resolution camera to capture images of the animal's movements, recording its facial expressions and actions as video. The sensor pendant also monitors the animal's biometric information, such as heart rate and body temperature, with high precision. This data is transmitted to the server in real time.

[0276] Server-based analysis: The server analyzes the video data transmitted from the terminal using a generating AI model. This AI model is trained on diverse animal data and can determine emotional states with high accuracy. Furthermore, the server compares biometric information with existing reference values ​​to assess health status and detect any abnormalities.

[0277] Providing Feedback: Analysis results are sent back from the server to the device and visually presented to the user through the goggles' display. The feedback includes text and icons indicating emotional states, as well as warnings, allowing the user to instantly understand the animal's condition. Users can also receive warnings, such as voice notifications, via a smartphone app.

[0278] User Interaction: Users can adjust their response to the animal based on the feedback provided by the system. For example, if the system indicates that "the dog is happy," they can extend playtime, or if it warns that "the dog's body temperature is higher than normal," they can take immediate action, such as letting the dog rest or consulting a veterinarian.

[0279] Examples of specific cases and prompt statements:

[0280] As a concrete example, consider a scenario where a user observes a flock of sheep at a zoo. Using this system, if it is determined that one of the sheep is experiencing stress, measures can be taken to improve its environment.

[0281] Examples of prompt messages are as follows:

[0282] "I want to know the mechanism of a system that can analyze the emotional state of a pet dog in real time and support appropriate health management."

[0283] Thus, the present invention provides an effective system for enriching the interaction between animals and humans and supporting the health and well-being of animals.

[0284] The flow of the specific process in Example 1 will be described using FIG. 11.

[0285] Step 1:

[0286] The terminal acquires the actions and biological information of the animal. The camera of the goggles takes pictures of the animal's posture and expression, and the sensor-equipped pendant measures the heart rate and body temperature. The input of this step is the visual information and biological information of the animal, and the output is the captured video data and the acquired vital data.

[0287] Step 2:

[0288] The terminal transmits the acquired video data and vital data to the server. The data is sent to the server in real time using a secure and high-speed communication protocol. In this step, the acquired data is used as the input, and data packets transmitted to the server are generated as the output.

[0289] Step 3:

[0290] The server analyzes the video data using a generative AI model to determine the emotional state of the animal. Specifically, the features of the video are extracted, and the trained AI model estimates emotions such as "happy" and "uneasy". The input of this step is the video data transmitted from the terminal, and the output is the estimated emotional state.

[0291] Step 4:

[0292] The server evaluates the animal's health status by comparing vital data to reference values. If the data deviates from the normal range, it is judged as abnormal. The input for this step is vital data from the terminal, and the output is the judged health status and the evaluation result.

[0293] Step 5:

[0294] The server integrates the analyzed emotional and health states to generate feedback information. This feedback includes text indicating the emotional state and health-related warnings. The input for this step is the analysis results obtained in the previous step, and the output is the feedback information provided to the user.

[0295] Step 6:

[0296] The device receives feedback information provided by the server and presents it to the user visually or audibly. This may involve displaying text or icons on the goggles, or providing audio notifications via a smartphone app. The input for this step is the feedback information from the server, and the output is the presentation of this information as a notification to the user.

[0297] Step 7:

[0298] Based on the feedback information provided by the user, decisions are made regarding how to respond to the animal. For example, if there is a warning of a health problem, a decision may be made to consult a veterinarian. The input for this step is notification information from the device, and the output is the user's specific response.

[0299] (Application Example 1)

[0300] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0301] At the site where the mechanical device operates, it is difficult to prevent all abnormalities solely through regular maintenance inspections, and unexpected stops and failures may have a significant impact on production. In addition, it is a heavy burden for the administrator to constantly monitor the status of the mechanical device, and an efficient monitoring method is required.

[0302] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0303] In this invention, the server includes an imaging means for imaging the operation of the mechanical device, an analysis means for analyzing the video data obtained by the imaging means and determining the operating state of the mechanical device, and a monitoring means for monitoring the operation pattern and temperature of the mechanical device. As a result, it becomes possible to analyze the state of the mechanical device in real time and provide accurate and rapid feedback to the administrator.

[0304] The "mechanical device" refers to equipment and devices that automatically perform specific operations at the production site or the like.

[0305] The "imaging means" is a device or function for taking pictures of the operation of an object using a camera or the like and acquiring video data.

[0306] The "analysis means" is a function or device for determining and analyzing the state and operation of an object using the acquired video data and sensor data.

[0307] The "monitoring means" is a device or function for constantly observing the sensed data such as the operation pattern and temperature of an object and collecting data as necessary.

[0308] The "evaluation means" is a function or device for judging the operation state and abnormalities of an object based on the sensed data and evaluating by comparing with a standard.

[0309] The "feedback means" is a means for notifying the user of the results of analysis and evaluation and providing necessary information.

[0310] The server constitutes a system for monitoring the operation of machinery and equipment installed within the factory, thereby enabling efficient and safe monitoring of the operating status of the machinery and equipment.

[0311] This system consists of multiple hardware components, primarily imaging devices, sensors, and analysis devices. The cameras capture images of the machinery's operation, acquiring video data in real time. The acquired video data is processed by an analysis device on a server to determine the machinery's operating status. The analysis device uses a generative AI model and is designed to detect operational anomalies. The analysis results are visually communicated to administrators via a feedback device, enabling them to take necessary actions quickly.

[0312] Sensors continuously record the temperature and vibration of machinery and transmit the data to a server. The server analyzes this data using evaluation equipment and compares it to baseline values ​​to proactively detect risks such as abnormal operation or overheating, and issues warnings. For example, if machinery vibrates beyond its normal operating pattern or if the temperature exceeds the designed tolerance range, the administrator is notified via the warning system.

[0313] As a concrete example, if a robotic arm operating in a factory begins to emit abnormal vibrations, this system will immediately detect the anomaly. A warning will be sent to the manager, allowing the robot to be stopped immediately, enabling investigation and rapid repair of the problem. An example of a prompt message to the generated AI model would be, "What kind of abnormal operation is there in this video frame?", prompting a specific analysis.

[0314] In this way, the integrated operation of the server and related devices ensures stable and efficient operation of the machinery and prevents production line shutdowns due to unexpected failures.

[0315] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0316] Step 1:

[0317] The server uses cameras installed within the factory to capture images of the machinery's operation. It acquires real-time video data as input and sends it to the server for processing. This video data records the machinery's movements and serves as the raw data for analysis.

[0318] Step 2:

[0319] The server processes the acquired video data using an analysis device. Using the video data obtained in step 1 as input, it performs data processing to detect operational anomalies using a generated AI model. The prompt message "What operational anomalies are present in this video frame?" is used to prompt the AI ​​for analysis. The output is a determination of whether the operating status of the machinery is abnormal or normal.

[0320] Step 3:

[0321] The server acquires operating patterns and temperature data from sensors attached to the machinery. It takes real-time sensor data as input and continuously records it through monitoring devices. This allows for the collection of detailed measurements regarding the operating status of the machinery.

[0322] Step 4:

[0323] The server analyzes the collected sensor data using an evaluation device. The operating pattern and temperature data obtained in step 3 are used as input. By comparing this data with reference values, it determines whether the machine is operating within acceptable limits. The output is a result indicating whether it is within normal limits or if there is an abnormality.

[0324] Step 5:

[0325] The server notifies the administrator of the analysis and evaluation results using a feedback device. It receives the judgment results obtained in steps 2 and 4 as input and presents them visually in a human-readable format. The output is displayed in real time through the management screen and warning alerts, allowing the administrator to quickly take necessary actions.

[0326] Step 6:

[0327] The user, or administrator, will check for any abnormalities in the machinery and equipment based on the information provided and take appropriate action. Based on the information obtained as output, they will, for example, issue emergency shutdown or repair instructions to ensure the stable operation of the system.

[0328] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0329] This invention is a system that simultaneously recognizes the emotional states of both the animal and the user (owner) and optimizes communication with the animal based on that recognition. This system mainly consists of three components: a terminal, a server, and the user.

[0330] First, the terminal is equipped with imaging capabilities to capture images of the animal's movements, continuously collecting the animal's facial expressions and actions. This data is transmitted to a server, where the server's analysis capabilities determine the animal's emotional state. Simultaneously, the terminal monitors the animal's vital signs through a sensor-equipped pendant, and the transmitted data is used by the server to assess the animal's health.

[0331] The server not only analyzes animal data but also recognizes the user's emotional state using an emotion engine. This includes methods for analyzing emotions from changes in the user's facial expressions and voice captured through the camera and microphone. The obtained user emotional state is compared with the animal's emotional state, and the correlation between the two is analyzed. Based on this analysis, the server generates advice and suggestions to optimize interaction with the pet and provides them to the user as feedback.

[0332] The device visually and audibly presents feedback received from the server to the user, informing them how to interact with their pet. It also notifies the user via an alert system if any abnormalities are detected in the animal's health. This allows users to respond quickly and use that information to improve their pet's care.

[0333] As a concrete example, consider a scenario where a user is playing with their pet. When the user's smile is detected through the goggles, the emotion engine determines that the user is "having fun." At the same time, the video data of the pet wagging its tail determines that the pet is "happy." In this situation, the server makes a suggestion such as "Please continue having fun," encouraging the user to continue the interaction.

[0334] This system allows users to understand their own emotions and those of their pets, enabling them to interact with their pets optimally based on that understanding. This invention offers a new form of relationship building that goes beyond conventional pet management.

[0335] The following describes the processing flow.

[0336] Step 1:

[0337] The device (goggles) captures the animal's movements with a camera and collects video data in real time. This data includes the animal's facial expressions and movement characteristics.

[0338] Step 2:

[0339] The device (a pendant with a sensor) continuously monitors and collects data on the animal's vital signs, such as heart rate and body temperature. This data is necessary to understand the animal's health status.

[0340] Step 3:

[0341] The server receives video data transmitted from the terminal and analyzes it using an AI model to determine the animal's emotional state. This allows it to determine whether the animal is feeling emotions such as "happy" or "calm."

[0342] Step 4:

[0343] The server analyzes vital sign data and assesses the animal's health status. If the assessment results exceed the standard values, the server determines it to be abnormal and generates information about it.

[0344] Step 5:

[0345] The device uses its camera and microphone to collect facial and audio data from the user in order to recognize the user's emotional state. This information is essential for interpreting the user's reactions and emotions.

[0346] Step 6:

[0347] The server analyzes the user's emotional data using an emotion engine to determine the user's current emotional state. For example, if a smile is detected, it is determined that the user is "having fun."

[0348] Step 7:

[0349] The server compares the emotional states of the animals and the users and analyzes their correlation. Simultaneously, it generates suggestions for optimizing the interaction based on the animals' health status and the users' emotional states.

[0350] Step 8:

[0351] The device displays feedback from the server to the user. This feedback includes information about the animal's emotions and health, as well as advice on what to do. This allows the user to take appropriate action.

[0352] Step 9:

[0353] Users interact with their pets based on information provided through the device's interface. If necessary, they can take appropriate action in response to health alerts. For example, if a health problem is detected, they can allow their pet to rest or consult a veterinarian.

[0354] (Example 2)

[0355] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0356] Conventional animal communication systems were unable to simultaneously analyze the emotional state of the animal and the emotions of the user, making it difficult to optimize interactions based on mutual emotions. Furthermore, there was a lack of systems that could quickly notify the user of abnormalities in the animal's health, making it impossible to provide appropriate feedback to deepen the relationship with the animal.

[0357] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0358] In this invention, the server includes video acquisition means for capturing images of animal movements, data analysis means for analyzing animal data, state evaluation means for evaluating health status based on biological information, user analysis means for analyzing the user's facial expressions and voice, correlation analysis means for comparing the emotional states of the animal and the user, and information presentation means for providing the analysis results to the user. This makes it possible to optimize interaction based on the emotions of both the animal and the user, and to quickly provide health warnings as needed.

[0359] "Image acquisition means" refers to a hardware and software system for capturing images of an animal's movements and facial expressions and acquiring that image data.

[0360] "Data analysis means" refers to a function that includes an algorithm for analyzing acquired video data and determining the emotional state of an animal.

[0361] "Indicator collection means" refers to equipment and systems for monitoring biological information such as the heart rate and body temperature of animals and continuously collecting data.

[0362] A "condition evaluation method" is a technology that performs analysis and makes judgments to evaluate the health status of an animal based on collected biological information.

[0363] "User analysis methods" refer to technologies that collect and analyze a user's facial expressions and voice to determine the user's emotional state.

[0364] A "correlation analysis method" is a method for comparing the emotional state of an animal with the emotional state of a user and analyzing their interrelationships.

[0365] "Information presentation means" refers to methods that utilize display devices and output devices to provide users with analysis results and suggestions visually or audibly.

[0366] The "warning function" is a mechanism that sends a warning to the user when it detects an abnormality in the animal's health condition.

[0367] This invention is a system that simultaneously recognizes the emotional states of both animals and users and optimizes their interactions. This system is primarily operated by a terminal, a server, and the user.

[0368] The terminal uses a camera as a means of acquiring images to capture the animal's movements. This camera continuously records the animal's facial expressions and movements, and acquires the data. Furthermore, the terminal also includes an indicator collection means for collecting biometric information, which is a pendant equipped with a sensor. This pendant monitors the animal's heart rate, body temperature, and other parameters. This data is transmitted from the terminal to a server.

[0369] The server processes the received data using data analysis tools to determine the animal's emotional state. It analyzes the state using specific algorithms based on the animal's facial expressions and movement data. Furthermore, a state evaluation tool is used to assess the animal's health based on biological information.

[0370] In addition, the device uses a camera and microphone as user analysis tools to analyze the user's emotions. This collects changes in the user's facial expressions and tone of voice, which are then analyzed on a server. As a result, the server compares the emotional states of the animal and the user using correlation analysis tools to confirm the relationship between their emotions.

[0371] Ultimately, the server provides feedback to the user through information presentation methods based on these analysis results. For example, if a smile is detected while the user is playing with their pet and the animal is wagging its tail, the server will generate and provide a suggestion such as "Let's continue having fun." In this way, the user can receive information to optimize their interaction with their pet.

[0372] The advantage of this system lies in its ability to comprehensively analyze the emotional states of both the user and the pet, deepening their understanding based on their respective emotions. An example of a prompt might be, "Read the pet's emotions from its behavioral patterns and compare them with the user's facial expressions and voice to provide the best suggestion." This prompt enables the generative AI model to provide an advanced communication tool based on emotional understanding.

[0373] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0374] Step 1:

[0375] The device uses a camera to capture images of animal movements, recording the animal's facial expressions and actions in real time. It acquires camera video data as input and continuously streams that video. The output obtained from this process is video data related to the animal's movements.

[0376] Step 2:

[0377] The device uses a sensor-equipped pendant to collect animal biometric information. Inputs include monitoring vital signs such as the animal's heart rate and body temperature. Outputs provide data related to the animal's health status.

[0378] Step 3:

[0379] The terminal transmits collected video and biometric data to the server. This data is acquired from the terminal as input and rapidly transferred to the server. The output consists of the animals' emotional state and health information, awaiting analysis.

[0380] Step 4:

[0381] The server processes the received video data using data analysis tools. Using the video data as input, it analyzes the animal's facial expressions and movements and executes an algorithm to determine its emotional state. The output is the animal's specific emotional state (e.g., "happy" or "anxious").

[0382] Step 5:

[0383] The server evaluates the health status of animals based on collected biometric data. It uses biometric data as input and performs an evaluation by comparing it to normal health standards. The output is the evaluation result, indicating whether the animal is healthy or not.

[0384] Step 6:

[0385] The device uses a camera and microphone to collect the user's facial expressions and voice. It takes the user's facial expression changes and voice tone as input, and outputs data on the user's emotional state.

[0386] Step 7:

[0387] The server uses correlation analysis to analyze the user's emotional state and compare it to the animal's emotional state. It takes the emotional data of both as input and analyzes their relationship. The output is an evaluation of the emotional correlation between the animal and the user.

[0388] Step 8:

[0389] Based on the analysis results obtained, the server generates suggestions for how the user should interact with animals. All analysis results are used as input, and the information presentation means creates appropriate feedback using a generating AI model. The output is a specific suggestion, such as "Let's continue playing."

[0390] Step 9:

[0391] The terminal presents suggestions from the server to the user visually and audibly. Input is feedback data from the server, which is presented through the user's display or speaker. Output is specific advice or warnings received by the user.

[0392] (Application Example 2)

[0393] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0394] In modern brick-and-mortar stores, improving services for customers who bring their pets is crucial, but it is difficult to understand the emotional state of both the customer and their pet in real time and provide appropriate services based on that understanding. In particular, there is a need to accurately understand the health and emotional state of pets and provide interactions accordingly, but conventional technologies do not adequately address this.

[0395] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0396] In this invention, the server includes means for a camera that captures images of an animal's movements, means for an analysis function that analyzes the video information obtained by the camera to determine the animal's emotional state, and means for an information presentation function that grasps the emotional state of users and pets in the store in real time and provides information to promote interaction between the users and pets. This makes it possible to provide appropriate services in physical stores based on the emotional state of customers and pets.

[0397] A "filming device" is a device used to record the movements of animals as video, and has the function of providing video information.

[0398] The "analysis function" is a function that processes video information obtained from the camera to determine the emotional state of the animal.

[0399] The "monitoring function" is a function that continuously observes an animal's vital information and manages its health status.

[0400] The "evaluation function" is a function that evaluates the health status of an animal based on vital information obtained through the monitoring function.

[0401] A "notification device" is a device that informs users of the analysis results obtained through analysis and evaluation functions.

[0402] The "information presentation function" is a function that analyzes the emotional state of customers and their pets in real time within the store and provides appropriate information to customers based on the results.

[0403] The "alarm function" is a feature that warns the user if an abnormality is detected in the animal's health condition.

[0404] In order to implement this invention, a system is needed that uses various hardware and software to efficiently analyze the emotional states of animals and users, and to optimize interactions in physical stores.

[0405] The server manages the imaging equipment used to capture images of animal movements and analyzes the captured video information. The software used includes OpenCV for video analysis and emotion analysis engines such as Hume AI for emotional state analysis. The server also acquires vital information from the animals, which is collected via medical sensors. This vital information is processed using cloud platforms such as Google Cloud.

[0406] The server provides store staff with real-time information based on the analyzed data. The information display function alerts customers about what services are appropriate in the store based on the emotional state of the user and their pet. For example, if a pet is detected as restless, the server will recommend relaxation items to the store staff.

[0407] For example, when a customer visits a store with their dog, the camera may capture the customer's smile and the movement of their pet's tail as they walk through the store. Based on this information, the server can provide feedback to the staff, such as, "That looks like a fun time. Please continue." If any abnormalities are detected in the animal's health or emotional state, a warning function allows for quick countermeasures to be taken.

[0408] By utilizing generative AI models, appropriate prompt messages can be created. Examples of prompt messages include, "Please diagnose this person's emotions using the facial expression data obtained from the camera," and "Identify the dog's emotions from this image and indicate them as either 'happy,' 'sad,' or 'nervous.'" This enables accurate emotion analysis, making in-store services more personalized.

[0409] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0410] Step 1:

[0411] The terminal uses a camera to visualize animal movements and acquire video information. The input is the animal's movements, and the output is video data. This video data is used for subsequent analysis.

[0412] Step 2:

[0413] The server uses OpenCV to analyze the video data sent from the terminal. In this step, the animal's facial expressions and movements are broken down frame by frame, and the emotional state for each frame is determined. The input is the video data obtained in step 1, and the output is data indicating the animal's emotions.

[0414] Step 3:

[0415] The device collects vital information from animals via medical sensors. The input is the animal's biological information, and the output is vital data. This data is monitored in real time.

[0416] Step 4:

[0417] The server processes vital data using the Google Cloud platform. In this step, the health status of the animals is assessed and any abnormalities are detected. The input is the vital data collected in step 3, and the output is health assessment data.

[0418] Step 5:

[0419] The server also analyzes the user's emotional state using an emotion analysis engine such as Hume AI. In this step, the user's facial expressions and voice collected through the camera and microphone are analyzed. The input is the user's facial expressions and voice data, and the output is the user's emotional state.

[0420] Step 6:

[0421] The server integrates the emotional states of animals and users and uses information presentation functions to provide appropriate service strategies to store staff in real time. The input is emotional state data from steps 2 and 5, and the output is service suggestion information. This result is displayed on the staff interface.

[0422] Step 7:

[0423] The server uses an alarm function to warn the user if an anomaly is detected. The input is the health assessment data from step 4, and the output is the warning information. This allows for a rapid response.

[0424] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0425] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0426] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0427] [Third Embodiment]

[0428] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0429] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0430] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0431] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0432] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0433] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0434] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0435] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0436] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0437] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0438] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0439] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0440] This invention is a system that monitors the emotional and health status of animals in real time and provides visual or audible feedback to the user. The system mainly consists of three components: a terminal (goggles and a sensor-equipped pendant), a server, and the user.

[0441] First, the goggles used as terminals have a built-in camera to capture images of the animals' movements. While observing the animals, the goggles capture their facial expressions and movements, recording them as video data. In addition, a sensor-equipped pendant is attached to the animals' bodies to acquire vital sign data such as heart rate and body temperature. This data is transmitted to a server via the goggles.

[0442] The server analyzes the received video data using an AI model to determine the animal's emotional state. For example, if a dog is wagging its tail, it infers that the animal is "happy" or "excited." The server also analyzes vital sign data and compares it to reference values ​​to assess the animal's health and determine if there are any abnormalities. For example, if the body temperature is higher than normal, it will be detected as an abnormality.

[0443] These analysis results are fed back from the server to the terminal. The goggles have a function to visually display the received data, showing the user the animal's emotions and intentions in real time. The display may include text information, icons, and graphic elements. In addition, the smartphone app notifies the user of health status reports and warnings based on vital sign information sent from the pendant.

[0444] Based on this feedback, users can understand the emotional and physical state of their animals and take appropriate action. For example, if they know their dog is "happy," they can continue playing with it, or if they receive a warning about an abnormal body temperature, they can immediately rest the animal or consult a veterinarian.

[0445] Therefore, the present invention provides specific embodiments to support animal health management and emotional communication, and to enrich life with pets.

[0446] The following describes the processing flow.

[0447] Step 1:

[0448] The device (goggles) continuously photographs animals with a camera and collects video data. This makes it possible to record the animals' facial expressions and movements in real time.

[0449] Step 2:

[0450] The device (sensor-equipped pendant) monitors vital signs such as heart rate and body temperature of the animal it is attached to. It measures and stores data at regular intervals.

[0451] Step 3:

[0452] The terminal wirelessly transfers captured video data and acquired vital sign data to the server. Data transfer is continuous, maintaining real-time performance.

[0453] Step 4:

[0454] The server applies an AI model to analyze the received video data, using pattern recognition to analyze the animals' facial expressions and movements. This allows it to infer the animals' emotional state. For example, it can determine emotions such as "happy" from the way the tail wags or the movement of the ears.

[0455] Step 5:

[0456] The server assesses the animal's health status based on vital sign data. It compares these values ​​to baselines to determine if heart rate and body temperature are within the normal range. If an abnormality is detected, it generates an alert.

[0457] Step 6:

[0458] The server compiles the analysis results and sends them back to the terminal as feedback data. This feedback data includes information on emotional state, health status assessment, and warning messages as needed.

[0459] Step 7:

[0460] The device (goggles) receives feedback data from the server and presents it to the user as a visual display. This includes icons representing animal emotions and simple text information.

[0461] Step 8:

[0462] The device (smartphone app) notifies the user of their health status and alert information. This allows the user to immediately understand the situation and take necessary actions.

[0463] Step 9:

[0464] Users take appropriate action towards their animals based on the information provided. They can monitor their pet's emotions and continue playing, encourage rest if there are health issues, or consult a veterinarian.

[0465] (Example 1)

[0466] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0467] In animal-human communication, accurately understanding an animal's emotional state and health condition is difficult. In particular, interpreting an animal's facial expressions and movements and taking appropriate action relies heavily on human senses and experience. In such situations, when immediate action is required regarding an animal's health, judgment errors are likely to occur. Therefore, there is a need for a means to deepen mutual understanding between animals and humans and to more effectively manage animal health by analyzing the animal's emotions and health condition in real time and providing feedback to the user.

[0468] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0469] In this invention, the server includes imaging means for acquiring animal movements as images, analysis means for analyzing the image information using a generation AI model to determine the animal's emotional state, and evaluation means for monitoring the animal's biological information, comparing it with reference values, and detecting abnormalities. This enables real-time determination of the animal's emotional state and analysis of its health status.

[0470] "An imaging device for acquiring motion as an image" refers to a device or mechanism for capturing the motion of an animal and recording it as digital image data.

[0471] "Means of analysis using generative AI models" refers to a method or apparatus that uses a generative AI model, a type of machine learning, to analyze collected image data and infer the emotional state of an animal.

[0472] An "evaluation means for monitoring biological information, comparing it to reference values, and detecting abnormalities" is a device or mechanism for collecting biological information such as an animal's body temperature and heart rate, and determining abnormalities by comparing it to a predetermined normal range.

[0473] "Feedback means" refers to a method or apparatus for providing a user with the results of an analysis regarding an animal's emotions or health status, either visually or audibly.

[0474] "Visual display means" refers to a device or mechanism that communicates an animal's emotional state or health condition to a user in the form of textual information, icons, graphic elements, etc.

[0475] A "warning device" is a method or device for notifying the user when an abnormality in the health condition of an animal is detected.

[0476] This invention is a system that monitors the emotional and health status of animals in real time and provides appropriate feedback to the user. This system primarily consists of three components: a terminal, a server, and the user.

[0477] Device Operation: The device includes goggles and a sensor pendant. The goggles have a built-in high-resolution camera to capture images of the animal's movements, recording its facial expressions and actions as video. The sensor pendant also monitors the animal's biometric information, such as heart rate and body temperature, with high precision. This data is transmitted to the server in real time.

[0478] Server-based analysis: The server analyzes the video data transmitted from the terminal using a generating AI model. This AI model is trained on diverse animal data and can determine emotional states with high accuracy. Furthermore, the server compares biometric information with existing reference values ​​to assess health status and detect any abnormalities.

[0479] Providing Feedback: Analysis results are sent back from the server to the device and visually presented to the user through the goggles' display. The feedback includes text and icons indicating emotional states, as well as warnings, allowing the user to instantly understand the animal's condition. Users can also receive warnings, such as voice notifications, via a smartphone app.

[0480] User Interaction: Users can adjust their response to the animal based on the feedback provided by the system. For example, if the system indicates that "the dog is happy," they can extend playtime, or if it warns that "the dog's body temperature is higher than normal," they can take immediate action, such as letting the dog rest or consulting a veterinarian.

[0481] Examples of specific cases and prompt statements:

[0482] As a concrete example, consider a scenario where a user observes a flock of sheep at a zoo. Using this system, if it is determined that one of the sheep is experiencing stress, measures can be taken to improve its environment.

[0483] Examples of prompt messages are as follows:

[0484] "I want to know how a system works that analyzes my dog's emotional state in real time and provides support for health management as needed."

[0485] Thus, the present invention provides an effective system to enrich the interaction between animals and humans and to support the health and well-being of animals.

[0486] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0487] Step 1:

[0488] The device acquires the animal's movements and biometric information. The goggles' camera captures the animal's posture and facial expressions, and a sensor-equipped pendant measures heart rate and body temperature. The input for this step is the animal's visual and biometric information, and the output is the captured video data and acquired vital data.

[0489] Step 2:

[0490] The terminal sends acquired video data and vital data to the server. The data is sent to the server in real time using a secure and high-speed communication protocol. In this step, the acquired data is used as input to generate data packets that are transmitted to the server as output.

[0491] Step 3:

[0492] The server analyzes the video data using an AI model to determine the animal's emotional state. Specifically, it extracts features from the video, and the trained AI model infers emotions such as "happy" or "anxious." The input for this step is the video data sent from the terminal, and the output is the inferred emotional state.

[0493] Step 4:

[0494] The server evaluates the animal's health status by comparing vital data to reference values. If the data deviates from the normal range, it is judged as abnormal. The input for this step is vital data from the terminal, and the output is the judged health status and the evaluation result.

[0495] Step 5:

[0496] The server integrates the analyzed emotional and health states to generate feedback information. This feedback includes text indicating the emotional state and health-related warnings. The input for this step is the analysis results obtained in the previous step, and the output is the feedback information provided to the user.

[0497] Step 6:

[0498] The device receives feedback information provided by the server and presents it to the user visually or audibly. This may involve displaying text or icons on the goggles, or providing audio notifications via a smartphone app. The input for this step is the feedback information from the server, and the output is the presentation of this information as a notification to the user.

[0499] Step 7:

[0500] Based on the feedback information provided by the user, decisions are made regarding how to respond to the animal. For example, if there is a warning of a health problem, a decision may be made to consult a veterinarian. The input for this step is notification information from the device, and the output is the user's specific response.

[0501] (Application Example 1)

[0502] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0503] In workplaces where machinery and equipment are in operation, regular maintenance and inspections alone are insufficient to prevent all malfunctions, and unexpected stoppages or breakdowns can significantly impact production. Furthermore, constantly monitoring the status of machinery and equipment is a heavy burden for managers, highlighting the need for more efficient monitoring methods.

[0504] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0505] In this invention, the server includes imaging means for capturing images of the operation of a machine device, analysis means for analyzing the video data obtained by the imaging means and determining the operating state of the machine device, and monitoring means for monitoring the operating pattern and temperature of the machine device. This makes it possible to analyze the state of the machine device in real time and provide accurate and rapid feedback to the administrator.

[0506] "Machinery and equipment" refers to equipment or devices that automatically perform specific tasks in production sites and other similar settings.

[0507] "Imaging means" refers to a device or function that captures the movement of an object using a camera or similar device and acquires video data.

[0508] "Analysis means" refers to a function or device that uses acquired video data and sensor data to determine and analyze the state and movement of an object.

[0509] "Monitoring means" refers to devices or functions that continuously observe sensory data such as the movement patterns and temperature of an object, and collect data as needed.

[0510] "Evaluation means" refers to functions or devices that determine the operating state or abnormalities of an object based on sensing data and evaluate them by comparing them with a standard.

[0511] A "feedback mechanism" is a means of notifying users of the results of analysis and evaluation and providing them with necessary information.

[0512] The server constitutes a system for monitoring the operation of machinery and equipment installed within the factory, thereby enabling efficient and safe monitoring of the operating status of the machinery and equipment.

[0513] This system consists of multiple hardware components, primarily imaging devices, sensors, and analysis devices. The cameras capture images of the machinery's operation, acquiring video data in real time. The acquired video data is processed by an analysis device on a server to determine the machinery's operating status. The analysis device uses a generative AI model and is designed to detect operational anomalies. The analysis results are visually communicated to administrators via a feedback device, enabling them to take necessary actions quickly.

[0514] Sensors continuously record the temperature and vibration of machinery and transmit the data to a server. The server analyzes this data using evaluation equipment and compares it to baseline values ​​to proactively detect risks such as abnormal operation or overheating, and issues warnings. For example, if machinery vibrates beyond its normal operating pattern or if the temperature exceeds the designed tolerance range, the administrator is notified via the warning system.

[0515] As a concrete example, if a robotic arm operating in a factory begins to emit abnormal vibrations, this system will immediately detect the anomaly. A warning will be sent to the manager, allowing the robot to be stopped immediately, enabling investigation and rapid repair of the problem. An example of a prompt message to the generated AI model would be, "What kind of abnormal operation is there in this video frame?", prompting a specific analysis.

[0516] In this way, the integrated operation of the server and related devices ensures stable and efficient operation of the machinery and prevents production line shutdowns due to unexpected failures.

[0517] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0518] Step 1:

[0519] The server uses cameras installed within the factory to capture images of the machinery's operation. It acquires real-time video data as input and sends it to the server for processing. This video data records the machinery's movements and serves as the raw data for analysis.

[0520] Step 2:

[0521] The server processes the acquired video data using an analysis device. Using the video data obtained in step 1 as input, it performs data processing to detect operational anomalies using a generated AI model. The prompt message "What operational anomalies are present in this video frame?" is used to prompt the AI ​​for analysis. The output is a determination of whether the operating status of the machinery is abnormal or normal.

[0522] Step 3:

[0523] The server acquires operating patterns and temperature data from sensors attached to the machinery. It takes real-time sensor data as input and continuously records it through monitoring devices. This allows for the collection of detailed measurements regarding the operating status of the machinery.

[0524] Step 4:

[0525] The server analyzes the collected sensor data using an evaluation device. The operating pattern and temperature data obtained in step 3 are used as input. By comparing this data with reference values, it determines whether the machine is operating within acceptable limits. The output is a result indicating whether it is within normal limits or if there is an abnormality.

[0526] Step 5:

[0527] The server notifies the administrator of the analysis and evaluation results using a feedback device. It receives the judgment results obtained in steps 2 and 4 as input and presents them visually in a human-readable format. The output is displayed in real time through the management screen and warning alerts, allowing the administrator to quickly take necessary actions.

[0528] Step 6:

[0529] The user, or administrator, will check for any abnormalities in the machinery and equipment based on the information provided and take appropriate action. Based on the information obtained as output, they will, for example, issue emergency shutdown or repair instructions to ensure the stable operation of the system.

[0530] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0531] This invention is a system that simultaneously recognizes the emotional states of both the animal and the user (owner) and optimizes communication with the animal based on that recognition. This system mainly consists of three components: a terminal, a server, and the user.

[0532] First, the terminal is equipped with imaging capabilities to capture images of the animal's movements, continuously collecting the animal's facial expressions and actions. This data is transmitted to a server, where the server's analysis capabilities determine the animal's emotional state. Simultaneously, the terminal monitors the animal's vital signs through a sensor-equipped pendant, and the transmitted data is used by the server to assess the animal's health.

[0533] The server not only analyzes animal data but also recognizes the user's emotional state using an emotion engine. This includes methods for analyzing emotions from changes in the user's facial expressions and voice captured through the camera and microphone. The obtained user emotional state is compared with the animal's emotional state, and the correlation between the two is analyzed. Based on this analysis, the server generates advice and suggestions to optimize interaction with the pet and provides them to the user as feedback.

[0534] The device visually and audibly presents feedback received from the server to the user, informing them how to interact with their pet. It also notifies the user via an alert system if any abnormalities are detected in the animal's health. This allows users to respond quickly and use that information to improve their pet's care.

[0535] As a concrete example, consider a scenario where a user is playing with their pet. When the user's smile is detected through the goggles, the emotion engine determines that the user is "having fun." At the same time, the video data of the pet wagging its tail determines that the pet is "happy." In this situation, the server makes a suggestion such as "Please continue having fun," encouraging the user to continue the interaction.

[0536] This system allows users to understand their own emotions and those of their pets, enabling them to interact with their pets optimally based on that understanding. This invention offers a new form of relationship building that goes beyond conventional pet management.

[0537] The following describes the processing flow.

[0538] Step 1:

[0539] The device (goggles) captures the animal's movements with a camera and collects video data in real time. This data includes the animal's facial expressions and movement characteristics.

[0540] Step 2:

[0541] The device (a pendant with a sensor) continuously monitors and collects data on the animal's vital signs, such as heart rate and body temperature. This data is necessary to understand the animal's health status.

[0542] Step 3:

[0543] The server receives video data transmitted from the terminal and analyzes it using an AI model to determine the animal's emotional state. This allows it to determine whether the animal is feeling emotions such as "happy" or "calm."

[0544] Step 4:

[0545] The server analyzes vital sign data and assesses the animal's health status. If the assessment results exceed the standard values, the server determines it to be abnormal and generates information about it.

[0546] Step 5:

[0547] The device uses its camera and microphone to collect facial and audio data from the user in order to recognize the user's emotional state. This information is essential for interpreting the user's reactions and emotions.

[0548] Step 6:

[0549] The server analyzes the user's emotional data using an emotion engine to determine the user's current emotional state. For example, if a smile is detected, it is determined that the user is "having fun."

[0550] Step 7:

[0551] The server compares the emotional states of the animals and the users and analyzes their correlation. Simultaneously, it generates suggestions for optimizing the interaction based on the animals' health status and the users' emotional states.

[0552] Step 8:

[0553] The device displays feedback from the server to the user. This feedback includes information about the animal's emotions and health, as well as advice on what to do. This allows the user to take appropriate action.

[0554] Step 9:

[0555] Users interact with their pets based on information provided through the device's interface. If necessary, they can take appropriate action in response to health alerts. For example, if a health problem is detected, they can allow their pet to rest or consult a veterinarian.

[0556] (Example 2)

[0557] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0558] Conventional animal communication systems were unable to simultaneously analyze the emotional state of the animal and the emotions of the user, making it difficult to optimize interactions based on mutual emotions. Furthermore, there was a lack of systems that could quickly notify the user of abnormalities in the animal's health, making it impossible to provide appropriate feedback to deepen the relationship with the animal.

[0559] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0560] In this invention, the server includes video acquisition means for capturing images of animal movements, data analysis means for analyzing animal data, state evaluation means for evaluating health status based on biological information, user analysis means for analyzing the user's facial expressions and voice, correlation analysis means for comparing the emotional states of the animal and the user, and information presentation means for providing the analysis results to the user. This makes it possible to optimize interaction based on the emotions of both the animal and the user, and to quickly provide health warnings as needed.

[0561] "Image acquisition means" refers to a hardware and software system for capturing images of an animal's movements and facial expressions and acquiring that image data.

[0562] "Data analysis means" refers to a function that includes an algorithm for analyzing acquired video data and determining the emotional state of an animal.

[0563] "Indicator collection means" refers to equipment and systems for monitoring biological information such as the heart rate and body temperature of animals and continuously collecting data.

[0564] A "condition evaluation method" is a technology that performs analysis and makes judgments to evaluate the health status of an animal based on collected biological information.

[0565] "User analysis methods" refer to technologies that collect and analyze a user's facial expressions and voice to determine the user's emotional state.

[0566] A "correlation analysis method" is a method for comparing the emotional state of an animal with the emotional state of a user and analyzing their interrelationships.

[0567] "Information presentation means" refers to methods that utilize display devices and output devices to provide users with analysis results and suggestions visually or audibly.

[0568] The "warning function" is a mechanism that sends a warning to the user when it detects an abnormality in the animal's health condition.

[0569] This invention is a system that simultaneously recognizes the emotional states of both animals and users and optimizes their interactions. This system is primarily operated by a terminal, a server, and the user.

[0570] The terminal uses a camera as a means of acquiring images to capture the animal's movements. This camera continuously records the animal's facial expressions and movements, and acquires the data. Furthermore, the terminal also includes an indicator collection means for collecting biometric information, which is a pendant equipped with a sensor. This pendant monitors the animal's heart rate, body temperature, and other parameters. This data is transmitted from the terminal to a server.

[0571] The server processes the received data using data analysis tools to determine the animal's emotional state. It analyzes the state using specific algorithms based on the animal's facial expressions and movement data. Furthermore, a state evaluation tool is used to assess the animal's health based on biological information.

[0572] In addition, the device uses a camera and microphone as user analysis tools to analyze the user's emotions. This collects changes in the user's facial expressions and tone of voice, which are then analyzed on a server. As a result, the server compares the emotional states of the animal and the user using correlation analysis tools to confirm the relationship between their emotions.

[0573] Ultimately, the server provides feedback to the user through information presentation methods based on these analysis results. For example, if a smile is detected while the user is playing with their pet and the animal is wagging its tail, the server will generate and provide a suggestion such as "Let's continue having fun." In this way, the user can receive information to optimize their interaction with their pet.

[0574] The advantage of this system lies in its ability to comprehensively analyze the emotional states of both the user and the pet, deepening their understanding based on their respective emotions. An example of a prompt might be, "Read the pet's emotions from its behavioral patterns and compare them with the user's facial expressions and voice to provide the best suggestion." This prompt enables the generative AI model to provide an advanced communication tool based on emotional understanding.

[0575] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0576] Step 1:

[0577] The device uses a camera to capture images of animal movements, recording the animal's facial expressions and actions in real time. It acquires camera video data as input and continuously streams that video. The output obtained from this process is video data related to the animal's movements.

[0578] Step 2:

[0579] The device uses a sensor-equipped pendant to collect animal biometric information. Inputs include monitoring vital signs such as the animal's heart rate and body temperature. Outputs provide data related to the animal's health status.

[0580] Step 3:

[0581] The terminal transmits collected video and biometric data to the server. This data is acquired from the terminal as input and rapidly transferred to the server. The output consists of the animals' emotional state and health information, awaiting analysis.

[0582] Step 4:

[0583] The server processes the received video data using data analysis tools. Using the video data as input, it analyzes the animal's facial expressions and movements and executes an algorithm to determine its emotional state. The output is the animal's specific emotional state (e.g., "happy" or "anxious").

[0584] Step 5:

[0585] The server evaluates the health status of animals based on collected biometric data. It uses biometric data as input and performs an evaluation by comparing it to normal health standards. The output is the evaluation result, indicating whether the animal is healthy or not.

[0586] Step 6:

[0587] The device uses a camera and microphone to collect the user's facial expressions and voice. It takes the user's facial expression changes and voice tone as input, and outputs data on the user's emotional state.

[0588] Step 7:

[0589] The server uses correlation analysis to analyze the user's emotional state and compare it to the animal's emotional state. It takes the emotional data of both as input and analyzes their relationship. The output is an evaluation of the emotional correlation between the animal and the user.

[0590] Step 8:

[0591] Based on the analysis results obtained, the server generates suggestions for how the user should interact with animals. All analysis results are used as input, and the information presentation means creates appropriate feedback using a generating AI model. The output is a specific suggestion, such as "Let's continue playing."

[0592] Step 9:

[0593] The terminal presents suggestions from the server to the user visually and audibly. Input is feedback data from the server, which is presented through the user's display or speaker. Output is specific advice or warnings received by the user.

[0594] (Application Example 2)

[0595] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0596] In modern brick-and-mortar stores, improving services for customers who bring their pets is crucial, but it is difficult to understand the emotional state of both the customer and their pet in real time and provide appropriate services based on that understanding. In particular, there is a need to accurately understand the health and emotional state of pets and provide interactions accordingly, but conventional technologies do not adequately address this.

[0597] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0598] In this invention, the server includes means for a camera that captures images of an animal's movements, means for an analysis function that analyzes the video information obtained by the camera to determine the animal's emotional state, and means for an information presentation function that grasps the emotional state of users and pets in the store in real time and provides information to promote interaction between the users and pets. This makes it possible to provide appropriate services in physical stores based on the emotional state of customers and pets.

[0599] A "filming device" is a device used to record the movements of animals as video, and has the function of providing video information.

[0600] The "analysis function" is a function that processes video information obtained from the camera to determine the emotional state of the animal.

[0601] The "monitoring function" is a function that continuously observes an animal's vital information and manages its health status.

[0602] The "evaluation function" is a function that evaluates the health status of an animal based on vital information obtained through the monitoring function.

[0603] A "notification device" is a device that informs users of the analysis results obtained through analysis and evaluation functions.

[0604] The "information presentation function" is a function that analyzes the emotional state of customers and their pets in real time within the store and provides appropriate information to customers based on the results.

[0605] The "alarm function" is a feature that warns the user if an abnormality is detected in the animal's health condition.

[0606] In order to implement this invention, a system is needed that uses various hardware and software to efficiently analyze the emotional states of animals and users, and to optimize interactions in physical stores.

[0607] The server manages the imaging equipment used to capture images of animal movements and analyzes the captured video information. The software used includes OpenCV for video analysis and emotion analysis engines such as Hume AI for emotional state analysis. The server also acquires vital information from the animals, which is collected via medical sensors. This vital information is processed using cloud platforms such as Google Cloud.

[0608] The server provides store staff with real-time information based on the analyzed data. The information display function alerts customers about what services are appropriate in the store based on the emotional state of the user and their pet. For example, if a pet is detected as restless, the server will recommend relaxation items to the store staff.

[0609] For example, when a customer visits a store with their dog, the camera may capture the customer's smile and the movement of their pet's tail as they walk through the store. Based on this information, the server can provide feedback to the staff, such as, "That looks like a fun time. Please continue." If any abnormalities are detected in the animal's health or emotional state, a warning function allows for quick countermeasures to be taken.

[0610] By utilizing generative AI models, appropriate prompt messages can be created. Examples of prompt messages include, "Please diagnose this person's emotions using the facial expression data obtained from the camera," and "Identify the dog's emotions from this image and indicate them as either 'happy,' 'sad,' or 'nervous.'" This enables accurate emotion analysis, making in-store services more personalized.

[0611] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0612] Step 1:

[0613] The terminal uses a camera to visualize animal movements and acquire video information. The input is the animal's movements, and the output is video data. This video data is used for subsequent analysis.

[0614] Step 2:

[0615] The server uses OpenCV to analyze the video data sent from the terminal. In this step, the animal's facial expressions and movements are broken down frame by frame, and the emotional state for each frame is determined. The input is the video data obtained in step 1, and the output is data indicating the animal's emotions.

[0616] Step 3:

[0617] The device collects vital information from animals via medical sensors. The input is the animal's biological information, and the output is vital data. This data is monitored in real time.

[0618] Step 4:

[0619] The server processes vital data using the Google Cloud platform. In this step, the health status of the animals is assessed and any abnormalities are detected. The input is the vital data collected in step 3, and the output is health assessment data.

[0620] Step 5:

[0621] The server also analyzes the user's emotional state using an emotion analysis engine such as Hume AI. In this step, the user's facial expressions and voice collected through the camera and microphone are analyzed. The input is the user's facial expressions and voice data, and the output is the user's emotional state.

[0622] Step 6:

[0623] The server integrates the emotional states of animals and users and uses information presentation functions to provide appropriate service strategies to store staff in real time. The input is emotional state data from steps 2 and 5, and the output is service suggestion information. This result is displayed on the staff interface.

[0624] Step 7:

[0625] The server uses an alarm function to warn the user if an anomaly is detected. The input is the health assessment data from step 4, and the output is the warning information. This allows for a rapid response.

[0626] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0627] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0628] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0629] [Fourth Embodiment]

[0630] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0631] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0632] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0633] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0634] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0635] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0636] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0637] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0638] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0639] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0640] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0641] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0642] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0643] This invention is a system that monitors the emotional and health status of animals in real time and provides visual or audible feedback to the user. The system mainly consists of three components: a terminal (goggles and a sensor-equipped pendant), a server, and the user.

[0644] First, the goggles used as terminals have a built-in camera to capture images of the animals' movements. While observing the animals, the goggles capture their facial expressions and movements, recording them as video data. In addition, a sensor-equipped pendant is attached to the animals' bodies to acquire vital sign data such as heart rate and body temperature. This data is transmitted to a server via the goggles.

[0645] The server analyzes the received video data using an AI model to determine the animal's emotional state. For example, if a dog is wagging its tail, it infers that the animal is "happy" or "excited." The server also analyzes vital sign data and compares it to reference values ​​to assess the animal's health and determine if there are any abnormalities. For example, if the body temperature is higher than normal, it will be detected as an abnormality.

[0646] These analysis results are fed back from the server to the terminal. The goggles have a function to visually display the received data, showing the user the animal's emotions and intentions in real time. The display may include text information, icons, and graphic elements. In addition, the smartphone app notifies the user of health status reports and warnings based on vital sign information sent from the pendant.

[0647] Based on this feedback, users can understand the emotional and physical state of their animals and take appropriate action. For example, if they know their dog is "happy," they can continue playing with it, or if they receive a warning about an abnormal body temperature, they can immediately rest the animal or consult a veterinarian.

[0648] Therefore, the present invention provides specific embodiments to support animal health management and emotional communication, and to enrich life with pets.

[0649] The following describes the processing flow.

[0650] Step 1:

[0651] The device (goggles) continuously photographs animals with a camera and collects video data. This makes it possible to record the animals' facial expressions and movements in real time.

[0652] Step 2:

[0653] The device (sensor-equipped pendant) monitors vital signs such as heart rate and body temperature of the animal it is attached to. It measures and stores data at regular intervals.

[0654] Step 3:

[0655] The terminal wirelessly transfers captured video data and acquired vital sign data to the server. Data transfer is continuous, maintaining real-time performance.

[0656] Step 4:

[0657] The server applies an AI model to analyze the received video data, using pattern recognition to analyze the animals' facial expressions and movements. This allows it to infer the animals' emotional state. For example, it can determine emotions such as "happy" from the way the tail wags or the movement of the ears.

[0658] Step 5:

[0659] The server assesses the animal's health status based on vital sign data. It compares these values ​​to baselines to determine if heart rate and body temperature are within the normal range. If an abnormality is detected, it generates an alert.

[0660] Step 6:

[0661] The server compiles the analysis results and sends them back to the terminal as feedback data. This feedback data includes information on emotional state, health status assessment, and warning messages as needed.

[0662] Step 7:

[0663] The device (goggles) receives feedback data from the server and presents it to the user as a visual display. This includes icons representing animal emotions and simple text information.

[0664] Step 8:

[0665] The device (smartphone app) notifies the user of their health status and alert information. This allows the user to immediately understand the situation and take necessary actions.

[0666] Step 9:

[0667] Users take appropriate action towards their animals based on the information provided. They can monitor their pet's emotions and continue playing, encourage rest if there are health issues, or consult a veterinarian.

[0668] (Example 1)

[0669] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0670] In animal-human communication, accurately understanding an animal's emotional state and health condition is difficult. In particular, interpreting an animal's facial expressions and movements and taking appropriate action relies heavily on human senses and experience. In such situations, when immediate action is required regarding an animal's health, judgment errors are likely to occur. Therefore, there is a need for a means to deepen mutual understanding between animals and humans and to more effectively manage animal health by analyzing the animal's emotions and health condition in real time and providing feedback to the user.

[0671] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0672] In this invention, the server includes imaging means for acquiring animal movements as images, analysis means for analyzing the image information using a generation AI model to determine the animal's emotional state, and evaluation means for monitoring the animal's biological information, comparing it with reference values, and detecting abnormalities. This enables real-time determination of the animal's emotional state and analysis of its health status.

[0673] "An imaging device for acquiring motion as an image" refers to a device or mechanism for capturing the motion of an animal and recording it as digital image data.

[0674] "Means of analysis using generative AI models" refers to a method or apparatus that uses a generative AI model, a type of machine learning, to analyze collected image data and infer the emotional state of an animal.

[0675] An "evaluation means for monitoring biological information, comparing it to reference values, and detecting abnormalities" is a device or mechanism for collecting biological information such as an animal's body temperature and heart rate, and determining abnormalities by comparing it to a predetermined normal range.

[0676] "Feedback means" refers to a method or apparatus for providing a user with the results of an analysis regarding an animal's emotions or health status, either visually or audibly.

[0677] "Visual display means" refers to a device or mechanism that communicates an animal's emotional state or health condition to a user in the form of textual information, icons, graphic elements, etc.

[0678] A "warning device" is a method or device for notifying the user when an abnormality in the health condition of an animal is detected.

[0679] This invention is a system that monitors the emotional and health status of animals in real time and provides appropriate feedback to the user. This system primarily consists of three components: a terminal, a server, and the user.

[0680] Device Operation: The device includes goggles and a sensor pendant. The goggles have a built-in high-resolution camera to capture images of the animal's movements, recording its facial expressions and actions as video. The sensor pendant also monitors the animal's biometric information, such as heart rate and body temperature, with high precision. This data is transmitted to the server in real time.

[0681] Server-based analysis: The server analyzes the video data transmitted from the terminal using a generating AI model. This AI model is trained on diverse animal data and can determine emotional states with high accuracy. Furthermore, the server compares biometric information with existing reference values ​​to assess health status and detect any abnormalities.

[0682] Providing Feedback: Analysis results are sent back from the server to the device and visually presented to the user through the goggles' display. The feedback includes text and icons indicating emotional states, as well as warnings, allowing the user to instantly understand the animal's condition. Users can also receive warnings, such as voice notifications, via a smartphone app.

[0683] User Interaction: Users can adjust their response to the animal based on the feedback provided by the system. For example, if the system indicates that "the dog is happy," they can extend playtime, or if it warns that "the dog's body temperature is higher than normal," they can take immediate action, such as letting the dog rest or consulting a veterinarian.

[0684] Examples of specific cases and prompt statements:

[0685] As a concrete example, consider a scenario where a user observes a flock of sheep at a zoo. Using this system, if it is determined that one of the sheep is experiencing stress, measures can be taken to improve its environment.

[0686] Examples of prompt messages are as follows:

[0687] "I want to know how a system works that analyzes my dog's emotional state in real time and provides support for health management as needed."

[0688] Thus, the present invention provides an effective system to enrich the interaction between animals and humans and to support the health and well-being of animals.

[0689] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0690] Step 1:

[0691] The device acquires the animal's movements and biometric information. The goggles' camera captures the animal's posture and facial expressions, and a sensor-equipped pendant measures heart rate and body temperature. The input for this step is the animal's visual and biometric information, and the output is the captured video data and acquired vital data.

[0692] Step 2:

[0693] The terminal sends acquired video data and vital data to the server. The data is sent to the server in real time using a secure and high-speed communication protocol. In this step, the acquired data is used as input to generate data packets that are transmitted to the server as output.

[0694] Step 3:

[0695] The server analyzes the video data using an AI model to determine the animal's emotional state. Specifically, it extracts features from the video, and the trained AI model infers emotions such as "happy" or "anxious." The input for this step is the video data sent from the terminal, and the output is the inferred emotional state.

[0696] Step 4:

[0697] The server evaluates the animal's health status by comparing vital data to reference values. If the data deviates from the normal range, it is judged as abnormal. The input for this step is vital data from the terminal, and the output is the judged health status and the evaluation result.

[0698] Step 5:

[0699] The server integrates the analyzed emotional and health states to generate feedback information. This feedback includes text indicating the emotional state and health-related warnings. The input for this step is the analysis results obtained in the previous step, and the output is the feedback information provided to the user.

[0700] Step 6:

[0701] The device receives feedback information provided by the server and presents it to the user visually or audibly. This may involve displaying text or icons on the goggles, or providing audio notifications via a smartphone app. The input for this step is the feedback information from the server, and the output is the presentation of this information as a notification to the user.

[0702] Step 7:

[0703] Based on the feedback information provided by the user, decisions are made regarding how to respond to the animal. For example, if there is a warning of a health problem, a decision may be made to consult a veterinarian. The input for this step is notification information from the device, and the output is the user's specific response.

[0704] (Application Example 1)

[0705] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0706] In workplaces where machinery and equipment are in operation, regular maintenance and inspections alone are insufficient to prevent all malfunctions, and unexpected stoppages or breakdowns can significantly impact production. Furthermore, constantly monitoring the status of machinery and equipment is a heavy burden for managers, highlighting the need for more efficient monitoring methods.

[0707] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0708] In this invention, the server includes imaging means for capturing images of the operation of a machine device, analysis means for analyzing the video data obtained by the imaging means and determining the operating state of the machine device, and monitoring means for monitoring the operating pattern and temperature of the machine device. This makes it possible to analyze the state of the machine device in real time and provide accurate and rapid feedback to the administrator.

[0709] "Machinery and equipment" refers to equipment or devices that automatically perform specific tasks in production sites and other similar settings.

[0710] "Imaging means" refers to a device or function that captures the movement of an object using a camera or similar device and acquires video data.

[0711] "Analysis means" refers to a function or device that uses acquired video data and sensor data to determine and analyze the state and movement of an object.

[0712] "Monitoring means" refers to devices or functions that continuously observe sensory data such as the movement patterns and temperature of an object, and collect data as needed.

[0713] "Evaluation means" refers to functions or devices that determine the operating state or abnormalities of an object based on sensing data and evaluate them by comparing them with a standard.

[0714] A "feedback mechanism" is a means of notifying users of the results of analysis and evaluation and providing them with necessary information.

[0715] The server constitutes a system for monitoring the operation of machinery and equipment installed within the factory, thereby enabling efficient and safe monitoring of the operating status of the machinery and equipment.

[0716] This system consists of multiple hardware components, primarily imaging devices, sensors, and analysis devices. The cameras capture images of the machinery's operation, acquiring video data in real time. The acquired video data is processed by an analysis device on a server to determine the machinery's operating status. The analysis device uses a generative AI model and is designed to detect operational anomalies. The analysis results are visually communicated to administrators via a feedback device, enabling them to take necessary actions quickly.

[0717] Sensors continuously record the temperature and vibration of machinery and transmit the data to a server. The server analyzes this data using evaluation equipment and compares it to baseline values ​​to proactively detect risks such as abnormal operation or overheating, and issues warnings. For example, if machinery vibrates beyond its normal operating pattern or if the temperature exceeds the designed tolerance range, the administrator is notified via the warning system.

[0718] As a concrete example, if a robotic arm operating in a factory begins to emit abnormal vibrations, this system will immediately detect the anomaly. A warning will be sent to the manager, allowing the robot to be stopped immediately, enabling investigation and rapid repair of the problem. An example of a prompt message to the generated AI model would be, "What kind of abnormal operation is there in this video frame?", prompting a specific analysis.

[0719] In this way, the integrated operation of the server and related devices ensures stable and efficient operation of the machinery and prevents production line shutdowns due to unexpected failures.

[0720] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0721] Step 1:

[0722] The server uses cameras installed within the factory to capture images of the machinery's operation. It acquires real-time video data as input and sends it to the server for processing. This video data records the machinery's movements and serves as the raw data for analysis.

[0723] Step 2:

[0724] The server processes the acquired video data using an analysis device. Using the video data obtained in step 1 as input, it performs data processing to detect operational anomalies using a generated AI model. The prompt message "What operational anomalies are present in this video frame?" is used to prompt the AI ​​for analysis. The output is a determination of whether the operating status of the machinery is abnormal or normal.

[0725] Step 3:

[0726] The server acquires operating patterns and temperature data from sensors attached to the machinery. It takes real-time sensor data as input and continuously records it through monitoring devices. This allows for the collection of detailed measurements regarding the operating status of the machinery.

[0727] Step 4:

[0728] The server analyzes the collected sensor data using an evaluation device. The operating pattern and temperature data obtained in step 3 are used as input. By comparing this data with reference values, it determines whether the machine is operating within acceptable limits. The output is a result indicating whether it is within normal limits or if there is an abnormality.

[0729] Step 5:

[0730] The server notifies the administrator of the analysis and evaluation results using a feedback device. It receives the judgment results obtained in steps 2 and 4 as input and presents them visually in a human-readable format. The output is displayed in real time through the management screen and warning alerts, allowing the administrator to quickly take necessary actions.

[0731] Step 6:

[0732] The user, or administrator, will check for any abnormalities in the machinery and equipment based on the information provided and take appropriate action. Based on the information obtained as output, they will, for example, issue emergency shutdown or repair instructions to ensure the stable operation of the system.

[0733] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0734] This invention is a system that simultaneously recognizes the emotional states of both the animal and the user (owner) and optimizes communication with the animal based on that recognition. This system mainly consists of three components: a terminal, a server, and the user.

[0735] First, the terminal is equipped with imaging capabilities to capture images of the animal's movements, continuously collecting the animal's facial expressions and actions. This data is transmitted to a server, where the server's analysis capabilities determine the animal's emotional state. Simultaneously, the terminal monitors the animal's vital signs through a sensor-equipped pendant, and the transmitted data is used by the server to assess the animal's health.

[0736] The server not only analyzes animal data but also recognizes the user's emotional state using an emotion engine. This includes methods for analyzing emotions from changes in the user's facial expressions and voice captured through the camera and microphone. The obtained user emotional state is compared with the animal's emotional state, and the correlation between the two is analyzed. Based on this analysis, the server generates advice and suggestions to optimize interaction with the pet and provides them to the user as feedback.

[0737] The device visually and audibly presents feedback received from the server to the user, informing them how to interact with their pet. It also notifies the user via an alert system if any abnormalities are detected in the animal's health. This allows users to respond quickly and use that information to improve their pet's care.

[0738] As a concrete example, consider a scenario where a user is playing with their pet. When the user's smile is detected through the goggles, the emotion engine determines that the user is "having fun." At the same time, the video data of the pet wagging its tail determines that the pet is "happy." In this situation, the server makes a suggestion such as "Please continue having fun," encouraging the user to continue the interaction.

[0739] This system allows users to understand their own emotions and those of their pets, enabling them to interact with their pets optimally based on that understanding. This invention offers a new form of relationship building that goes beyond conventional pet management.

[0740] The following describes the processing flow.

[0741] Step 1:

[0742] The device (goggles) captures the animal's movements with a camera and collects video data in real time. This data includes the animal's facial expressions and movement characteristics.

[0743] Step 2:

[0744] The device (a pendant with a sensor) continuously monitors and collects data on the animal's vital signs, such as heart rate and body temperature. This data is necessary to understand the animal's health status.

[0745] Step 3:

[0746] The server receives video data transmitted from the terminal and analyzes it using an AI model to determine the animal's emotional state. This allows it to determine whether the animal is feeling emotions such as "happy" or "calm."

[0747] Step 4:

[0748] The server analyzes vital sign data and assesses the animal's health status. If the assessment results exceed the standard values, the server determines it to be abnormal and generates information about it.

[0749] Step 5:

[0750] The device uses its camera and microphone to collect facial and audio data from the user in order to recognize the user's emotional state. This information is essential for interpreting the user's reactions and emotions.

[0751] Step 6:

[0752] The server analyzes the user's emotional data using an emotion engine to determine the user's current emotional state. For example, if a smile is detected, it is determined that the user is "having fun."

[0753] Step 7:

[0754] The server compares the emotional states of the animals and the users and analyzes their correlation. Simultaneously, it generates suggestions for optimizing the interaction based on the animals' health status and the users' emotional states.

[0755] Step 8:

[0756] The device displays feedback from the server to the user. This feedback includes information about the animal's emotions and health, as well as advice on what to do. This allows the user to take appropriate action.

[0757] Step 9:

[0758] Users interact with their pets based on information provided through the device's interface. If necessary, they can take appropriate action in response to health alerts. For example, if a health problem is detected, they can allow their pet to rest or consult a veterinarian.

[0759] (Example 2)

[0760] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0761] Conventional animal communication systems were unable to simultaneously analyze the emotional state of the animal and the emotions of the user, making it difficult to optimize interactions based on mutual emotions. Furthermore, there was a lack of systems that could quickly notify the user of abnormalities in the animal's health, making it impossible to provide appropriate feedback to deepen the relationship with the animal.

[0762] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0763] In this invention, the server includes video acquisition means for capturing images of animal movements, data analysis means for analyzing animal data, state evaluation means for evaluating health status based on biological information, user analysis means for analyzing the user's facial expressions and voice, correlation analysis means for comparing the emotional states of the animal and the user, and information presentation means for providing the analysis results to the user. This makes it possible to optimize interaction based on the emotions of both the animal and the user, and to quickly provide health warnings as needed.

[0764] "Image acquisition means" refers to a hardware and software system for capturing images of an animal's movements and facial expressions and acquiring that image data.

[0765] "Data analysis means" refers to a function that includes an algorithm for analyzing acquired video data and determining the emotional state of an animal.

[0766] "Indicator collection means" refers to equipment and systems for monitoring biological information such as the heart rate and body temperature of animals and continuously collecting data.

[0767] A "condition evaluation method" is a technology that performs analysis and makes judgments to evaluate the health status of an animal based on collected biological information.

[0768] "User analysis methods" refer to technologies that collect and analyze a user's facial expressions and voice to determine the user's emotional state.

[0769] A "correlation analysis method" is a method for comparing the emotional state of an animal with the emotional state of a user and analyzing their interrelationships.

[0770] "Information presentation means" refers to methods that utilize display devices and output devices to provide users with analysis results and suggestions visually or audibly.

[0771] The "warning function" is a mechanism that sends a warning to the user when it detects an abnormality in the animal's health condition.

[0772] This invention is a system that simultaneously recognizes the emotional states of both animals and users and optimizes their interactions. This system is primarily operated by a terminal, a server, and the user.

[0773] The terminal uses a camera as a means of acquiring images to capture the animal's movements. This camera continuously records the animal's facial expressions and movements, and acquires the data. Furthermore, the terminal also includes an indicator collection means for collecting biometric information, which is a pendant equipped with a sensor. This pendant monitors the animal's heart rate, body temperature, and other parameters. This data is transmitted from the terminal to a server.

[0774] The server processes the received data using data analysis tools to determine the animal's emotional state. It analyzes the state using specific algorithms based on the animal's facial expressions and movement data. Furthermore, a state evaluation tool is used to assess the animal's health based on biological information.

[0775] In addition, the device uses a camera and microphone as user analysis tools to analyze the user's emotions. This collects changes in the user's facial expressions and tone of voice, which are then analyzed on a server. As a result, the server compares the emotional states of the animal and the user using correlation analysis tools to confirm the relationship between their emotions.

[0776] Ultimately, the server provides feedback to the user through information presentation methods based on these analysis results. For example, if a smile is detected while the user is playing with their pet and the animal is wagging its tail, the server will generate and provide a suggestion such as "Let's continue having fun." In this way, the user can receive information to optimize their interaction with their pet.

[0777] The advantage of this system lies in its ability to comprehensively analyze the emotional states of both the user and the pet, deepening their understanding based on their respective emotions. An example of a prompt might be, "Read the pet's emotions from its behavioral patterns and compare them with the user's facial expressions and voice to provide the best suggestion." This prompt enables the generative AI model to provide an advanced communication tool based on emotional understanding.

[0778] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0779] Step 1:

[0780] The device uses a camera to capture images of animal movements, recording the animal's facial expressions and actions in real time. It acquires camera video data as input and continuously streams that video. The output obtained from this process is video data related to the animal's movements.

[0781] Step 2:

[0782] The device uses a sensor-equipped pendant to collect animal biometric information. Inputs include monitoring vital signs such as the animal's heart rate and body temperature. Outputs provide data related to the animal's health status.

[0783] Step 3:

[0784] The terminal transmits collected video and biometric data to the server. This data is acquired from the terminal as input and rapidly transferred to the server. The output consists of the animals' emotional state and health information, awaiting analysis.

[0785] Step 4:

[0786] The server processes the received video data using data analysis tools. Using the video data as input, it analyzes the animal's facial expressions and movements and executes an algorithm to determine its emotional state. The output is the animal's specific emotional state (e.g., "happy" or "anxious").

[0787] Step 5:

[0788] The server evaluates the health status of animals based on collected biometric data. It uses biometric data as input and performs an evaluation by comparing it to normal health standards. The output is the evaluation result, indicating whether the animal is healthy or not.

[0789] Step 6:

[0790] The device uses a camera and microphone to collect the user's facial expressions and voice. It takes the user's facial expression changes and voice tone as input, and outputs data on the user's emotional state.

[0791] Step 7:

[0792] The server uses correlation analysis to analyze the user's emotional state and compare it to the animal's emotional state. It takes the emotional data of both as input and analyzes their relationship. The output is an evaluation of the emotional correlation between the animal and the user.

[0793] Step 8:

[0794] Based on the analysis results obtained, the server generates suggestions for how the user should interact with animals. All analysis results are used as input, and the information presentation means creates appropriate feedback using a generating AI model. The output is a specific suggestion, such as "Let's continue playing."

[0795] Step 9:

[0796] The terminal presents suggestions from the server to the user visually and audibly. Input is feedback data from the server, which is presented through the user's display or speaker. Output is specific advice or warnings received by the user.

[0797] (Application Example 2)

[0798] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0799] In modern brick-and-mortar stores, improving services for customers who bring their pets is crucial, but it is difficult to understand the emotional state of both the customer and their pet in real time and provide appropriate services based on that understanding. In particular, there is a need to accurately understand the health and emotional state of pets and provide interactions accordingly, but conventional technologies do not adequately address this.

[0800] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0801] In this invention, the server includes means for a camera that captures images of an animal's movements, means for an analysis function that analyzes the video information obtained by the camera to determine the animal's emotional state, and means for an information presentation function that grasps the emotional state of users and pets in the store in real time and provides information to promote interaction between the users and pets. This makes it possible to provide appropriate services in physical stores based on the emotional state of customers and pets.

[0802] A "filming device" is a device used to record the movements of animals as video, and has the function of providing video information.

[0803] The "analysis function" is a function that processes video information obtained from the camera to determine the emotional state of the animal.

[0804] The "monitoring function" is a function that continuously observes an animal's vital information and manages its health status.

[0805] The "evaluation function" is a function that evaluates the health status of an animal based on vital information obtained through the monitoring function.

[0806] A "notification device" is a device that informs users of the analysis results obtained through analysis and evaluation functions.

[0807] The "information presentation function" is a function that analyzes the emotional state of customers and their pets in real time within the store and provides appropriate information to customers based on the results.

[0808] The "alarm function" is a feature that warns the user if an abnormality is detected in the animal's health condition.

[0809] In order to implement this invention, a system is needed that uses various hardware and software to efficiently analyze the emotional states of animals and users, and to optimize interactions in physical stores.

[0810] The server manages the imaging equipment used to capture images of animal movements and analyzes the captured video information. The software used includes OpenCV for video analysis and emotion analysis engines such as Hume AI for emotional state analysis. The server also acquires vital information from the animals, which is collected via medical sensors. This vital information is processed using cloud platforms such as Google Cloud.

[0811] The server provides store staff with real-time information based on the analyzed data. The information display function alerts customers about what services are appropriate in the store based on the emotional state of the user and their pet. For example, if a pet is detected as restless, the server will recommend relaxation items to the store staff.

[0812] For example, when a customer visits a store with their dog, the camera may capture the customer's smile and the movement of their pet's tail as they walk through the store. Based on this information, the server can provide feedback to the staff, such as, "That looks like a fun time. Please continue." If any abnormalities are detected in the animal's health or emotional state, a warning function allows for quick countermeasures to be taken.

[0813] By utilizing generative AI models, appropriate prompt messages can be created. Examples of prompt messages include, "Please diagnose this person's emotions using the facial expression data obtained from the camera," and "Identify the dog's emotions from this image and indicate them as either 'happy,' 'sad,' or 'nervous.'" This enables accurate emotion analysis, making in-store services more personalized.

[0814] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0815] Step 1:

[0816] The terminal uses a camera to visualize animal movements and acquire video information. The input is the animal's movements, and the output is video data. This video data is used for subsequent analysis.

[0817] Step 2:

[0818] The server uses OpenCV to analyze the video data sent from the terminal. In this step, the animal's facial expressions and movements are broken down frame by frame, and the emotional state for each frame is determined. The input is the video data obtained in step 1, and the output is data indicating the animal's emotions.

[0819] Step 3:

[0820] The device collects vital information from animals via medical sensors. The input is the animal's biological information, and the output is vital data. This data is monitored in real time.

[0821] Step 4:

[0822] The server processes vital data using the Google Cloud platform. In this step, the health status of the animals is assessed and any abnormalities are detected. The input is the vital data collected in step 3, and the output is health assessment data.

[0823] Step 5:

[0824] The server also analyzes the user's emotional state using an emotion analysis engine such as Hume AI. In this step, the user's facial expressions and voice collected through the camera and microphone are analyzed. The input is the user's facial expressions and voice data, and the output is the user's emotional state.

[0825] Step 6:

[0826] The server integrates the emotional states of animals and users and uses information presentation functions to provide appropriate service strategies to store staff in real time. The input is emotional state data from steps 2 and 5, and the output is service suggestion information. This result is displayed on the staff interface.

[0827] Step 7:

[0828] The server uses an alarm function to warn the user if an anomaly is detected. The input is the health assessment data from step 4, and the output is the warning information. This allows for a rapid response.

[0829] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0830] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0831] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0832] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0833] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0834] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0835] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0836] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0837] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0838] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0839] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0840] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0841] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0842] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0843] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0844] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0845] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0846] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0847] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0848] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0849] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0850] The following is further disclosed regarding the embodiments described above.

[0851] (Claim 1)

[0852] An imaging means for capturing images of animal movements,

[0853] An analysis means for analyzing video data obtained by the imaging means and determining the emotional state of the animal,

[0854] A monitoring device for monitoring the vital signs of animals,

[0855] An evaluation method for evaluating the health status of an animal based on the vital signs,

[0856] A feedback means for providing the results of the analysis means and the evaluation means to the user,

[0857] A system that includes this.

[0858] (Claim 2)

[0859] The system according to claim 1, further comprising a display means for visually presenting the emotional state of an animal to a user.

[0860] (Claim 3)

[0861] The system according to claim 1, further comprising a warning means for notifying the user of an abnormality in the health condition of an animal.

[0862] "Example 1"

[0863] (Claim 1)

[0864] An imaging means for acquiring images of animal movements,

[0865] An analysis means for analyzing image information obtained by the imaging means and determining the emotional state of an animal, comprising an analysis means that uses a generative AI model,

[0866] A monitoring device for monitoring the biological information of animals, comprising a monitoring device for acquiring heart rate and body temperature,

[0867] An evaluation means for assessing the health status of an animal based on the said biological information and detecting abnormalities by comparing it with reference values,

[0868] A feedback means for providing the results of the analysis means and the evaluation means to the user, comprising a feedback means for providing visual or audible notification,

[0869] A system that includes this.

[0870] (Claim 2)

[0871] The system according to claim 1, further comprising display means for visually presenting the emotional state of an animal to a user, the display means including text information, icons, or graphic elements.

[0872] (Claim 3)

[0873] The system according to claim 1, further comprising a warning means for notifying the user of a warning when a difference in the animal's health condition from a standard value is detected, and having a warning means for providing audio or visual feedback.

[0874] "Application Example 1"

[0875] (Claim 1)

[0876] An imaging means for capturing images of the operation of a mechanical device,

[0877] An analysis means for analyzing video data obtained by the imaging means and determining the operating status of the machine,

[0878] Monitoring means for monitoring the operating patterns and temperature of the machine equipment,

[0879] An evaluation means for evaluating the operating state of a machine based on the operating pattern and temperature,

[0880] A feedback means for providing the results of the analysis means and the evaluation means to the administrator,

[0881] A system that includes this.

[0882] (Claim 2)

[0883] The system according to claim 1, further comprising a display means for visually presenting the operating status of a machine or device to an administrator.

[0884] (Claim 3)

[0885] The system according to claim 1, further comprising a warning means for notifying an administrator of a warning when an abnormality is detected in the operating state of a machine or device.

[0886] "Example 2 of combining an emotion engine"

[0887] (Claim 1)

[0888] A means for acquiring images of animal movements,

[0889] A data analysis means for analyzing animal data obtained by the video acquisition means and determining the emotional state of the animal,

[0890] A means of collecting indicators for monitoring the biological information of animals,

[0891] A condition evaluation means for evaluating the health status of an animal based on the said biological information,

[0892] A user analysis means that collects the user's facial expressions and voice, and determines the emotional state from said facial expressions and voice,

[0893] A correlation analysis means for comparing the emotional state of the animal with the emotional state of the user,

[0894] Information presentation means for providing the results of the data analysis means, the state evaluation means, and the correlation analysis means to the user,

[0895] A system that includes this.

[0896] (Claim 2)

[0897] The system according to claim 1, further comprising a display function for visually presenting the emotional state of an animal and suggestions to the user.

[0898] (Claim 3)

[0899] The system according to claim 1, further comprising a warning function that notifies the user of an abnormality in the animal's health condition.

[0900] "Application example 2 when combining with an emotional engine"

[0901] (Claim 1)

[0902] A camera that captures images of animal movements,

[0903] The camera has an analytical function that analyzes the video information obtained from the camera to determine the emotional state of the animal,

[0904] A monitoring function that monitors the vital information of animals,

[0905] An evaluation function that assesses the health status of animals based on the vital information,

[0906] A notification device that provides the user with the results of the analysis function and the evaluation function,

[0907] An information display function that grasps the emotional state of customers and their pets in real time within the store and provides information to promote interaction between the customers and their pets,

[0908] A system that includes this.

[0909] (Claim 2)

[0910] The system according to claim 1, further comprising a display device for visually presenting the emotional state of an animal to a user.

[0911] (Claim 3)

[0912] The system according to claim 1, further comprising an alarm function that notifies the user of an alarm when an abnormality is detected in the health condition of an animal. [Explanation of Symbols]

[0913] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An imaging means for capturing images of animal movements, An analysis means for analyzing video data obtained by the imaging means and determining the emotional state of the animal, A monitoring device for monitoring the vital signs of animals, An evaluation method for evaluating the health status of an animal based on the vital signs, A feedback means for providing the results of the analysis means and the evaluation means to the user, A system that includes this.

2. The system according to claim 1, further comprising a display means for visually presenting the emotional state of an animal to a user.

3. The system according to claim 1, further comprising a warning means for notifying the user of an abnormality in the health condition of an animal.

Images