System

Abstract

To provide a system for improving communication between a pet and an owner.SOLUTION: A sensor means for acquiring voice, expression, and gesture of the pet, a means for preprocessing the acquired data, a communication means for transmitting the preprocessed data to a server, a server means for analyzing the data and determining an intention of the pet, a message generation means for generating a natural language message based on the determined intention, a means for displaying the received message, a user interface means for receiving an input from a user, a communication means for transmitting the input from the user to the server, and a server means for analyzing the user input received by the server, the system includes a voice synthesis means for generating a voice instruction, a means for transmitting the generated voice instruction to a terminal, and a voice output means for reproducing the received voice instruction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technology has limited communication between pets and their owners, making it difficult to accurately understand their pets' needs and conditions. There is also a lack of means to properly analyze pets' voices and behaviors and notify owners in real time. Therefore, there is a need for technology that can quickly understand what pets want and when and how to respond. [Means for solving the problem]

[0005] This invention improves communication between pets and their owners through a system including a sensor means for acquiring the voice, facial expressions, and gestures of a pet, a means for preprocessing the acquired data, a communication means for transmitting the preprocessed data to a server, a server means for analyzing the data and determining the pet's intention, a message generation means for generating a message in natural language based on the determined intention, a means for transmitting the generated message to a terminal, a user interface means for displaying the received message, a means for accepting user input, a communication means for transmitting the user input to the server, a voice synthesis means for analyzing the user input received by the server and generating voice instructions, a means for transmitting the generated voice instructions to the terminal, and a voice output means for playing back the received voice instructions.

[0006] This system makes it possible to accurately analyze the pet's needs and condition and notify the owner in real time, allowing them to quickly understand and respond to the pet's desired behavior and condition.

[0007] "Sensor means" refers to devices such as cameras and microphones that detect and collect pet movements, sounds, and facial expressions.

[0008] "Preprocessing means" refers to a system or software that performs processing to convert the acquired data into a format that is easy for the machine learning model to analyze.

[0009] "Communication means" refers to the network interface and protocol for transmitting the acquired and pre-processed data to the server.

[0010] "Server means" refers to a computer system and associated software for analyzing the transmitted data and determining the pet's intentions.

[0011] The "message generating means" is a technology or program for converting the pet's intention determined by the server into a message in natural language.

[0012] The "user interface means" refers to a display screen and input device that allow the owner to check messages and perform operations on the terminal.

[0013] "Speech synthesis means" refers to a text-to-speech engine or related technology for converting the owner's input into voice data.

[0014] "Audio output means" refers to a speaker or related device that plays back the audio generated by the terminal and conveys instructions to the pet. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention is a system that analyzes the facial expressions, sounds, and behaviors of pets and enables communication between owners and pets based on this information. The configuration and specific operation of the system are described below.

[0037] 1. System Configuration

[0038] 1.1. Sensor means

[0039] The sensor means installed in the terminal includes a camera and microphone for detecting and collecting the voice, facial expressions, and behavior of the pet, thereby enabling the status of the pet to be acquired in real time.

[0040] 1.2. Pre-processing methods

[0041] The pre-processing means in the terminal pre-processes the acquired data in order to properly analyze it, for example, by dividing the video data into frames and removing noise.

[0042] 1.3. Means of communication

[0043] The terminal transmits the preprocessed data to the server using a communication means, which can be Wi-Fi or mobile communication.

[0044] 1.4. Server Means

[0045] The server uses a computer system with advanced computing power to analyze the received data, which is then input into an image recognition model or a voice recognition model for analysis.

[0046] 1.5. Message Creation Methods

[0047] The message generation means in the server generates a message in natural language based on the analysis results. This message is intended to convey the pet's intentions to the owner in an easy-to-understand manner.

[0048] 1.6. User Interface Methods

[0049] The terminal displays the messages sent from the server to the owner via the user interface means, allowing the owner to easily check the status and needs of their pet.

[0050] 1.7. Speech synthesis methods

[0051] The voice synthesis means in the server converts the owner's input into voice data, allowing the owner to give voice instructions to their pet.

[0052] 1.8. Audio Output Methods

[0053] The audio output means installed in the terminal reproduces the received audio data and conveys instructions to the pet.

[0054] 2. System Operation

[0055] Example 1: When your pet needs to go to the toilet

[0056] 1. Device: The device's camera and microphone detect your pet's facial expressions and behaviors and collect data.

[0057] 2. Terminal: The preprocessing means preprocesses the data and prepares it for transmission to the server.

[0058] 3. Terminal: Uses communication means to send the preprocessed data to the server.

[0059] 4. Server: The server inputs the received data into image and voice recognition models to analyze the pet's intentions.

[0060] 5. Server: Based on the analysis results, generate a message saying "I want to go to the toilet."

[0061] 6. Server: Sends the generated message to the terminal.

[0062] 7. Terminal: Displays the received message to the owner using a user interface.

[0063] Example 2: When an owner says to their pet, "Let's go for a walk"

[0064] 1. User: The owner types "Let's go for a walk" in the chat on the LINE app or gives voice instructions.

[0065] 2. Terminal: Sends the owner's input text and voice data to the server.

[0066] 3. Server: Analyzes the received data and generates audio data as needed.

[0067] 4. Server: Uses a speech synthesis tool to generate the speech "Let's go for a walk."

[0068] 5. Server: Sends the generated voice data to the device.

[0069] 6. Terminal: The received voice data is played back using the voice output means, and instructions are conveyed to the pet.

[0070] As described above, the system of the present invention is designed to realize smooth and intuitive communication between pets and their owners, allowing them to quickly understand the pet's condition and needs and take the necessary measures.

[0071] The processing flow will be explained below.

[0072] Processing flow

[0073] Example 1: When your pet needs to go to the toilet

[0074] Step 1:

[0075] Terminal

[0076] The device's camera and microphone detect your pet's facial expressions and movements in real time, collecting video and audio data.

[0077] Step 2:

[0078] Terminal

[0079] The pre-processing means divides the collected video data into frames and removes noise, and also pre-processes the audio data in the same way.

[0080] Step 3:

[0081] Terminal

[0082] The communication means prepares the pre-processed data for transmission to the server, where it is standardized and converted into a format suitable for communication.

[0083] Step 4:

[0084] server

[0085] The server receives the data sent from the terminal and temporarily stores it in a database.

[0086] Step 5:

[0087] server

[0088] The received video data is input into an image recognition model (e.g., CNN) to analyze the pet's facial expressions and behaviors. Specifically, features such as the pet's ears, tail, and posture are extracted.

[0089] Step 6:

[0090] server

[0091] The audio data is input into a speech recognition model (e.g., RNN or LSTM) to analyze the tone and patterns of the pet's voice.

[0092] Step 7:

[0093] server

[0094] The results of image and voice recognition are combined to comprehensively determine the pet's intention. In this case, the intention is determined to be "I want to go to the toilet."

[0095] Step 8:

[0096] server

[0097] The message generating means generates a natural language message saying "I want to go to the toilet."

[0098] Step 9:

[0099] server

[0100] Send the generated message to the terminal.

[0101] Step 10:

[0102] Terminal

[0103] The terminal receives the message sent from the server and displays it on the user interface.

[0104] Example 2: When an owner says to their pet, "Let's go for a walk"

[0105] Step 1:

[0106] User

[0107] The owner can either type "Let's go for a walk" in the chat on the LINE app or give the command "Let's go for a walk" by voice.

[0108] Step 2:

[0109] Terminal

[0110] The device receives text messages and voice data, performs preprocessing, and converts the voice data into text.

[0111] Step 3:

[0112] Terminal

[0113] The acquired data is sent to the server using a communication means.

[0114] Step 4:

[0115] server

[0116] The server receives the data sent from the terminal and begins analyzing it.

[0117] Step 5:

[0118] server

[0119] It analyzes voice command data and understands the command "Let's go for a walk."

[0120] Step 6:

[0121] server

[0122] A voice synthesis means converts the text data into voice data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[0123] Step 7:

[0124] server

[0125] The generated voice data is transmitted to the terminal.

[0126] Step 8:

[0127] Terminal

[0128] The device receives voice data from the server and plays it through the speaker, instructing your pet to "go for a walk."

[0129] In this way, the system of the present invention allows the terminal and server to work together to realize smooth communication between pets and their owners.

[0130] Example 1

[0131] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0132] In general, communication between pets and owners is difficult to achieve quickly and accurately because it is difficult to understand the intentions and state of the pet. In particular, owners often cannot adequately meet their pets' needs because they do not have a way to accurately grasp what their pets want and communicate it in human language.

[0133] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0134] In this invention, the server includes sensor means for acquiring the voice, facial expressions, and behaviors of the pet, data processing means for preprocessing the acquired data, communication means for transmitting the preprocessed data to the server, server means for analyzing the data and determining the intention of the pet, message generation means for generating a message in natural language based on the determined intention, data transmission means for transmitting the generated message to the terminal, user interface means for displaying the received message, input means for accepting user input, additional communication means for transmitting the user input to the server, voice synthesis means for analyzing the user input received by the server and generating voice instructions, additional data transmission means for transmitting the generated voice instructions to the terminal, and voice output means for playing back the received voice instructions. This allows the owner to quickly and accurately grasp the condition and needs of the pet and take appropriate measures.

[0135] The "sensor means" is a device for acquiring the voice, facial expressions, and behavior of a pet.

[0136] "Data processing means" refers to a device or software for preprocessing acquired data.

[0137] "Communication means" is a device or software for transmitting pre-processed data to a server.

[0138] The "server means" is a system for analyzing data on a server and determining the intentions of the pet.

[0139] A "message generator" is a device or software for generating a natural language message based on the determined intent.

[0140] The "data transmission means" is a function for transmitting the generated message to the terminal.

[0141] "User interface means" is a device or software for displaying received messages.

[0142] "Input means" refers to a device or software for accepting user input.

[0143] "Additional communication means" is a function for sending input from the user to the server.

[0144] A "voice synthesis means" is a device or software for analyzing user input received at the server and generating voice instructions.

[0145] The "additional data transmission means" is a function for transmitting the generated voice instructions to the terminal.

[0146] "Audio output means" refers to a device or software for reproducing received audio instructions.

[0147] The present invention is a system for analyzing a pet's facial expressions, voice, and behavior, and facilitating smooth communication between the owner and the pet based on the results. This system is realized through the interaction of a terminal, a server, and a user.

[0148] System Configuration

[0149] Sensor means

[0150] The device is equipped with a camera and microphone to capture your pet's facial expressions, voice, and behavior in real time. The camera captures your pet's facial expressions and movements, while the microphone collects barks and other sounds. These sensory tools are important for accurately understanding your pet's condition and behavior.

[0151] Data Processing Means

[0152] The device has a data processing means that preprocesses the acquired data. Specifically, image processing libraries such as OpenCV are used to divide the video data into frames and remove noise. Additionally, collected audio data is filtered to remove noise. This preprocessing converts the data into a format that is easier to analyze.

[0153] communication means

[0154] The device then sends the pre-processed data to the server over Wi-Fi or mobile data, typically using HTTP or WebSocket protocols, which is important to ensure fast and secure data transfer.

[0155] Server Means

[0156] The server has advanced computing power to analyze the received data. Specifically, it uses deep learning frameworks such as TENSORFLOW (registered trademark) and PyTorch to input data into image recognition and voice recognition models for analysis. For example, the image recognition model analyzes the pet's facial expressions and gestures, and the voice recognition model analyzes the meaning of its cries.

[0157] Message Generation Method

[0158] The server's message generation means generates a natural language message based on the analysis results. This process uses a generative AI model such as GPT-3 (registered trademark). For example, if a pet's behavior is interpreted as "I want to go to the toilet," the server generates a message to inform the owner.

[0159] Data transmission method

[0160] The message generated by the server is sent to the terminal using the data transmission means, which uses the HTTP protocol.

[0161] User Interface Means

[0162] The device displays the received message to the user via a user interface means, for example, as a notification on a smartphone app, allowing the owner to understand the status and needs of their pet.

[0163] Input Method

[0164] The user can give instructions to their pet by voice or text using the smartphone's microphone or keyboard.

[0165] Additional Communication Methods

[0166] The device sends user input to the server using an additional communication method. For example, if an owner types "Let's go for a walk" into the LINE app, the message is sent to the server.

[0167] Voice synthesis means

[0168] The server converts the received instructions from the owner into voice data using a voice synthesis means. Specifically, it uses a service such as Google (registered trademark) Cloud Text-to-Speech to generate voice data such as "Let's go for a walk."

[0169] Additional Data Transmission Methods

[0170] The generated voice data is transmitted to the terminal using additional data transmission means.

[0171] Audio output means

[0172] The device then plays the received voice data via the voice output means to communicate instructions to the pet. For example, a smart speaker might play a voice saying, "Let's go for a walk."

[0173] Specific examples

[0174] Example 1: When your pet needs to go to the toilet

[0175] 1. The device's camera and microphone detect your pet's facial expressions and behaviors and collect data.

[0176] 2. The data processing means of the terminal divides the video data into frames and removes noise.

[0177] 3. The device sends the preprocessed data to the server.

[0178] 4. The server analyzes the received data and generates a message saying "I want to go to the toilet."

[0179] 5. The server sends the generated message to the terminal.

[0180] 6. The device displays the received message to the owner using the user interface.

[0181] Example 2: When an owner says to their pet, "Let's go for a walk"

[0182] 1. The user types "Let's go for a walk" in the chat on the LINE app.

[0183] 2. The device sends the owner's input text to the server.

[0184] 3. The server analyzes the received data and generates audio data.

[0185] 4. The server uses a speech synthesis means to generate the speech "Let's go for a walk."

[0186] 5. The server sends the generated voice data to the terminal.

[0187] 6. The terminal uses the audio output means to play back the received audio data and convey instructions to the pet.

[0188] Prompt Sentence Examples

[0189] Prompt: "Generate a prompt to ask a pet owner if they want to go for a walk."

[0190] Output: "Shall we go for a walk?"

[0191] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0192] Step 1:

[0193] The device uses sensors to detect your pet's facial expressions, voice, and gestures. The input is your pet's real-time movements and voice, which are captured using a camera and microphone. Specifically, the camera captures your pet's facial expressions and movements, and the microphone collects its barks and other sounds. The output is the captured video and audio data.

[0194] Step 2:

[0195] The device preprocesses the acquired data. The preprocessing means divides the video data into frames and removes noise. Similarly, audio data is filtered to remove noise. The processing involves preprocessing the video data using an image processing library such as OpenCV, and preprocessing the audio data using audio filtering techniques. The output is video and audio data in a clean, easy-to-analyze format.

[0196] Step 3:

[0197] The device sends the preprocessed data to the server via a communication means. The input is the preprocessed video and audio data, which is sent via Wi-Fi or mobile communication. The HTTP protocol or WebSocket protocol is used for data transmission. The output is the completion of data transfer to the server.

[0198] Step 4:

[0199] The server analyzes the received data. The input is pre-processed video and audio data. For the analysis, deep learning frameworks such as TensorFlow and PyTorch are used to apply image recognition and voice recognition models. Specifically, the image recognition model analyzes the pet's facial expressions and gestures, while the voice recognition model analyzes the meaning of the pet's cries. The output is an analysis result that indicates the pet's intentions.

[0200] Step 5:

[0201] The server generates a natural language message based on the analysis results. The input is the analysis result indicating the pet's intention. A generative AI model such as GPT-3 is used for this process. Specifically, for example, if the pet's gesture is interpreted as "I want to go to the toilet," a message to that effect is generated for the owner. The output is a message written in natural language.

[0202] Step 6:

[0203] The server sends the generated message to the terminal using a data transmission means. The input is the generated natural language message. The HTTP protocol is used for transmission. The output is the completion of message transfer to the terminal.

[0204] Step 7:

[0205] The terminal displays the received message to the user via the user interface means. The input is a natural language message received from the server. The message is displayed as a notification on the smartphone app. The output is a message displayed to the owner.

[0206] Step 8:

[0207] The user inputs instructions for the pet by voice or text. The smartphone's microphone or keyboard is used as the input means. The input data is text or voice instructions given by the user, such as "Let's take your pet for a walk." The output is instruction data sent from the user to the device.

[0208] Step 9:

[0209] The device sends user input to the server using additional communication means. The input is the user's instruction data (text or voice). This is sent using Wi-Fi or mobile communication. The output is the transfer of instruction data to the server.

[0210] Step 10:

[0211] The server analyzes the received user instructions and converts them into voice data using a voice synthesis means. The input is the user's instruction data. This process uses services such as Google Cloud Text-to-Speech. Specifically, the text instruction "Let's go for a walk" is converted into voice data. The output is voice data.

[0212] Step 11:

[0213] The generated voice data is transmitted to the terminal using an additional data transmission means of the server. The input is the generated voice data, and the HTTP protocol is used for transmission. The output is the transfer of the voice data to the terminal.

[0214] Step 12:

[0215] The terminal plays the received voice data via the voice output means and transmits instructions to the pet. The input is the voice data received from the server. For example, a smart speaker plays the voice "Let's go for a walk." The output is the transmission of instructions to the pet.

[0216] (Application example 1)

[0217] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0218] In today's world, pets are important members of the family, and there is a need to monitor their condition and safety while their owners are away. However, there is no effective system that analyzes a pet's facial expressions, gestures, and voice, detects their intentions and abnormal behavior in real time, and notifies the owner. As a result, if a pet is in trouble while the owner is away, it is difficult to detect the problem early and take appropriate action. There is also a lack of systems that can accurately analyze a pet's condition and issue appropriate instructions. There is a need to solve these issues.

[0219] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0220] In this invention, the server includes an abnormal behavior analysis means for analyzing abnormal behavior of a pet and generating a warning message based on the analysis, a means for sending a push notification of the generated warning message to the owner's device, and a server means for using an image recognition model and a voice recognition model to determine the pet's intentions and abnormal behavior. This allows the owner to monitor the pet's condition in real time even when the owner is away, and to be notified promptly when an abnormality is detected. Furthermore, an effective system for maintaining the safety and health of pets can be provided, providing appropriate countermeasures.

[0221] The "sensor means" is a device including a camera and a microphone for capturing the sounds, facial expressions, and behaviors of a pet.

[0222] The "pre-processing means" is a device that divides the acquired data into frames and performs pre-processing such as removing noise in order to properly analyze the data.

[0223] A "communication means" is a device that uses Wi-Fi or mobile communication to transmit preprocessed data to a server.

[0224] The "server means" is a computer system with advanced computing power for analyzing data and determining the intentions of the pet.

[0225] The "message generating means" is a device that generates a message in natural language based on the determined intention.

[0226] "User interface means" is a device for displaying the generated message on a terminal.

[0227] The "means for accepting user input" is an interface device for accepting instructions and inputs from the owner.

[0228] A "voice synthesis means" is a device for analyzing user input received by the server and generating voice instructions.

[0229] "Audio output means" refers to a device for reproducing the generated audio instructions.

[0230] The "abnormal behavior analysis means" is a device that analyzes abnormal behavior of a pet and generates a warning message based on the analysis.

[0231] The "push notification means" is a device that transmits the generated warning message to the owner's terminal in real time.

[0232] The "image recognition model" is an AI model that analyzes acquired video data and determines a pet's facial expressions and behavior.

[0233] A "voice recognition model" is an AI model that analyzes acquired voice data and identifies the voice of a pet.

[0234] This system facilitates smooth communication between pets and their owners, and also has a security function that sends instant notifications if something unusual happens to the pet. It mainly consists of the following elements:

[0235] 1. System Overview

[0236] 1.1. Sensor means

[0237] The device is equipped with a camera and microphone that captures the pet's facial expressions, movements, and voice in real time. The camera and microphone are controlled using the OpenCV library.

[0238] 1.2. Pre-processing methods

[0239] The preprocessing means in the terminal divides the acquired video data into frames and removes noise. The preprocessed data is sent to the server using the communication means described below.

[0240] 1.3. Means of communication

[0241] The preprocessed data is sent to the server via Wi-Fi or mobile communication, using the Requests library.

[0242] 1.4. Server Means

[0243] The server is a computer system for advanced analysis, analyzing data using image and voice recognition models to determine the pet's intentions and abnormal behavior. The server runs the AI ​​models using the Keras / TensorFlow library.

[0244] 1.5. Message Generation and User Interface Methods

[0245] The server generates a natural language message based on the intention or abnormal behavior determined by the server and sends it to the device, which displays the message to the owner through a user interface and prompts them to take the necessary action.

[0246] 1.6. Abnormal behavior analysis means

[0247] The system analyzes pet behavior data and generates a warning message if it detects abnormal behavior. This message is then pushed to the owner's device in real time.

[0248] 2. Processing Flow

[0249] The server performs analysis using image recognition models and voice recognition models. Based on the analysis results, it generates natural language messages and warning messages, and generates voice instructions using a voice synthesis means as needed. The generated voice instructions are played back by the voice output means of the terminal.

[0250] 3. Specific Examples

[0251] Example 1: When your pet exhibits abnormal behavior

[0252] 1. The device's camera and microphone record your pet's behavior.

[0253] 2. The preprocessing means preprocesses the data and sends it to the server.

[0254] 3. The server analyzes the data and detects any abnormal behavior.

[0255] 4. A warning message is generated and pushed to the owner's device.

[0256] Prompt Sentence Examples

[0257] "Your pet is barking anxiously. Abnormal behavior may have been detected. Owner should check immediately."

[0258] Example 2: When an owner says to their pet, "Let's go for a walk"

[0259] 1. The owner inputs the information using voice or text.

[0260] 2. The user interface means sends the input to the server.

[0261] 3. The server analyzes the data and generates voice instructions using a voice synthesis means.

[0262] 4. The generated voice instructions are played on the device.

[0263] In this way, the system of the present invention not only monitors the condition of pets in real time even when the owner is away, but also sends prompt notifications when an abnormality is detected, providing an effective way to maintain the safety and health of pets.

[0264] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0265] Step 1:

[0266] The device uses a camera and microphone to capture the pet's facial expressions, gestures, and voice in real time. The input is video frames and audio data. Specifically, the device uses the OpenCV library to control the camera and capture video. At the same time, it uses the microphone to capture audio data.

[0267] Step 2:

[0268] The preprocessing means in the terminal divides the acquired video data into frames and removes noise. The input is the video frames and audio data acquired in step 1. Specifically, the video data is resized to the required resolution and divided into frames. Noise is also removed from the audio data. The output is the preprocessed video frames and audio data.

[0269] Step 3:

[0270] The communication means sends the preprocessed data to the server. The input is the video frame and audio data preprocessed in step 2. Specifically, the device uses the Requests library to send the data to the server using an HTTP POST request. The output is the data sent to the server.

[0271] Step 4:

[0272] The server analyzes the received data and determines the pet's intention. The input is the preprocessed data sent in step 3. The server uses the Keras / TensorFlow library to execute image recognition and speech recognition models to determine the pet's intention from its facial expressions, gestures, and voice. The output is the intention data as the analysis result.

[0273] Step 5:

[0274] The server generates a natural language message based on the analysis results. The input is the intent data obtained in step 4. The server uses a generative AI model to generate a natural language message based on the intent data. The output is the generated natural language message.

[0275] Step 6:

[0276] The message generation means transmits the generated message to the terminal. The input is the natural language message generated in step 5. The message is transmitted to the owner's terminal via the communication means. The output is the message displayed on the owner's terminal.

[0277] Step 7:

[0278] The terminal displays the received message to the owner using a user interface. The input is the message received in step 6. As a specific operation, the terminal displays the message on a GUI (Graphical User Interface). The output is the message displayed to the owner.

[0279] Step 8:

[0280] The user provides input through the user interface. The input can be text or voice, such as instructions or questions. The user interface means transmits this input to the server. The output is the user input transmitted to the server.

[0281] Step 9:

[0282] The server analyzes input from the user and generates appropriate voice instructions. The input is the user input, such as the instruction or question submitted in step 8. The server analyzes the text using a natural language processing model and generates the voice instructions using a speech synthesis means. The output is the generated voice instructions.

[0283] Step 10:

[0284] The generated voice instruction is transmitted to the terminal. The input is the voice instruction generated in step 9. The voice instruction is transmitted to the terminal through the communication means. The output is the voice instruction transmitted to the terminal.

[0285] Step 11:

[0286] The terminal plays the received voice instruction using the voice output means. The input is the voice instruction received in step 10. In particular, the terminal plays the voice instruction through a speaker. The output is the played voice instruction.

[0287] Step 12:

[0288] The server periodically analyzes pet behavior data and generates a warning message if it detects abnormal behavior. The input is preprocessed video and audio data. If abnormal behavior is detected, the server immediately generates a warning message and describes it in natural language using a generative AI model. The output is a warning message based on the abnormal behavior.

[0289] Step 13:

[0290] The warning message generated by the server is sent to the owner's device using a push notification means. The input is the warning message generated in step 12. The push notification server sends it to the owner's smartphone or other device. The output is the warning message received by the owner's device.

[0291] Step 14:

[0292] The owner's device displays the received warning message on the user interface and immediately notifies the owner of the abnormal behavior. The input is the warning message received in step 13. As a specific operation, the device immediately warns the owner by using an alert, notification sound, banner display, etc. The output is the warning message displayed to the owner.

[0293] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0294] This invention is a system that analyzes a pet's facial expressions, voice, and behavior, and uses that information to enable effective communication between the owner and the pet. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, richer communication is realized. The system configuration and specific operation are explained below.

[0295] 1. System Configuration

[0296] 1.1. Sensor means

[0297] The sensor means installed in the terminal includes a camera and microphone that detect and collect the voice, facial expressions, and behavior of the pet, allowing you to obtain the pet's status in real time.

[0298] 1.2. Pre-processing methods

[0299] The preprocessing means in the terminal preprocesses the acquired data in order to properly analyze it. For example, video data is divided into frames and noise is removed. Audio data is also preprocessed in the same way.

[0300] 1.3. Means of communication

[0301] The terminal transmits the preprocessed data to the server using a communication means, which can be Wi-Fi or mobile communication.

[0302] 1.4. Server Means

[0303] The server means uses a computer system with advanced computing power, which inputs the received data into an image recognition model or a voice recognition model and analyzes the data.

[0304] 1.5. Message Creation Methods

[0305] The message generation means in the server generates a message in natural language based on the analysis results. This message is intended to convey the pet's intentions to the owner in an easy-to-understand manner.

[0306] 1.6. User Interface Methods

[0307] The terminal displays the messages sent from the server to the owner via the user interface means, allowing the owner to easily check the status and needs of their pet.

[0308] 1.7. Speech synthesis methods

[0309] The voice synthesis means in the server converts the owner's input into voice data, allowing the owner to give voice instructions to their pet.

[0310] 1.8. Audio Output Methods

[0311] The audio output means installed in the terminal reproduces the received audio data and conveys instructions to the pet.

[0312] 1.9. Emotion Engine

[0313] The emotion engine includes technology to analyze the user's voice and facial expressions and recognize their emotions, allowing the system to understand the user's emotional state and reflect it in message generation and voice instructions.

[0314] 2. System Operation

[0315] Example 1: When your pet needs to go to the toilet

[0316] 1. Device: The device's camera and microphone detect the pet's facial expressions and gestures and collect video and audio data.

[0317] 2. Terminal: The preprocessing means divides the collected video data into frames and removes noise. The audio data is also preprocessed in the same way.

[0318] 3. Terminal: Uses communication means to send preprocessed data to the server.

[0319] 4. Server: The server receives the data sent from the device and temporarily stores it in a database.

[0320] 5. Server: The received video data is input into an image recognition model (e.g., CNN) to analyze the pet's facial expressions and behaviors. Specifically, it extracts features such as the pet's ears, tail, and posture.

[0321] 6. Server: Input the audio data into a speech recognition model (e.g., RNN or LSTM) to analyze the tone and patterns of the pet's voice.

[0322] 7. Server: Integrates the results of image and voice recognition to determine the pet's intention. In this case, the intention is determined to be "I want to go to the toilet."

[0323] 8. Server: The message generation means generates a natural language message saying "I want to go to the toilet."

[0324] 9. Server: Sends the generated message to the terminal.

[0325] 10. Terminal: The terminal receives the message sent from the server and displays it on the user interface.

[0326] Example 2: When an owner says to their pet, "Let's go for a walk"

[0327] 1. User: The owner types "Let's go for a walk" in the chat on the LINE app, or gives the command "Let's go for a walk" by voice.

[0328] 2. Device: The device receives the text message and voice data and performs preprocessing. The voice data is converted to text.

[0329] 3. Terminal: Uses communication means to send the acquired data to the server.

[0330] 4. Server: The server receives the data sent from the device and begins analyzing it.

[0331] 5. Server: Analyzes the voice instruction data and understands the instruction "Let's go for a walk."

[0332] 6. Server: The speech synthesis means converts the text data into speech data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[0333] 7. Server: Sends the generated audio data to the device.

[0334] 8. Device: The device receives the voice data from the server and plays it through the speaker, instructing the pet to "go for a walk."

[0335] 9. Server: The emotion engine analyzes the user's voice and facial expressions to recognize their emotions. For example, it can distinguish between commands from a happy owner and commands from a tired owner.

[0336] 10. Server: Based on the results of the emotion engine, adjusts message generation and voice synthesis, and conveys instructions to the pet in an appropriate tone.

[0337] In this way, the system of the present invention enables smooth and intuitive communication between pets and their owners by linking the terminal and server, and also enables more effective interactions by taking the user's emotions into consideration.

[0338] The processing flow will be explained below.

[0339] Example 1: When your pet needs to go to the toilet

[0340] Step 1:

[0341] Terminal

[0342] The device's camera and microphone detect your pet's facial expressions and movements in real time, collecting video and audio data.

[0343] Step 2:

[0344] Terminal

[0345] The pre-processing means divides the collected video data into frames and removes noise, and also pre-processes the audio data in the same way.

[0346] Step 3:

[0347] Terminal

[0348] The communication means prepares the pre-processed data for transmission to the server, where it is standardized and converted into a format suitable for communication.

[0349] Step 4:

[0350] Terminal

[0351] A communication means is used to transmit the pre-processed data to a server.

[0352] Step 5:

[0353] server

[0354] The server receives the data sent from the terminal and temporarily stores it in a database.

[0355] Step 6:

[0356] server

[0357] The received video data is input into an image recognition model (e.g., CNN) to analyze the pet's facial expressions and behaviors. Specifically, features such as the pet's ears, tail, and posture are extracted.

[0358] Step 7:

[0359] server

[0360] The audio data is input into a speech recognition model (e.g., RNN or LSTM) to analyze the tone and patterns of the pet's voice.

[0361] Step 8:

[0362] server

[0363] The results of image and voice recognition are combined to comprehensively determine the pet's intention. In this case, the intention is determined to be "I want to go to the toilet."

[0364] Step 9:

[0365] server

[0366] The message generating means generates a natural language message saying "I want to go to the toilet."

[0367] Step 10:

[0368] server

[0369] Send the generated message to the terminal.

[0370] Step 11:

[0371] Terminal

[0372] The terminal receives the message sent from the server and displays it on the user interface.

[0373] Step 12:

[0374] User

[0375] The user (owner) checks the notification on the device and understands that the pet wants to go to the toilet.

[0376] Example 2: When an owner says to their pet, "Let's go for a walk"

[0377] Step 1:

[0378] User

[0379] The owner can either type "Let's go for a walk" in the chat on the LINE app or give the command "Let's go for a walk" by voice.

[0380] Step 2:

[0381] Terminal

[0382] The device receives text messages and voice data, performs preprocessing, and converts the voice data into text through speech recognition.

[0383] Step 3:

[0384] Terminal

[0385] The acquired text message and voice data are sent to a server using a communication means.

[0386] Step 4:

[0387] server

[0388] The server receives the data sent from the terminal and begins analyzing it.

[0389] Step 5:

[0390] server

[0391] It analyzes received text messages and voice data and understands the instructions such as "Let's go for a walk."

[0392] Step 6:

[0393] server

[0394] A voice synthesis means converts the text data into voice data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[0395] Step 7:

[0396] server

[0397] The generated voice data is transmitted to the terminal.

[0398] Step 8:

[0399] Terminal

[0400] The device receives voice data from the server and plays it through the speaker, instructing your pet to "go for a walk."

[0401] Step 9:

[0402] server

[0403] The emotion engine analyzes the user's voice and facial expressions to recognize their emotions. For example, it can distinguish between commands from a happy owner and commands from a tired owner.

[0404] Step 10:

[0405] server

[0406] Based on the results of the emotion engine, message generation and voice synthesis are adjusted to convey instructions to pets in an appropriate tone.

[0407] In this way, the system of the present invention enables smooth and intuitive communication between pets and their owners by linking the terminal and server, and also enables more effective interactions by taking the user's emotions into consideration.

[0408] Example 2

[0409] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0410] In today's communication between pets and their owners, it is difficult to accurately understand the pet's intentions and emotions, which often results in delayed appropriate responses. Furthermore, when an owner gives a command to a pet, the pet may not accurately understand the owner's intentions, making effective communication difficult. To solve these issues, a system is needed that can accurately detect and analyze a pet's condition and appropriately communicate the results to the owner.

[0411] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a sensor means for detecting and collecting the pet's status, a preprocessing means for dividing the acquired video data into frames and removing noise, a communication means for transmitting the preprocessed data to the server, a calculation means for analyzing the data and determining the pet's intention, a message generation means for generating a message in natural language based on the determined intention, a communication means for transmitting the generated message to the terminal, a user interface means for displaying the received message, an input means for accepting user input, a communication means for transmitting the user input to the server, a synthesis means for analyzing the user input received by the server and generating voice instructions, a communication means for transmitting the generated voice instructions to the terminal, an output means for playing back the received voice instructions, and an emotion recognition means for analyzing the user's voice and facial expressions to recognize emotions. This enables smooth and intuitive communication between the pet and its owner.

[0412] "Sensor means" refers to means that use devices such as cameras and microphones to detect and collect information about the condition of pets.

[0413] The "preprocessing means" is a means for processing the acquired data into a form that is easier to analyze. Specifically, the preprocessing means divides the video data into frames and performs processing to remove noise.

[0414] "Communication means" refers to the means for sending and receiving preprocessed data and generated messages between servers and terminals, and uses Wi-Fi or mobile communications.

[0415] The "calculation means" is a means for analyzing the received data and determining the pet's intentions, and uses an image recognition model or a voice recognition model.

[0416] The "message generating means" is a means for generating a natural language message that conveys the pet's intention based on the analysis results.

[0417] "User interface means" refers to means for displaying received messages to the owner, allowing the owner to check the status and needs of their pet.

[0418] The "input means" is a means for accepting instructions or input from the user, and accepts text or voice input.

[0419] A "synthesizer" is a means for analyzing a user's input and generating appropriate voice instructions.

[0420] The "output means" refers to a means for reproducing the generated voice instructions, and uses a device such as a speaker.

[0421] The "emotion recognition means" is a means for analyzing the user's voice and facial expression and recognizing the user's emotion.

[0422] MODE FOR CARRYING OUT THE INVENTION

[0423] The present invention is a system that analyzes a pet's facial expressions, voice, and behavior, and uses that information to enable effective communication between the owner and the pet. This system includes multiple technical elements that operate in conjunction with a terminal and a server. The system configuration and specific operation are described below.

[0424] System Configuration

[0425] Sensor means

[0426] The device is equipped with a camera and microphone, which detect and collect information about your pet's condition in real time. For example, the camera captures images of your pet's ears, tail, and posture, while the microphone records their cries and barks.

[0427] Pretreatment means

[0428] The pre-processing means within the device has the function of processing the collected data into a form that is easier to analyze. Specifically, it divides the video data into frames and removes noise. It also performs noise removal processing on audio data. For example, the device divides the video into frames every few seconds and automatically adjusts the brightness and color tone from each frame.

[0429] communication means

[0430] The device has a communication means to send the preprocessed data to the server. This communication means can be Wi-Fi or mobile communication. For example, the device checks for a Wi-Fi connection and uploads the data to the server via a stable network.

[0431] Server Means

[0432] The server is a computer system with advanced computing power that inputs the received data into an image recognition model (e.g., CNN) or a voice recognition model (e.g., RNN or LSTM) to analyze the pet's facial expressions and behaviors. Specifically, the server uses a deep learning model to detect the position of the ears and tail from the video and stress or happiness from the voice.

[0433] Message Generation Method

[0434] The message generation means in the server generates and transmits a natural language message that conveys the pet's intentions based on the analysis results, such as "I want to go to the toilet" or "I want to play."

[0435] User Interface Means

[0436] The device displays messages sent from the server to the owner through a user interface, allowing the owner to easily check the pet's status and needs. For example, messages can be displayed as a pop-up notification on a smartphone or tablet.

[0437] Voice synthesis means

[0438] The server's voice synthesis means converts user input into voice data, allowing users to give voice commands to their pets. For example, if an owner sends a message to their pet via the LINE app saying "Let's go for a walk," the server converts the message into natural-sounding voice.

[0439] Audio output means

[0440] The device's built-in voice output means plays back the received voice data to communicate instructions to your pet. For example, a natural voice will be played from the device's speaker saying, "Let's go for a walk."

[0441] emotion recognition means

[0442] The emotion recognition means includes technology that analyzes the user's voice and facial expressions and recognizes their emotions. This allows the system to understand the user's emotional state and reflect it in message generation and voice instructions. For example, the system can distinguish between instructions from an owner who is happy and instructions from an owner who is tired and communicate them to the pet in an appropriate tone.

[0443] Specific examples

[0444] Example 1: When your pet needs to go to the toilet

[0445] The device's camera and microphone detect your pet's facial expressions and gestures, collecting video and audio data.

[0446] The pre-processing means divides the collected video data into frames and removes noise, and also pre-processes the audio data in the same way.

[0447] The preprocessed data is sent to the server using a communication means.

[0448] The data received by the server is input into image and voice recognition models for analysis, and the intent is determined to be "I want to go to the toilet."

[0449] The message generating means generates a natural language message saying "I want to go to the toilet" and transmits it to the terminal.

[0450] The terminal receives the message sent from the server and displays it on the user interface.

[0451] Example 2: When an owner says to their pet, "Let's go for a walk"

[0452] The owner can either type "Let's go for a walk" in the chat on the LINE app or give the command "Let's go for a walk" by voice.

[0453] The device receives text messages and voice data, performs preprocessing, and converts the voice data into text.

[0454] The acquired data is sent to the server using a communication means.

[0455] The server analyzes the received data and understands the instruction "Let's go for a walk."

[0456] A voice synthesis means converts the text data into voice data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[0457] The generated voice data is transmitted to the terminal.

[0458] The device receives voice data from the server and plays it through the speaker, instructing your pet to "go for a walk."

[0459] Examples of prompt statements

[0460] Example inputs to a generative AI model:

[0461] "Please explain how to analyze a pet's gestures and voice to communicate its intentions to its owner."

[0462] "Please explain how the system works to notify pet owners appropriately when their pets behave in a certain way."

[0463] As described above, the system of the present invention enables smooth and effective communication between pets and their owners by linking the terminal and server, and also takes into account the user's emotions, enabling more intuitive and friendly communication.

[0464] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0465] Step 1: Collect pet status

[0466] Subject: Device

[0467] Description: The device uses a camera and microphone to detect your pet's condition (facial expressions, voice, and behavior) in real time and collects this data.

[0468] Input: Your pet's real-life facial expressions, sounds, and gestures

[0469] Output: Collected video and audio data

[0470] Specific operations: The device's camera captures your pet's ears, tail, posture, etc., and the microphone records any meows or barks.

[0471] Step 2: Preprocessing the data

[0472] Subject: Device

[0473] Description: The preprocessing means divides the collected video data into frames and removes noise. The audio data is also subjected to noise removal.

[0474] Input: Collected video and audio data

[0475] Output: Preprocessed video frames and audio data

[0476] What it does: The device splits the video into frames every few seconds, then automatically adjusts the brightness and color of each frame to remove noise.

[0477] Step 3: Sending data

[0478] Subject: Device

[0479] Description: The pre-processed data is sent to the server via a communication method, in this case Wi-Fi or mobile communication.

[0480] Input: Preprocessed video frames and audio data

[0481] Output: Data sent to the server

[0482] What happens: The device checks for a Wi-Fi connection and uploads data to the server over a stable network.

[0483] Step 4: Analyze the data

[0484] Subject: Server

[0485] Description: The server inputs the received data into an image recognition model (e.g., CNN) or a speech recognition model (e.g., RNN or LSTM) for analysis.

[0486] Input: Preprocessed data sent to the server

[0487] Output: Parsed pet's intention and state information

[0488] Specific operation: The server uses a deep learning model to detect the position of the ears and tail from video footage, and emotions and intentions such as stress or joy from audio.

[0489] Step 5: Generate a message

[0490] Subject: Server

[0491] Description: Based on the analysis results, the message generator generates a natural language message that conveys the pet's intentions.

[0492] Input: Parsed pet intent and state information

[0493] Output: Natural language message

[0494] Concrete behavior: The process of generating simple messages such as "I want to go to the toilet" or "I want to play."

[0495] Step 6: Sending a message

[0496] Subject: Server (terminal)

[0497] Description: The server generates a message and sends it to the terminal, which receives it.

[0498] Input: A natural language message

[0499] Output: Message sent to the terminal

[0500] Specific operation: The server sends a message to the terminal, and the terminal receives the message.

[0501] Step 7: Displaying messages

[0502] Subject: Device

[0503] Description: Displays received messages to the owner through user interface means.

[0504] Input: Message sent to terminal

[0505] Output: The message displayed in the user interface

[0506] What happens: A message will pop up through your smartphone or tablet's notification feature.

[0507] Step 8: Accepting User Input

[0508] Subject: User

[0509] Description: The user enters instructions in the form of speech or text.

[0510] Input: User instructions (voice or text message)

[0511] Output: User's instruction data

[0512] Specific actions: The owner types "Let's go for a walk" in the chat on the LINE app, or gives voice instructions.

[0513] Step 9: Sending User Input

[0514] Subject: Device

[0515] Description: Preprocesses input from the user and sends it to the server via a communication medium.

[0516] Input: User-specified data

[0517] Output: User instruction data sent to the server

[0518] Specific operation: The process of converting input voice data into text and sending it to the server.

[0519] Step 10: Generate voice instructions

[0520] Subject: Server

[0521] Description: The server analyzes the user's input and generates appropriate voice instructions using a speech synthesis method.

[0522] Input: User instruction data sent to the server

[0523] Output: Generated voice instruction data

[0524] Specific operation: Convert the text message "Let's go for a walk" into voice data with natural pronunciation.

[0525] Step 11: Sending voice instructions

[0526] Subject: Server (terminal)

[0527] Description: The server generates and sends voice instructions to the device, which receives them.

[0528] Input: Generated voice command data

[0529] Output: Audio instructions sent to the device

[0530] Specific operation: The server sends voice data to the terminal, and the terminal receives the data.

[0531] Step 12: Play audio instructions

[0532] Subject: Device

[0533] Description: Plays back the received audio instructions through the audio output means.

[0534] Input: Voice commands sent to the device

[0535] Output: The audio instructions played

[0536] Specific operation: "Let's go for a walk" is played in a natural voice from the device's speaker.

[0537] (Application example 2)

[0538] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0539] Currently, stores that cater to customers with pets need to quickly and accurately understand the pet's condition and respond appropriately. However, it is difficult to understand the pet's intentions in real time, which can lead to delayed responses, especially in emergencies. In addition, smooth communication between pets, owners, and store staff is required, but an efficient system to achieve this does not exist.

[0540] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for determining the pet's intention, a notification means for sending a notification to the user and staff, and a means for analyzing the pet's condition using an image recognition model and a voice recognition model. This makes it possible to monitor the pet's condition in real time and take appropriate measures, including emergency notifications. It also enables the user and staff to quickly understand the pet's condition and take appropriate measures.

[0541] "Pets" are animals kept by humans, such as dogs and cats.

[0542] "Audio" refers to the voices and sounds made by pets.

[0543] "Expression" refers to the facial expressions and changes in your pet's face.

[0544] "Gestures" refer to the body movements and actions of pets.

[0545] "Sensor means" refers to a device such as a camera or microphone for detecting the voice, facial expressions, and behavior of a pet.

[0546] "Preprocessing means" refers to means for converting collected data into a form that is easy to analyze.

[0547] "Communication means" refers to a means for transmitting pre-processed data to a server.

[0548] The "server means" refers to a means for analyzing the received data and determining the intention of the pet.

[0549] The "message generating means" refers to a means for generating a message in natural language based on the determined intention.

[0550] "User interface means" refers to means for displaying generated messages to a user.

[0551] A "voice synthesis means" is a means for analyzing input from a user and generating voice instructions.

[0552] "Audio output means" refers to means for reproducing the generated audio instructions.

[0553] "Notification means" refers to a means for determining the pet's condition and sending notifications to the user and staff.

[0554] An "image recognition model" is a machine learning model for analyzing pets' facial expressions and behaviors.

[0555] A "voice recognition model" is a machine learning model for analyzing pet voices.

[0556] This invention is a support system for customers who bring pets to physical stores, and as a specific example, we will explain a "pet support app." This application has the function of monitoring the pet's condition in real time and sending appropriate notifications to users and staff.

[0557] Hardware and software used

[0558] Hardware

[0559] Camera: Used to capture your pet's expressions and behavior in real time.

[0560] Microphone: Used to record your pet's voice.

[0561] Smartphone: The above sensors (camera and microphone) are built in.

[0562] Server: A high-performance computer system that analyzes data and stores results.

[0563] software

[0564] OpenCV: A library for acquiring and preprocessing video data from a camera.

[0565] TensorFlow / Keras: Machine learning libraries for implementing image recognition models to analyze pet facial expressions and behaviors.

[0566] SpeechRecognition: A library for converting your pet's spoken commands into text.

[0567] Requests: A library for sending preprocessed data to a server.

[0568] Data processing and calculation

[0569] Terminal handling

[0570] The device activates the camera and microphone to capture the pet's facial expressions, gestures, and voice in real time. Then, it splits the video data into frames and performs preprocessing using OpenCV. It also converts the voice data into text using SpeechRecognition. This preprocessed data is sent to the server using Requests.

[0571] Server Processing

[0572] The server analyzes the received data in real time. It uses a TensorFlow / Keras-based image recognition model to extract features from the pet's facial expressions and gestures and determine the pet's intentions. It also uses a voice recognition model to analyze the pet's voice patterns. These analysis results are combined to determine the pet's condition. The determination result is generated as a natural language message by a message generation means and sent to the terminal.

[0573] Specific examples

[0574] Example 1: When your pet needs to go to the toilet

[0575] The device's camera and microphone capture the pet's state and perform preprocessing.

[0576] The server analyzes the data and determines the intention to "go to the toilet."

[0577] The result of the judgment is notified to the user and staff, and appropriate action is taken.

[0578] Example 2: When the owner says, "Let's go for a walk"

[0579] The user types in the text "Let's go for a walk" or gives the voice command.

[0580] The device preprocesses the data and sends it to the server.

[0581] The server analyzes the instructions, generates appropriate voice instructions, and sends them to the terminal.

[0582] The device will play audio instructions and communicate them to your pet.

[0583] Prompt Sentence Examples

[0584] Message prompt to display in the user interface:

[0585] When a pet's condition is emergency, generate the following message: 'Your pet is in [condition]. Please take appropriate action.'

[0586] Staff notification message prompt:

[0587] Generate an emergency notification message to send to staff: 'Pet is in [condition]. Please respond immediately.'

[0588] In this way, the system of the present invention enables support for customers with pets by linking the server and terminals to quickly and accurately grasp the condition of pets and provide appropriate notifications to users and staff.

[0589] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0590] Step 1:

[0591] The device activates the camera and microphone to capture the pet's facial expressions, gestures, and voice in real time. The input is the camera image and the microphone audio, and the output is image data and voice data for each frame.

[0592] Step 2:

[0593] The device uses OpenCV to preprocess the video data. Specifically, it splits it into frames and performs noise reduction and resizing as needed. The input is the captured image data for each frame, and the output is the preprocessed image data.

[0594] Step 3:

[0595] The device uses SpeechRecognition to preprocess the audio data, specifically converting it to text. The input is the captured audio data, and the output is text data.

[0596] Step 4:

[0597] The terminal uses Requests to send preprocessed image data and text data to the server. The input is the preprocessed image data and text data, and the output is the data sent to the server.

[0598] Step 5:

[0599] The server analyzes the received image data using a TensorFlow / Keras-based image recognition model. Specifically, it extracts features from the pet's facial expressions and behavior. The input is the preprocessed image data, and the output is the analysis results.

[0600] Step 6:

[0601] The server uses a speech recognition model to analyze the received text data. Specifically, it analyzes the pet's voice patterns and intentions from the text data. The input is the preprocessed text data, and the output is the analysis results.

[0602] Step 7:

[0603] The server integrates the results of image and voice recognition analysis to comprehensively judge the pet's condition. The input is the results of image and voice analysis, and the output is a judgment result indicating the pet's condition.

[0604] Step 8:

[0605] The server generates a natural language message based on the judgment result. Using the message generation means, the judgment result is converted into a text message that is easy for users and staff to understand. The input is the judgment result, and the output is a natural language message.

[0606] Step 9:

[0607] The server sends the generated message to the terminal. The input is the generated natural language message, and the output is the message sent to the terminal.

[0608] Step 10:

[0609] The terminal displays the received messages to the user through a user interface means, where the input is the message sent from the server and the output is the message displayed to the user.

[0610] Step 11:

[0611] The user inputs commands to the pet through the terminal. For example, "Let's go for a walk" is given by voice or text. The input is the user's command, and the output is text or voice data.

[0612] Step 12:

[0613] The terminal sends text data and voice data to the server. The input is the text or voice data containing the user's instructions, and the output is the data sent to the server.

[0614] Step 13:

[0615] The server analyzes the received instructions and generates voice instructions for the pet using a voice synthesis means. The input is the user's instruction data, and the output is the synthesized voice instruction.

[0616] Step 14:

[0617] The server sends the synthesized voice instructions to the terminal, where the input is the generated voice instructions and the output is the voice data sent to the terminal.

[0618] Step 15:

[0619] The terminal uses the audio output means to play audio instructions to the pet, with the input being audio data sent from the server and the output being the played audio instructions.

[0620] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0621] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0622] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0623] [Second embodiment]

[0624] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0625] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0626] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0627] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0628] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0629] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0630] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0631] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0632] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0633] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0634] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0635] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0636] The present invention is a system that analyzes the facial expressions, sounds, and behaviors of pets and enables communication between owners and pets based on this information. The configuration and specific operation of the system are described below.

[0637] 1. System Configuration

[0638] 1.1. Sensor means

[0639] The sensor means installed in the terminal includes a camera and microphone for detecting and collecting the voice, facial expressions, and behavior of the pet, thereby enabling the status of the pet to be acquired in real time.

[0640] 1.2. Pre-processing methods

[0641] The pre-processing means in the terminal pre-processes the acquired data in order to properly analyze it, for example, by dividing the video data into frames and removing noise.

[0642] 1.3. Means of communication

[0643] The terminal transmits the preprocessed data to the server using a communication means, which can be Wi-Fi or mobile communication.

[0644] 1.4. Server Means

[0645] The server uses a computer system with advanced computing power to analyze the received data, which is then input into an image recognition model or a voice recognition model for analysis.

[0646] 1.5. Message Creation Methods

[0647] The message generation means in the server generates a message in natural language based on the analysis results. This message is intended to convey the pet's intentions to the owner in an easy-to-understand manner.

[0648] 1.6. User Interface Methods

[0649] The terminal displays the messages sent from the server to the owner via the user interface means, allowing the owner to easily check the status and needs of their pet.

[0650] 1.7. Speech synthesis methods

[0651] The voice synthesis means in the server converts the owner's input into voice data, allowing the owner to give voice instructions to their pet.

[0652] 1.8. Audio Output Methods

[0653] The audio output means installed in the terminal reproduces the received audio data and conveys instructions to the pet.

[0654] 2. System Operation

[0655] Example 1: When your pet needs to go to the toilet

[0656] 1. Device: The device's camera and microphone detect your pet's facial expressions and behaviors and collect data.

[0657] 2. Terminal: The preprocessing means preprocesses the data and prepares it for transmission to the server.

[0658] 3. Terminal: Uses communication means to send the preprocessed data to the server.

[0659] 4. Server: The server inputs the received data into image and voice recognition models to analyze the pet's intentions.

[0660] 5. Server: Based on the analysis results, generate a message saying "I want to go to the toilet."

[0661] 6. Server: Sends the generated message to the terminal.

[0662] 7. Terminal: Displays the received message to the owner using a user interface.

[0663] Example 2: When an owner says to their pet, "Let's go for a walk"

[0664] 1. User: The owner types "Let's go for a walk" in the chat on the LINE app or gives voice instructions.

[0665] 2. Terminal: Sends the owner's input text and voice data to the server.

[0666] 3. Server: Analyzes the received data and generates audio data as needed.

[0667] 4. Server: Uses a speech synthesis tool to generate the speech "Let's go for a walk."

[0668] 5. Server: Sends the generated voice data to the device.

[0669] 6. Terminal: The received voice data is played back using the voice output means, and instructions are conveyed to the pet.

[0670] As described above, the system of the present invention is designed to realize smooth and intuitive communication between pets and their owners, allowing them to quickly understand the pet's condition and needs and take the necessary measures.

[0671] The processing flow will be explained below.

[0672] Processing flow

[0673] Example 1: When your pet needs to go to the toilet

[0674] Step 1:

[0675] Terminal

[0676] The device's camera and microphone detect your pet's facial expressions and movements in real time, collecting video and audio data.

[0677] Step 2:

[0678] Terminal

[0679] The pre-processing means divides the collected video data into frames and removes noise, and also pre-processes the audio data in the same way.

[0680] Step 3:

[0681] Terminal

[0682] The communication means prepares the pre-processed data for transmission to the server, where it is standardized and converted into a format suitable for communication.

[0683] Step 4:

[0684] server

[0685] The server receives the data sent from the terminal and temporarily stores it in a database.

[0686] Step 5:

[0687] server

[0688] The received video data is input into an image recognition model (e.g., CNN) to analyze the pet's facial expressions and behaviors. Specifically, features such as the pet's ears, tail, and posture are extracted.

[0689] Step 6:

[0690] server

[0691] The audio data is input into a speech recognition model (e.g., RNN or LSTM) to analyze the tone and patterns of the pet's voice.

[0692] Step 7:

[0693] server

[0694] The results of image and voice recognition are combined to comprehensively determine the pet's intention. In this case, the intention is determined to be "I want to go to the toilet."

[0695] Step 8:

[0696] server

[0697] The message generating means generates a natural language message saying "I want to go to the toilet."

[0698] Step 9:

[0699] server

[0700] Send the generated message to the terminal.

[0701] Step 10:

[0702] Terminal

[0703] The terminal receives the message sent from the server and displays it on the user interface.

[0704] Example 2: When an owner says to their pet, "Let's go for a walk"

[0705] Step 1:

[0706] User

[0707] The owner can either type "Let's go for a walk" in the chat on the LINE app or give the command "Let's go for a walk" by voice.

[0708] Step 2:

[0709] Terminal

[0710] The device receives text messages and voice data, performs preprocessing, and converts the voice data into text.

[0711] Step 3:

[0712] Terminal

[0713] The acquired data is sent to the server using a communication means.

[0714] Step 4:

[0715] server

[0716] The server receives the data sent from the terminal and begins analyzing it.

[0717] Step 5:

[0718] server

[0719] It analyzes voice command data and understands the command "Let's go for a walk."

[0720] Step 6:

[0721] server

[0722] A voice synthesis means converts the text data into voice data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[0723] Step 7:

[0724] server

[0725] The generated voice data is transmitted to the terminal.

[0726] Step 8:

[0727] Terminal

[0728] The device receives voice data from the server and plays it through the speaker, instructing your pet to "go for a walk."

[0729] In this way, the system of the present invention allows the terminal and server to work together to realize smooth communication between pets and their owners.

[0730] Example 1

[0731] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0732] In general, communication between pets and owners is difficult to achieve quickly and accurately because it is difficult to understand the intentions and state of the pet. In particular, owners often cannot adequately meet their pets' needs because they do not have a way to accurately grasp what their pets want and communicate it in human language.

[0733] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0734] In this invention, the server includes sensor means for acquiring the voice, facial expressions, and behaviors of the pet, data processing means for preprocessing the acquired data, communication means for transmitting the preprocessed data to the server, server means for analyzing the data and determining the intention of the pet, message generation means for generating a message in natural language based on the determined intention, data transmission means for transmitting the generated message to the terminal, user interface means for displaying the received message, input means for accepting user input, additional communication means for transmitting the user input to the server, voice synthesis means for analyzing the user input received by the server and generating voice instructions, additional data transmission means for transmitting the generated voice instructions to the terminal, and voice output means for playing back the received voice instructions. This allows the owner to quickly and accurately grasp the condition and needs of the pet and take appropriate measures.

[0735] The "sensor means" is a device for acquiring the voice, facial expressions, and behavior of a pet.

[0736] "Data processing means" refers to a device or software for preprocessing acquired data.

[0737] "Communication means" is a device or software for transmitting pre-processed data to a server.

[0738] The "server means" is a system for analyzing data on a server and determining the intentions of the pet.

[0739] A "message generator" is a device or software for generating a natural language message based on the determined intent.

[0740] The "data transmission means" is a function for transmitting the generated message to the terminal.

[0741] "User interface means" is a device or software for displaying received messages.

[0742] "Input means" refers to a device or software for accepting user input.

[0743] "Additional communication means" is a function for sending input from the user to the server.

[0744] A "voice synthesis means" is a device or software for analyzing user input received at the server and generating voice instructions.

[0745] The "additional data transmission means" is a function for transmitting the generated voice instructions to the terminal.

[0746] "Audio output means" refers to a device or software for reproducing received audio instructions.

[0747] The present invention is a system for analyzing a pet's facial expressions, voice, and behavior, and facilitating smooth communication between the owner and the pet based on the results. This system is realized through the interaction of a terminal, a server, and a user.

[0748] System Configuration

[0749] Sensor means

[0750] The device is equipped with a camera and microphone to capture your pet's facial expressions, voice, and behavior in real time. The camera captures your pet's facial expressions and movements, while the microphone collects barks and other sounds. These sensory tools are important for accurately understanding your pet's condition and behavior.

[0751] Data Processing Means

[0752] The device has a data processing means that preprocesses the acquired data. Specifically, image processing libraries such as OpenCV are used to divide the video data into frames and remove noise. Additionally, collected audio data is filtered to remove noise. This preprocessing converts the data into a format that is easier to analyze.

[0753] communication means

[0754] The device then sends the pre-processed data to the server over Wi-Fi or mobile data, typically using HTTP or WebSocket protocols, which is important to ensure fast and secure data transfer.

[0755] Server Means

[0756] The server has advanced computing power to analyze the received data. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to input the data into image recognition and voice recognition models for analysis. For example, the image recognition model analyzes the pet's facial expressions and gestures, and the voice recognition model analyzes the meaning of its cries.

[0757] Message Generation Method

[0758] The server's message generation means generates natural language messages based on the analysis results. This process uses generative AI models such as GPT-3. For example, if a pet's behavior is interpreted as "I want to go to the toilet," a message to that effect is generated for the owner.

[0759] Data transmission method

[0760] The message generated by the server is sent to the terminal using the data transmission means, which uses the HTTP protocol.

[0761] User Interface Means

[0762] The device displays the received message to the user via a user interface means, for example, as a notification on a smartphone app, allowing the owner to understand the status and needs of their pet.

[0763] Input Method

[0764] The user can give instructions to their pet by voice or text using the smartphone's microphone or keyboard.

[0765] Additional Communication Methods

[0766] The device sends user input to the server using an additional communication method. For example, if an owner types "Let's go for a walk" into the LINE app, the message is sent to the server.

[0767] Voice synthesis means

[0768] The server converts the received instructions from the owner into voice data using a voice synthesis tool. Specifically, it uses a service such as Google Cloud Text-to-Speech to generate voice data such as "Let's go for a walk."

[0769] Additional Data Transmission Methods

[0770] The generated voice data is transmitted to the terminal using additional data transmission means.

[0771] Audio output means

[0772] The device then plays the received voice data via the voice output means to communicate instructions to the pet. For example, a smart speaker might play a voice saying, "Let's go for a walk."

[0773] Specific examples

[0774] Example 1: When your pet needs to go to the toilet

[0775] 1. The device's camera and microphone detect your pet's facial expressions and behaviors and collect data.

[0776] 2. The data processing means of the terminal divides the video data into frames and removes noise.

[0777] 3. The device sends the preprocessed data to the server.

[0778] 4. The server analyzes the received data and generates a message saying "I want to go to the toilet."

[0779] 5. The server sends the generated message to the terminal.

[0780] 6. The device displays the received message to the owner using the user interface.

[0781] Example 2: When an owner says to their pet, "Let's go for a walk"

[0782] 1. The user types "Let's go for a walk" in the chat on the LINE app.

[0783] 2. The device sends the owner's input text to the server.

[0784] 3. The server analyzes the received data and generates audio data.

[0785] 4. The server uses a speech synthesis means to generate the speech "Let's go for a walk."

[0786] 5. The server sends the generated voice data to the terminal.

[0787] 6. The terminal uses the audio output means to play back the received audio data and convey instructions to the pet.

[0788] Prompt Sentence Examples

[0789] Prompt: "Generate a prompt to ask a pet owner if they want to go for a walk."

[0790] Output: "Shall we go for a walk?"

[0791] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0792] Step 1:

[0793] The device uses sensors to detect your pet's facial expressions, voice, and gestures. The input is your pet's real-time movements and voice, which are captured using a camera and microphone. Specifically, the camera captures your pet's facial expressions and movements, and the microphone collects its barks and other sounds. The output is the captured video and audio data.

[0794] Step 2:

[0795] The device preprocesses the acquired data. The preprocessing means divides the video data into frames and removes noise. Similarly, audio data is filtered to remove noise. The processing involves preprocessing the video data using an image processing library such as OpenCV, and preprocessing the audio data using audio filtering techniques. The output is video and audio data in a clean, easy-to-analyze format.

[0796] Step 3:

[0797] The device sends the preprocessed data to the server via a communication means. The input is the preprocessed video and audio data, which is sent via Wi-Fi or mobile communication. The HTTP protocol or WebSocket protocol is used for data transmission. The output is the completion of data transfer to the server.

[0798] Step 4:

[0799] The server analyzes the received data. The input is pre-processed video and audio data. For the analysis, deep learning frameworks such as TensorFlow and PyTorch are used to apply image recognition and voice recognition models. Specifically, the image recognition model analyzes the pet's facial expressions and gestures, while the voice recognition model analyzes the meaning of the pet's cries. The output is an analysis result that indicates the pet's intentions.

[0800] Step 5:

[0801] The server generates a natural language message based on the analysis results. The input is the analysis result indicating the pet's intention. A generative AI model such as GPT-3 is used for this process. Specifically, for example, if the pet's gesture is interpreted as "I want to go to the toilet," a message to that effect is generated for the owner. The output is a message written in natural language.

[0802] Step 6:

[0803] The server sends the generated message to the terminal using a data transmission means. The input is the generated natural language message. The HTTP protocol is used for transmission. The output is the completion of message transfer to the terminal.

[0804] Step 7:

[0805] The terminal displays the received message to the user via the user interface means. The input is a natural language message received from the server. The message is displayed as a notification on the smartphone app. The output is a message displayed to the owner.

[0806] Step 8:

[0807] The user inputs instructions for the pet by voice or text. The smartphone's microphone or keyboard is used as the input means. The input data is text or voice instructions given by the user, such as "Let's take your pet for a walk." The output is instruction data sent from the user to the device.

[0808] Step 9:

[0809] The device sends user input to the server using additional communication means. The input is the user's instruction data (text or voice). This is sent using Wi-Fi or mobile communication. The output is the transfer of instruction data to the server.

[0810] Step 10:

[0811] The server analyzes the received user instructions and converts them into voice data using a voice synthesis means. The input is the user's instruction data. This process uses services such as Google Cloud Text-to-Speech. Specifically, the text instruction "Let's go for a walk" is converted into voice data. The output is voice data.

[0812] Step 11:

[0813] The generated voice data is transmitted to the terminal using an additional data transmission means of the server. The input is the generated voice data, and the HTTP protocol is used for transmission. The output is the transfer of the voice data to the terminal.

[0814] Step 12:

[0815] The terminal plays the received voice data via the voice output means and transmits instructions to the pet. The input is the voice data received from the server. For example, a smart speaker plays the voice "Let's go for a walk." The output is the transmission of instructions to the pet.

[0816] (Application example 1)

[0817] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0818] In today's world, pets are important members of the family, and there is a need to monitor their condition and safety while their owners are away. However, there is no effective system that analyzes a pet's facial expressions, gestures, and voice, detects their intentions and abnormal behavior in real time, and notifies the owner. As a result, if a pet is in trouble while the owner is away, it is difficult to detect the problem early and take appropriate action. There is also a lack of systems that can accurately analyze a pet's condition and issue appropriate instructions. There is a need to solve these issues.

[0819] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0820] In this invention, the server includes an abnormal behavior analysis means for analyzing abnormal behavior of a pet and generating a warning message based on the analysis, a means for sending a push notification of the generated warning message to the owner's device, and a server means for using an image recognition model and a voice recognition model to determine the pet's intentions and abnormal behavior. This allows the owner to monitor the pet's condition in real time even when the owner is away, and to be notified promptly when an abnormality is detected. Furthermore, an effective system for maintaining the safety and health of pets can be provided, providing appropriate countermeasures.

[0821] The "sensor means" is a device including a camera and a microphone for capturing the sounds, facial expressions, and behaviors of a pet.

[0822] The "pre-processing means" is a device that divides the acquired data into frames and performs pre-processing such as removing noise in order to properly analyze the data.

[0823] A "communication means" is a device that uses Wi-Fi or mobile communication to transmit preprocessed data to a server.

[0824] The "server means" is a computer system with advanced computing power for analyzing data and determining the intentions of the pet.

[0825] The "message generating means" is a device that generates a message in natural language based on the determined intention.

[0826] "User interface means" is a device for displaying the generated message on a terminal.

[0827] The "means for accepting user input" is an interface device for accepting instructions and inputs from the owner.

[0828] A "voice synthesis means" is a device for analyzing user input received by the server and generating voice instructions.

[0829] "Audio output means" refers to a device for reproducing the generated audio instructions.

[0830] The "abnormal behavior analysis means" is a device that analyzes abnormal behavior of a pet and generates a warning message based on the analysis.

[0831] The "push notification means" is a device that transmits the generated warning message to the owner's terminal in real time.

[0832] The "image recognition model" is an AI model that analyzes acquired video data and determines a pet's facial expressions and behavior.

[0833] A "voice recognition model" is an AI model that analyzes acquired voice data and identifies the voice of a pet.

[0834] This system facilitates smooth communication between pets and their owners, and also has a security function that sends instant notifications if something unusual happens to the pet. It mainly consists of the following elements:

[0835] 1. System Overview

[0836] 1.1. Sensor means

[0837] The device is equipped with a camera and microphone that captures the pet's facial expressions, movements, and voice in real time. The camera and microphone are controlled using the OpenCV library.

[0838] 1.2. Pre-processing methods

[0839] The preprocessing means in the terminal divides the acquired video data into frames and removes noise. The preprocessed data is sent to the server using the communication means described below.

[0840] 1.3. Means of communication

[0841] The preprocessed data is sent to the server via Wi-Fi or mobile communication, using the Requests library.

[0842] 1.4. Server Means

[0843] The server is a computer system for advanced analysis, analyzing data using image and voice recognition models to determine the pet's intentions and abnormal behavior. The server runs the AI ​​models using the Keras / TensorFlow library.

[0844] 1.5. Message Generation and User Interface Methods

[0845] The server generates a natural language message based on the intention or abnormal behavior determined by the server and sends it to the device, which displays the message to the owner through a user interface and prompts them to take the necessary action.

[0846] 1.6. Abnormal behavior analysis means

[0847] The system analyzes pet behavior data and generates a warning message if it detects abnormal behavior. This message is then pushed to the owner's device in real time.

[0848] 2. Processing Flow

[0849] The server performs analysis using image recognition models and voice recognition models. Based on the analysis results, it generates natural language messages and warning messages, and generates voice instructions using a voice synthesis means as needed. The generated voice instructions are played back by the voice output means of the terminal.

[0850] 3. Specific Examples

[0851] Example 1: When your pet exhibits abnormal behavior

[0852] 1. The device's camera and microphone record your pet's behavior.

[0853] 2. The preprocessing means preprocesses the data and sends it to the server.

[0854] 3. The server analyzes the data and detects any abnormal behavior.

[0855] 4. A warning message is generated and pushed to the owner's device.

[0856] Prompt Sentence Examples

[0857] "Your pet is barking anxiously. Abnormal behavior may have been detected. Owner should check immediately."

[0858] Example 2: When an owner says to their pet, "Let's go for a walk"

[0859] 1. The owner inputs the information using voice or text.

[0860] 2. The user interface means sends the input to the server.

[0861] 3. The server analyzes the data and generates voice instructions using a voice synthesis means.

[0862] 4. The generated voice instructions are played on the device.

[0863] In this way, the system of the present invention not only monitors the condition of pets in real time even when the owner is away, but also sends prompt notifications when an abnormality is detected, providing an effective way to maintain the safety and health of pets.

[0864] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0865] Step 1:

[0866] The device uses a camera and microphone to capture the pet's facial expressions, gestures, and voice in real time. The input is video frames and audio data. Specifically, the device uses the OpenCV library to control the camera and capture video. At the same time, it uses the microphone to capture audio data.

[0867] Step 2:

[0868] The preprocessing means in the terminal divides the acquired video data into frames and removes noise. The input is the video frames and audio data acquired in step 1. Specifically, the video data is resized to the required resolution and divided into frames. Noise is also removed from the audio data. The output is the preprocessed video frames and audio data.

[0869] Step 3:

[0870] The communication means sends the preprocessed data to the server. The input is the video frame and audio data preprocessed in step 2. Specifically, the device uses the Requests library to send the data to the server using an HTTP POST request. The output is the data sent to the server.

[0871] Step 4:

[0872] The server analyzes the received data and determines the pet's intention. The input is the preprocessed data sent in step 3. The server uses the Keras / TensorFlow library to execute image recognition and speech recognition models to determine the pet's intention from its facial expressions, gestures, and voice. The output is the intention data as the analysis result.

[0873] Step 5:

[0874] The server generates a natural language message based on the analysis results. The input is the intent data obtained in step 4. The server uses a generative AI model to generate a natural language message based on the intent data. The output is the generated natural language message.

[0875] Step 6:

[0876] The message generation means transmits the generated message to the terminal. The input is the natural language message generated in step 5. The message is transmitted to the owner's terminal via the communication means. The output is the message displayed on the owner's terminal.

[0877] Step 7:

[0878] The terminal displays the received message to the owner using a user interface. The input is the message received in step 6. As a specific operation, the terminal displays the message on a GUI (Graphical User Interface). The output is the message displayed to the owner.

[0879] Step 8:

[0880] The user provides input through the user interface. The input can be text or voice, such as instructions or questions. The user interface means transmits this input to the server. The output is the user input transmitted to the server.

[0881] Step 9:

[0882] The server analyzes input from the user and generates appropriate voice instructions. The input is the user input, such as the instruction or question submitted in step 8. The server analyzes the text using a natural language processing model and generates the voice instructions using a speech synthesis means. The output is the generated voice instructions.

[0883] Step 10:

[0884] The generated voice instruction is transmitted to the terminal. The input is the voice instruction generated in step 9. The voice instruction is transmitted to the terminal through the communication means. The output is the voice instruction transmitted to the terminal.

[0885] Step 11:

[0886] The terminal plays the received voice instruction using the voice output means. The input is the voice instruction received in step 10. In particular, the terminal plays the voice instruction through a speaker. The output is the played voice instruction.

[0887] Step 12:

[0888] The server periodically analyzes pet behavior data and generates a warning message if it detects abnormal behavior. The input is preprocessed video and audio data. If abnormal behavior is detected, the server immediately generates a warning message and describes it in natural language using a generative AI model. The output is a warning message based on the abnormal behavior.

[0889] Step 13:

[0890] The warning message generated by the server is sent to the owner's device using a push notification means. The input is the warning message generated in step 12. The push notification server sends it to the owner's smartphone or other device. The output is the warning message received by the owner's device.

[0891] Step 14:

[0892] The owner's device displays the received warning message on the user interface and immediately notifies the owner of the abnormal behavior. The input is the warning message received in step 13. As a specific operation, the device immediately warns the owner by using an alert, notification sound, banner display, etc. The output is the warning message displayed to the owner.

[0893] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0894] This invention is a system that analyzes a pet's facial expressions, voice, and behavior, and uses that information to enable effective communication between the owner and the pet. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, richer communication is realized. The system configuration and specific operation are explained below.

[0895] 1. System Configuration

[0896] 1.1. Sensor means

[0897] The sensor means installed in the terminal includes a camera and microphone that detect and collect the voice, facial expressions, and behavior of the pet, allowing you to obtain the pet's status in real time.

[0898] 1.2. Pre-processing methods

[0899] The preprocessing means in the terminal preprocesses the acquired data in order to properly analyze it. For example, video data is divided into frames and noise is removed. Audio data is also preprocessed in the same way.

[0900] 1.3. Means of communication

[0901] The terminal transmits the preprocessed data to the server using a communication means, which can be Wi-Fi or mobile communication.

[0902] 1.4. Server Means

[0903] The server means uses a computer system with advanced computing power, which inputs the received data into an image recognition model or a voice recognition model and analyzes the data.

[0904] 1.5. Message Creation Methods

[0905] The message generation means in the server generates a message in natural language based on the analysis results. This message is intended to convey the pet's intentions to the owner in an easy-to-understand manner.

[0906] 1.6. User Interface Methods

[0907] The terminal displays the messages sent from the server to the owner via the user interface means, allowing the owner to easily check the status and needs of their pet.

[0908] 1.7. Speech synthesis methods

[0909] The voice synthesis means in the server converts the owner's input into voice data, allowing the owner to give voice instructions to their pet.

[0910] 1.8. Audio Output Methods

[0911] The audio output means installed in the terminal reproduces the received audio data and conveys instructions to the pet.

[0912] 1.9. Emotion Engine

[0913] The emotion engine includes technology to analyze the user's voice and facial expressions and recognize their emotions, allowing the system to understand the user's emotional state and reflect it in message generation and voice instructions.

[0914] 2. System Operation

[0915] Example 1: When your pet needs to go to the toilet

[0916] 1. Device: The device's camera and microphone detect the pet's facial expressions and gestures and collect video and audio data.

[0917] 2. Terminal: The preprocessing means divides the collected video data into frames and removes noise. The audio data is also preprocessed in the same way.

[0918] 3. Terminal: Uses communication means to send preprocessed data to the server.

[0919] 4. Server: The server receives the data sent from the device and temporarily stores it in a database.

[0920] 5. Server: The received video data is input into an image recognition model (e.g., CNN) to analyze the pet's facial expressions and behaviors. Specifically, it extracts features such as the pet's ears, tail, and posture.

[0921] 6. Server: Input the audio data into a speech recognition model (e.g., RNN or LSTM) to analyze the tone and patterns of the pet's voice.

[0922] 7. Server: Integrates the results of image and voice recognition to determine the pet's intention. In this case, the intention is determined to be "I want to go to the toilet."

[0923] 8. Server: The message generation means generates a natural language message saying "I want to go to the toilet."

[0924] 9. Server: Sends the generated message to the terminal.

[0925] 10. Terminal: The terminal receives the message sent from the server and displays it on the user interface.

[0926] Example 2: When an owner says to their pet, "Let's go for a walk"

[0927] 1. User: The owner types "Let's go for a walk" in the chat on the LINE app, or gives the command "Let's go for a walk" by voice.

[0928] 2. Device: The device receives the text message and voice data and performs preprocessing. The voice data is converted to text.

[0929] 3. Terminal: Uses communication means to send the acquired data to the server.

[0930] 4. Server: The server receives the data sent from the device and begins analyzing it.

[0931] 5. Server: Analyzes the voice instruction data and understands the instruction "Let's go for a walk."

[0932] 6. Server: The speech synthesis means converts the text data into speech data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[0933] 7. Server: Sends the generated audio data to the device.

[0934] 8. Device: The device receives the voice data from the server and plays it through the speaker, instructing the pet to "go for a walk."

[0935] 9. Server: The emotion engine analyzes the user's voice and facial expressions to recognize their emotions. For example, it can distinguish between commands from a happy owner and commands from a tired owner.

[0936] 10. Server: Based on the results of the emotion engine, adjusts message generation and voice synthesis, and conveys instructions to the pet in an appropriate tone.

[0937] In this way, the system of the present invention enables smooth and intuitive communication between pets and their owners by linking the terminal and server, and also enables more effective interactions by taking the user's emotions into consideration.

[0938] The processing flow will be explained below.

[0939] Example 1: When your pet needs to go to the toilet

[0940] Step 1:

[0941] Terminal

[0942] The device's camera and microphone detect your pet's facial expressions and movements in real time, collecting video and audio data.

[0943] Step 2:

[0944] Terminal

[0945] The pre-processing means divides the collected video data into frames and removes noise, and also pre-processes the audio data in the same way.

[0946] Step 3:

[0947] Terminal

[0948] The communication means prepares the pre-processed data for transmission to the server, where it is standardized and converted into a format suitable for communication.

[0949] Step 4:

[0950] Terminal

[0951] A communication means is used to transmit the pre-processed data to a server.

[0952] Step 5:

[0953] server

[0954] The server receives the data sent from the terminal and temporarily stores it in a database.

[0955] Step 6:

[0956] server

[0957] The received video data is input into an image recognition model (e.g., CNN) to analyze the pet's facial expressions and behaviors. Specifically, features such as the pet's ears, tail, and posture are extracted.

[0958] Step 7:

[0959] server

[0960] The audio data is input into a speech recognition model (e.g., RNN or LSTM) to analyze the tone and patterns of the pet's voice.

[0961] Step 8:

[0962] server

[0963] The results of image and voice recognition are combined to comprehensively determine the pet's intention. In this case, the intention is determined to be "I want to go to the toilet."

[0964] Step 9:

[0965] server

[0966] The message generating means generates a natural language message saying "I want to go to the toilet."

[0967] Step 10:

[0968] server

[0969] Send the generated message to the terminal.

[0970] Step 11:

[0971] Terminal

[0972] The terminal receives the message sent from the server and displays it on the user interface.

[0973] Step 12:

[0974] User

[0975] The user (owner) checks the notification on the device and understands that the pet wants to go to the toilet.

[0976] Example 2: When an owner says to their pet, "Let's go for a walk"

[0977] Step 1:

[0978] User

[0979] The owner can either type "Let's go for a walk" in the chat on the LINE app or give the command "Let's go for a walk" by voice.

[0980] Step 2:

[0981] Terminal

[0982] The device receives text messages and voice data, performs preprocessing, and converts the voice data into text through speech recognition.

[0983] Step 3:

[0984] Terminal

[0985] The acquired text message and voice data are sent to a server using a communication means.

[0986] Step 4:

[0987] server

[0988] The server receives the data sent from the terminal and begins analyzing it.

[0989] Step 5:

[0990] server

[0991] It analyzes received text messages and voice data and understands the instructions such as "Let's go for a walk."

[0992] Step 6:

[0993] server

[0994] A voice synthesis means converts the text data into voice data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[0995] Step 7:

[0996] server

[0997] The generated voice data is transmitted to the terminal.

[0998] Step 8:

[0999] Terminal

[1000] The device receives voice data from the server and plays it through the speaker, instructing your pet to "go for a walk."

[1001] Step 9:

[1002] server

[1003] The emotion engine analyzes the user's voice and facial expressions to recognize their emotions. For example, it can distinguish between commands from a happy owner and commands from a tired owner.

[1004] Step 10:

[1005] server

[1006] Based on the results of the emotion engine, message generation and voice synthesis are adjusted to convey instructions to pets in an appropriate tone.

[1007] In this way, the system of the present invention enables smooth and intuitive communication between pets and their owners by linking the terminal and server, and also enables more effective interactions by taking the user's emotions into consideration.

[1008] Example 2

[1009] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1010] In today's communication between pets and their owners, it is difficult to accurately understand the pet's intentions and emotions, which often results in delayed appropriate responses. Furthermore, when an owner gives a command to a pet, the pet may not accurately understand the owner's intentions, making effective communication difficult. To solve these issues, a system is needed that can accurately detect and analyze a pet's condition and appropriately communicate the results to the owner.

[1011] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a sensor means for detecting and collecting the pet's status, a preprocessing means for dividing the acquired video data into frames and removing noise, a communication means for transmitting the preprocessed data to the server, a calculation means for analyzing the data and determining the pet's intention, a message generation means for generating a message in natural language based on the determined intention, a communication means for transmitting the generated message to the terminal, a user interface means for displaying the received message, an input means for accepting user input, a communication means for transmitting the user input to the server, a synthesis means for analyzing the user input received by the server and generating voice instructions, a communication means for transmitting the generated voice instructions to the terminal, an output means for playing back the received voice instructions, and an emotion recognition means for analyzing the user's voice and facial expressions to recognize emotions. This enables smooth and intuitive communication between the pet and its owner.

[1012] "Sensor means" refers to means that use devices such as cameras and microphones to detect and collect information about the condition of pets.

[1013] The "preprocessing means" is a means for processing the acquired data into a form that is easier to analyze. Specifically, the preprocessing means divides the video data into frames and performs processing to remove noise.

[1014] "Communication means" refers to the means for sending and receiving preprocessed data and generated messages between servers and terminals, and uses Wi-Fi or mobile communications.

[1015] The "calculation means" is a means for analyzing the received data and determining the pet's intentions, and uses an image recognition model or a voice recognition model.

[1016] The "message generating means" is a means for generating a natural language message that conveys the pet's intention based on the analysis results.

[1017] "User interface means" refers to means for displaying received messages to the owner, allowing the owner to check the status and needs of their pet.

[1018] The "input means" is a means for accepting instructions or input from the user, and accepts text or voice input.

[1019] A "synthesizer" is a means for analyzing a user's input and generating appropriate voice instructions.

[1020] The "output means" refers to a means for reproducing the generated voice instructions, and uses a device such as a speaker.

[1021] The "emotion recognition means" is a means for analyzing the user's voice and facial expression and recognizing the user's emotion.

[1022] MODE FOR CARRYING OUT THE INVENTION

[1023] The present invention is a system that analyzes a pet's facial expressions, voice, and behavior, and uses that information to enable effective communication between the owner and the pet. This system includes multiple technical elements that operate in conjunction with a terminal and a server. The system configuration and specific operation are described below.

[1024] System Configuration

[1025] Sensor means

[1026] The device is equipped with a camera and microphone, which detect and collect information about your pet's condition in real time. For example, the camera captures images of your pet's ears, tail, and posture, while the microphone records their cries and barks.

[1027] Pretreatment means

[1028] The pre-processing means within the device has the function of processing the collected data into a form that is easier to analyze. Specifically, it divides the video data into frames and removes noise. It also performs noise removal processing on audio data. For example, the device divides the video into frames every few seconds and automatically adjusts the brightness and color tone from each frame.

[1029] communication means

[1030] The device has a communication means to send the preprocessed data to the server. This communication means can be Wi-Fi or mobile communication. For example, the device checks for a Wi-Fi connection and uploads the data to the server via a stable network.

[1031] Server Means

[1032] The server is a computer system with advanced computing power that inputs the received data into an image recognition model (e.g., CNN) or a voice recognition model (e.g., RNN or LSTM) to analyze the pet's facial expressions and behaviors. Specifically, the server uses a deep learning model to detect the position of the ears and tail from the video and stress or happiness from the voice.

[1033] Message Generation Method

[1034] The message generation means in the server generates and transmits a natural language message that conveys the pet's intentions based on the analysis results, such as "I want to go to the toilet" or "I want to play."

[1035] User Interface Means

[1036] The device displays messages sent from the server to the owner through a user interface, allowing the owner to easily check the pet's status and needs. For example, messages can be displayed as a pop-up notification on a smartphone or tablet.

[1037] Voice synthesis means

[1038] The server's voice synthesis means converts user input into voice data, allowing users to give voice commands to their pets. For example, if an owner sends a message to their pet via the LINE app saying "Let's go for a walk," the server converts the message into natural-sounding voice.

[1039] Audio output means

[1040] The device's built-in voice output means plays back the received voice data to communicate instructions to your pet. For example, a natural voice will be played from the device's speaker saying, "Let's go for a walk."

[1041] emotion recognition means

[1042] The emotion recognition means includes technology that analyzes the user's voice and facial expressions and recognizes their emotions. This allows the system to understand the user's emotional state and reflect it in message generation and voice instructions. For example, the system can distinguish between instructions from an owner who is happy and instructions from an owner who is tired and communicate them to the pet in an appropriate tone.

[1043] Specific examples

[1044] Example 1: When your pet needs to go to the toilet

[1045] The device's camera and microphone detect your pet's facial expressions and gestures, collecting video and audio data.

[1046] The pre-processing means divides the collected video data into frames and removes noise, and also pre-processes the audio data in the same way.

[1047] The preprocessed data is sent to the server using a communication means.

[1048] The data received by the server is input into image and voice recognition models for analysis, and the intent is determined to be "I want to go to the toilet."

[1049] The message generating means generates a natural language message saying "I want to go to the toilet" and transmits it to the terminal.

[1050] The terminal receives the message sent from the server and displays it on the user interface.

[1051] Example 2: When an owner says to their pet, "Let's go for a walk"

[1052] The owner can either type "Let's go for a walk" in the chat on the LINE app or give the command "Let's go for a walk" by voice.

[1053] The device receives text messages and voice data, performs preprocessing, and converts the voice data into text.

[1054] The acquired data is sent to the server using a communication means.

[1055] The server analyzes the received data and understands the instruction "Let's go for a walk."

[1056] A voice synthesis means converts the text data into voice data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[1057] The generated voice data is transmitted to the terminal.

[1058] The device receives voice data from the server and plays it through the speaker, instructing your pet to "go for a walk."

[1059] Examples of prompt statements

[1060] Example inputs to a generative AI model:

[1061] "Please explain how to analyze a pet's gestures and voice to communicate its intentions to its owner."

[1062] "Please explain how the system works to notify pet owners appropriately when their pets behave in a certain way."

[1063] As described above, the system of the present invention enables smooth and effective communication between pets and their owners by linking the terminal and server, and also takes into account the user's emotions, enabling more intuitive and friendly communication.

[1064] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1065] Step 1: Collect pet status

[1066] Subject: Device

[1067] Description: The device uses a camera and microphone to detect your pet's condition (facial expressions, voice, and behavior) in real time and collects this data.

[1068] Input: Your pet's real-life facial expressions, sounds, and gestures

[1069] Output: Collected video and audio data

[1070] Specific operations: The device's camera captures your pet's ears, tail, posture, etc., and the microphone records any meows or barks.

[1071] Step 2: Preprocessing the data

[1072] Subject: Device

[1073] Description: The preprocessing means divides the collected video data into frames and removes noise. The audio data is also subjected to noise removal.

[1074] Input: Collected video and audio data

[1075] Output: Preprocessed video frames and audio data

[1076] What it does: The device splits the video into frames every few seconds, then automatically adjusts the brightness and color of each frame to remove noise.

[1077] Step 3: Sending data

[1078] Subject: Device

[1079] Description: The pre-processed data is sent to the server via a communication method, in this case Wi-Fi or mobile communication.

[1080] Input: Preprocessed video frames and audio data

[1081] Output: Data sent to the server

[1082] What happens: The device checks for a Wi-Fi connection and uploads data to the server over a stable network.

[1083] Step 4: Analyze the data

[1084] Subject: Server

[1085] Description: The server inputs the received data into an image recognition model (e.g., CNN) or a speech recognition model (e.g., RNN or LSTM) for analysis.

[1086] Input: Preprocessed data sent to the server

[1087] Output: Parsed pet's intention and state information

[1088] Specific operation: The server uses a deep learning model to detect the position of the ears and tail from video footage, and emotions and intentions such as stress or joy from audio.

[1089] Step 5: Generate a message

[1090] Subject: Server

[1091] Description: Based on the analysis results, the message generator generates a natural language message that conveys the pet's intentions.

[1092] Input: Parsed pet intent and state information

[1093] Output: Natural language message

[1094] Concrete behavior: The process of generating simple messages such as "I want to go to the toilet" or "I want to play."

[1095] Step 6: Sending a message

[1096] Subject: Server (terminal)

[1097] Description: The server generates a message and sends it to the terminal, which receives it.

[1098] Input: A natural language message

[1099] Output: Message sent to the terminal

[1100] Specific operation: The server sends a message to the terminal, and the terminal receives the message.

[1101] Step 7: Displaying messages

[1102] Subject: Device

[1103] Description: Displays received messages to the owner through user interface means.

[1104] Input: Message sent to terminal

[1105] Output: The message displayed in the user interface

[1106] What happens: A message will pop up through your smartphone or tablet's notification feature.

[1107] Step 8: Accepting User Input

[1108] Subject: User

[1109] Description: The user enters instructions in the form of speech or text.

[1110] Input: User instructions (voice or text message)

[1111] Output: User's instruction data

[1112] Specific actions: The owner types "Let's go for a walk" in the chat on the LINE app, or gives voice instructions.

[1113] Step 9: Sending User Input

[1114] Subject: Device

[1115] Description: Preprocesses input from the user and sends it to the server via a communication medium.

[1116] Input: User-specified data

[1117] Output: User instruction data sent to the server

[1118] Specific operation: The process of converting input voice data into text and sending it to the server.

[1119] Step 10: Generate voice instructions

[1120] Subject: Server

[1121] Description: The server analyzes the user's input and generates appropriate voice instructions using a speech synthesis method.

[1122] Input: User instruction data sent to the server

[1123] Output: Generated voice instruction data

[1124] Specific operation: Convert the text message "Let's go for a walk" into voice data with natural pronunciation.

[1125] Step 11: Sending voice instructions

[1126] Subject: Server (terminal)

[1127] Description: The server generates and sends voice instructions to the device, which receives them.

[1128] Input: Generated voice command data

[1129] Output: Audio instructions sent to the device

[1130] Specific operation: The server sends voice data to the terminal, and the terminal receives the data.

[1131] Step 12: Play audio instructions

[1132] Subject: Device

[1133] Description: Plays back the received audio instructions through the audio output means.

[1134] Input: Voice commands sent to the device

[1135] Output: The audio instructions played

[1136] Specific operation: "Let's go for a walk" is played in a natural voice from the device's speaker.

[1137] (Application example 2)

[1138] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1139] Currently, stores that cater to customers with pets need to quickly and accurately understand the pet's condition and respond appropriately. However, it is difficult to understand the pet's intentions in real time, which can lead to delayed responses, especially in emergencies. In addition, smooth communication between pets, owners, and store staff is required, but an efficient system to achieve this does not exist.

[1140] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for determining the pet's intention, a notification means for sending a notification to the user and staff, and a means for analyzing the pet's condition using an image recognition model and a voice recognition model. This makes it possible to monitor the pet's condition in real time and take appropriate measures, including emergency notifications. It also enables the user and staff to quickly understand the pet's condition and take appropriate measures.

[1141] "Pets" are animals kept by humans, such as dogs and cats.

[1142] "Audio" refers to the voices and sounds made by pets.

[1143] "Expression" refers to the facial expressions and changes in your pet's face.

[1144] "Gestures" refer to the body movements and actions of pets.

[1145] "Sensor means" refers to a device such as a camera or microphone for detecting the voice, facial expressions, and behavior of a pet.

[1146] "Preprocessing means" refers to means for converting collected data into a form that is easy to analyze.

[1147] "Communication means" refers to a means for transmitting pre-processed data to a server.

[1148] The "server means" refers to a means for analyzing the received data and determining the intention of the pet.

[1149] The "message generating means" refers to a means for generating a message in natural language based on the determined intention.

[1150] "User interface means" refers to means for displaying generated messages to a user.

[1151] A "voice synthesis means" is a means for analyzing input from a user and generating voice instructions.

[1152] "Audio output means" refers to means for reproducing the generated audio instructions.

[1153] "Notification means" refers to a means for determining the pet's condition and sending notifications to the user and staff.

[1154] An "image recognition model" is a machine learning model for analyzing pets' facial expressions and behaviors.

[1155] A "voice recognition model" is a machine learning model for analyzing pet voices.

[1156] This invention is a support system for customers who bring pets to physical stores, and as a specific example, we will explain a "pet support app." This application has the function of monitoring the pet's condition in real time and sending appropriate notifications to users and staff.

[1157] Hardware and software used

[1158] Hardware

[1159] Camera: Used to capture your pet's expressions and behavior in real time.

[1160] Microphone: Used to record your pet's voice.

[1161] Smartphone: The above sensors (camera and microphone) are built in.

[1162] Server: A high-performance computer system that analyzes data and stores results.

[1163] software

[1164] OpenCV: A library for acquiring and preprocessing video data from a camera.

[1165] TensorFlow / Keras: Machine learning libraries for implementing image recognition models to analyze pet facial expressions and behaviors.

[1166] SpeechRecognition: A library for converting your pet's spoken commands into text.

[1167] Requests: A library for sending preprocessed data to a server.

[1168] Data processing and calculation

[1169] Terminal handling

[1170] The device activates the camera and microphone to capture the pet's facial expressions, gestures, and voice in real time. Then, it splits the video data into frames and performs preprocessing using OpenCV. It also converts the voice data into text using SpeechRecognition. This preprocessed data is sent to the server using Requests.

[1171] Server Processing

[1172] The server analyzes the received data in real time. It uses a TensorFlow / Keras-based image recognition model to extract features from the pet's facial expressions and gestures and determine the pet's intentions. It also uses a voice recognition model to analyze the pet's voice patterns. These analysis results are combined to determine the pet's condition. The determination result is generated as a natural language message by a message generation means and sent to the terminal.

[1173] Specific examples

[1174] Example 1: When your pet needs to go to the toilet

[1175] The device's camera and microphone capture the pet's state and perform preprocessing.

[1176] The server analyzes the data and determines the intention to "go to the toilet."

[1177] The result of the judgment is notified to the user and staff, and appropriate action is taken.

[1178] Example 2: When the owner says, "Let's go for a walk"

[1179] The user types in the text "Let's go for a walk" or gives the voice command.

[1180] The device preprocesses the data and sends it to the server.

[1181] The server analyzes the instructions, generates appropriate voice instructions, and sends them to the terminal.

[1182] The device will play audio instructions and communicate them to your pet.

[1183] Prompt Sentence Examples

[1184] Message prompt to display in the user interface:

[1185] When a pet's condition is emergency, generate the following message: 'Your pet is in [condition]. Please take appropriate action.'

[1186] Staff notification message prompt:

[1187] Generate an emergency notification message to send to staff: 'Pet is in [condition]. Please respond immediately.'

[1188] In this way, the system of the present invention enables support for customers with pets by linking the server and terminals to quickly and accurately grasp the condition of pets and provide appropriate notifications to users and staff.

[1189] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1190] Step 1:

[1191] The device activates the camera and microphone to capture the pet's facial expressions, gestures, and voice in real time. The input is the camera image and the microphone audio, and the output is image data and voice data for each frame.

[1192] Step 2:

[1193] The device uses OpenCV to preprocess the video data. Specifically, it splits it into frames and performs noise reduction and resizing as needed. The input is the captured image data for each frame, and the output is the preprocessed image data.

[1194] Step 3:

[1195] The device uses SpeechRecognition to preprocess the audio data, specifically converting it to text. The input is the captured audio data, and the output is text data.

[1196] Step 4:

[1197] The terminal uses Requests to send preprocessed image data and text data to the server. The input is the preprocessed image data and text data, and the output is the data sent to the server.

[1198] Step 5:

[1199] The server analyzes the received image data using a TensorFlow / Keras-based image recognition model. Specifically, it extracts features from the pet's facial expressions and behavior. The input is the preprocessed image data, and the output is the analysis results.

[1200] Step 6:

[1201] The server uses a speech recognition model to analyze the received text data. Specifically, it analyzes the pet's voice patterns and intentions from the text data. The input is the preprocessed text data, and the output is the analysis results.

[1202] Step 7:

[1203] The server integrates the results of image and voice recognition analysis to comprehensively judge the pet's condition. The input is the results of image and voice analysis, and the output is a judgment result indicating the pet's condition.

[1204] Step 8:

[1205] The server generates a natural language message based on the judgment result. Using the message generation means, the judgment result is converted into a text message that is easy for users and staff to understand. The input is the judgment result, and the output is a natural language message.

[1206] Step 9:

[1207] The server sends the generated message to the terminal. The input is the generated natural language message, and the output is the message sent to the terminal.

[1208] Step 10:

[1209] The terminal displays the received messages to the user through a user interface means, where the input is the message sent from the server and the output is the message displayed to the user.

[1210] Step 11:

[1211] The user inputs commands to the pet through the terminal. For example, "Let's go for a walk" is given by voice or text. The input is the user's command, and the output is text or voice data.

[1212] Step 12:

[1213] The terminal sends text data and voice data to the server. The input is the text or voice data containing the user's instructions, and the output is the data sent to the server.

[1214] Step 13:

[1215] The server analyzes the received instructions and generates voice instructions for the pet using a voice synthesis means. The input is the user's instruction data, and the output is the synthesized voice instruction.

[1216] Step 14:

[1217] The server sends the synthesized voice instructions to the terminal, where the input is the generated voice instructions and the output is the voice data sent to the terminal.

[1218] Step 15:

[1219] The terminal uses the audio output means to play audio instructions to the pet, with the input being audio data sent from the server and the output being the played audio instructions.

[1220] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1221] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1222] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1223] [Third embodiment]

[1224] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1225] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1226] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1227] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1228] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1229] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1230] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1231] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1232] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1233] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1234] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1235] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1236] The present invention is a system that analyzes the facial expressions, sounds, and behaviors of pets and enables communication between owners and pets based on this information. The configuration and specific operation of the system are described below.

[1237] 1. System Configuration

[1238] 1.1. Sensor means

[1239] The sensor means installed in the terminal includes a camera and microphone for detecting and collecting the voice, facial expressions, and behavior of the pet, thereby enabling the status of the pet to be acquired in real time.

[1240] 1.2. Pre-processing methods

[1241] The pre-processing means in the terminal pre-processes the acquired data in order to properly analyze it, for example, by dividing the video data into frames and removing noise.

[1242] 1.3. Means of communication

[1243] The terminal transmits the preprocessed data to the server using a communication means, which can be Wi-Fi or mobile communication.

[1244] 1.4. Server Means

[1245] The server uses a computer system with advanced computing power to analyze the received data, which is then input into an image recognition model or a voice recognition model for analysis.

[1246] 1.5. Message Creation Methods

[1247] The message generation means in the server generates a message in natural language based on the analysis results. This message is intended to convey the pet's intentions to the owner in an easy-to-understand manner.

[1248] 1.6. User Interface Methods

[1249] The terminal displays the messages sent from the server to the owner via the user interface means, allowing the owner to easily check the status and needs of their pet.

[1250] 1.7. Speech synthesis methods

[1251] The voice synthesis means in the server converts the owner's input into voice data, allowing the owner to give voice instructions to their pet.

[1252] 1.8. Audio Output Methods

[1253] The audio output means installed in the terminal reproduces the received audio data and conveys instructions to the pet.

[1254] 2. System Operation

[1255] Example 1: When your pet needs to go to the toilet

[1256] 1. Device: The device's camera and microphone detect your pet's facial expressions and behaviors and collect data.

[1257] 2. Terminal: The preprocessing means preprocesses the data and prepares it for transmission to the server.

[1258] 3. Terminal: Uses communication means to send the preprocessed data to the server.

[1259] 4. Server: The server inputs the received data into image and voice recognition models to analyze the pet's intentions.

[1260] 5. Server: Based on the analysis results, generate a message saying "I want to go to the toilet."

[1261] 6. Server: Sends the generated message to the terminal.

[1262] 7. Terminal: Displays the received message to the owner using a user interface.

[1263] Example 2: When an owner says to their pet, "Let's go for a walk"

[1264] 1. User: The owner types "Let's go for a walk" in the chat on the LINE app or gives voice instructions.

[1265] 2. Terminal: Sends the owner's input text and voice data to the server.

[1266] 3. Server: Analyzes the received data and generates audio data as needed.

[1267] 4. Server: Uses a speech synthesis tool to generate the speech "Let's go for a walk."

[1268] 5. Server: Sends the generated voice data to the device.

[1269] 6. Terminal: The received voice data is played back using the voice output means, and instructions are conveyed to the pet.

[1270] As described above, the system of the present invention is designed to realize smooth and intuitive communication between pets and their owners, allowing them to quickly understand the pet's condition and needs and take the necessary measures.

[1271] The processing flow will be explained below.

[1272] Processing flow

[1273] Example 1: When your pet needs to go to the toilet

[1274] Step 1:

[1275] Terminal

[1276] The device's camera and microphone detect your pet's facial expressions and movements in real time, collecting video and audio data.

[1277] Step 2:

[1278] Terminal

[1279] The pre-processing means divides the collected video data into frames and removes noise, and also pre-processes the audio data in the same way.

[1280] Step 3:

[1281] Terminal

[1282] The communication means prepares the pre-processed data for transmission to the server, where it is standardized and converted into a format suitable for communication.

[1283] Step 4:

[1284] server

[1285] The server receives the data sent from the terminal and temporarily stores it in a database.

[1286] Step 5:

[1287] server

[1288] The received video data is input into an image recognition model (e.g., CNN) to analyze the pet's facial expressions and behaviors. Specifically, features such as the pet's ears, tail, and posture are extracted.

[1289] Step 6:

[1290] server

[1291] The audio data is input into a speech recognition model (e.g., RNN or LSTM) to analyze the tone and patterns of the pet's voice.

[1292] Step 7:

[1293] server

[1294] The results of image and voice recognition are combined to comprehensively determine the pet's intention. In this case, the intention is determined to be "I want to go to the toilet."

[1295] Step 8:

[1296] server

[1297] The message generating means generates a natural language message saying "I want to go to the toilet."

[1298] Step 9:

[1299] server

[1300] Send the generated message to the terminal.

[1301] Step 10:

[1302] Terminal

[1303] The terminal receives the message sent from the server and displays it on the user interface.

[1304] Example 2: When an owner says to their pet, "Let's go for a walk"

[1305] Step 1:

[1306] User

[1307] The owner can either type "Let's go for a walk" in the chat on the LINE app or give the command "Let's go for a walk" by voice.

[1308] Step 2:

[1309] Terminal

[1310] The device receives text messages and voice data, performs preprocessing, and converts the voice data into text.

[1311] Step 3:

[1312] Terminal

[1313] The acquired data is sent to the server using a communication means.

[1314] Step 4:

[1315] server

[1316] The server receives the data sent from the terminal and begins analyzing it.

[1317] Step 5:

[1318] server

[1319] It analyzes voice command data and understands the command "Let's go for a walk."

[1320] Step 6:

[1321] server

[1322] A voice synthesis means converts the text data into voice data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[1323] Step 7:

[1324] server

[1325] The generated voice data is transmitted to the terminal.

[1326] Step 8:

[1327] Terminal

[1328] The device receives voice data from the server and plays it through the speaker, instructing your pet to "go for a walk."

[1329] In this way, the system of the present invention allows the terminal and server to work together to realize smooth communication between pets and their owners.

[1330] Example 1

[1331] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1332] In general, communication between pets and owners is difficult to achieve quickly and accurately because it is difficult to understand the intentions and state of the pet. In particular, owners often cannot adequately meet their pets' needs because they do not have a way to accurately grasp what their pets want and communicate it in human language.

[1333] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1334] In this invention, the server includes sensor means for acquiring the voice, facial expressions, and behaviors of the pet, data processing means for preprocessing the acquired data, communication means for transmitting the preprocessed data to the server, server means for analyzing the data and determining the intention of the pet, message generation means for generating a message in natural language based on the determined intention, data transmission means for transmitting the generated message to the terminal, user interface means for displaying the received message, input means for accepting user input, additional communication means for transmitting the user input to the server, voice synthesis means for analyzing the user input received by the server and generating voice instructions, additional data transmission means for transmitting the generated voice instructions to the terminal, and voice output means for playing back the received voice instructions. This allows the owner to quickly and accurately grasp the condition and needs of the pet and take appropriate measures.

[1335] The "sensor means" is a device for acquiring the voice, facial expressions, and behavior of a pet.

[1336] "Data processing means" refers to a device or software for preprocessing acquired data.

[1337] "Communication means" is a device or software for transmitting pre-processed data to a server.

[1338] The "server means" is a system for analyzing data on a server and determining the intentions of the pet.

[1339] A "message generator" is a device or software for generating a natural language message based on the determined intent.

[1340] The "data transmission means" is a function for transmitting the generated message to the terminal.

[1341] "User interface means" is a device or software for displaying received messages.

[1342] "Input means" refers to a device or software for accepting user input.

[1343] "Additional communication means" is a function for sending input from the user to the server.

[1344] A "voice synthesis means" is a device or software for analyzing user input received at the server and generating voice instructions.

[1345] The "additional data transmission means" is a function for transmitting the generated voice instructions to the terminal.

[1346] "Audio output means" refers to a device or software for reproducing received audio instructions.

[1347] The present invention is a system for analyzing a pet's facial expressions, voice, and behavior, and facilitating smooth communication between the owner and the pet based on the results. This system is realized through the interaction of a terminal, a server, and a user.

[1348] System Configuration

[1349] Sensor means

[1350] The device is equipped with a camera and microphone to capture your pet's facial expressions, voice, and behavior in real time. The camera captures your pet's facial expressions and movements, while the microphone collects barks and other sounds. These sensory tools are important for accurately understanding your pet's condition and behavior.

[1351] Data Processing Means

[1352] The device has a data processing means that preprocesses the acquired data. Specifically, image processing libraries such as OpenCV are used to divide the video data into frames and remove noise. Additionally, collected audio data is filtered to remove noise. This preprocessing converts the data into a format that is easier to analyze.

[1353] communication means

[1354] The device then sends the pre-processed data to the server over Wi-Fi or mobile data, typically using HTTP or WebSocket protocols, which is important to ensure fast and secure data transfer.

[1355] Server Means

[1356] The server has advanced computing power to analyze the received data. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to input the data into image recognition and voice recognition models for analysis. For example, the image recognition model analyzes the pet's facial expressions and gestures, and the voice recognition model analyzes the meaning of its cries.

[1357] Message Generation Method

[1358] The server's message generation means generates natural language messages based on the analysis results. This process uses generative AI models such as GPT-3. For example, if a pet's behavior is interpreted as "I want to go to the toilet," a message to that effect is generated for the owner.

[1359] Data transmission method

[1360] The message generated by the server is sent to the terminal using the data transmission means, which uses the HTTP protocol.

[1361] User Interface Means

[1362] The device displays the received message to the user via a user interface means, for example, as a notification on a smartphone app, allowing the owner to understand the status and needs of their pet.

[1363] Input Method

[1364] The user can give instructions to their pet by voice or text using the smartphone's microphone or keyboard.

[1365] Additional Communication Methods

[1366] The device sends user input to the server using an additional communication method. For example, if an owner types "Let's go for a walk" into the LINE app, the message is sent to the server.

[1367] Voice synthesis means

[1368] The server converts the received instructions from the owner into voice data using a voice synthesis tool. Specifically, it uses a service such as Google Cloud Text-to-Speech to generate voice data such as "Let's go for a walk."

[1369] Additional Data Transmission Methods

[1370] The generated voice data is transmitted to the terminal using additional data transmission means.

[1371] Audio output means

[1372] The device then plays the received voice data via the voice output means to communicate instructions to the pet. For example, a smart speaker might play a voice saying, "Let's go for a walk."

[1373] Specific examples

[1374] Example 1: When your pet needs to go to the toilet

[1375] 1. The device's camera and microphone detect your pet's facial expressions and behaviors and collect data.

[1376] 2. The data processing means of the terminal divides the video data into frames and removes noise.

[1377] 3. The device sends the preprocessed data to the server.

[1378] 4. The server analyzes the received data and generates a message saying "I want to go to the toilet."

[1379] 5. The server sends the generated message to the terminal.

[1380] 6. The device displays the received message to the owner using the user interface.

[1381] Example 2: When an owner says to their pet, "Let's go for a walk"

[1382] 1. The user types "Let's go for a walk" in the chat on the LINE app.

[1383] 2. The device sends the owner's input text to the server.

[1384] 3. The server analyzes the received data and generates audio data.

[1385] 4. The server uses a speech synthesis means to generate the speech "Let's go for a walk."

[1386] 5. The server sends the generated voice data to the terminal.

[1387] 6. The terminal uses the audio output means to play back the received audio data and convey instructions to the pet.

[1388] Prompt Sentence Examples

[1389] Prompt: "Generate a prompt to ask a pet owner if they want to go for a walk."

[1390] Output: "Shall we go for a walk?"

[1391] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1392] Step 1:

[1393] The device uses sensors to detect your pet's facial expressions, voice, and gestures. The input is your pet's real-time movements and voice, which are captured using a camera and microphone. Specifically, the camera captures your pet's facial expressions and movements, and the microphone collects its barks and other sounds. The output is the captured video and audio data.

[1394] Step 2:

[1395] The device preprocesses the acquired data. The preprocessing means divides the video data into frames and removes noise. Similarly, audio data is filtered to remove noise. The processing involves preprocessing the video data using an image processing library such as OpenCV, and preprocessing the audio data using audio filtering techniques. The output is video and audio data in a clean, easy-to-analyze format.

[1396] Step 3:

[1397] The device sends the preprocessed data to the server via a communication means. The input is the preprocessed video and audio data, which is sent via Wi-Fi or mobile communication. The HTTP protocol or WebSocket protocol is used for data transmission. The output is the completion of data transfer to the server.

[1398] Step 4:

[1399] The server analyzes the received data. The input is pre-processed video and audio data. For the analysis, deep learning frameworks such as TensorFlow and PyTorch are used to apply image recognition and voice recognition models. Specifically, the image recognition model analyzes the pet's facial expressions and gestures, while the voice recognition model analyzes the meaning of the pet's cries. The output is an analysis result that indicates the pet's intentions.

[1400] Step 5:

[1401] The server generates a natural language message based on the analysis results. The input is the analysis result indicating the pet's intention. A generative AI model such as GPT-3 is used for this process. Specifically, for example, if the pet's gesture is interpreted as "I want to go to the toilet," a message to that effect is generated for the owner. The output is a message written in natural language.

[1402] Step 6:

[1403] The server sends the generated message to the terminal using a data transmission means. The input is the generated natural language message. The HTTP protocol is used for transmission. The output is the completion of message transfer to the terminal.

[1404] Step 7:

[1405] The terminal displays the received message to the user via the user interface means. The input is a natural language message received from the server. The message is displayed as a notification on the smartphone app. The output is a message displayed to the owner.

[1406] Step 8:

[1407] The user inputs instructions for the pet by voice or text. The smartphone's microphone or keyboard is used as the input means. The input data is text or voice instructions given by the user, such as "Let's take your pet for a walk." The output is instruction data sent from the user to the device.

[1408] Step 9:

[1409] The device sends user input to the server using additional communication means. The input is the user's instruction data (text or voice). This is sent using Wi-Fi or mobile communication. The output is the transfer of instruction data to the server.

[1410] Step 10:

[1411] The server analyzes the received user instructions and converts them into voice data using a voice synthesis means. The input is the user's instruction data. This process uses services such as Google Cloud Text-to-Speech. Specifically, the text instruction "Let's go for a walk" is converted into voice data. The output is voice data.

[1412] Step 11:

[1413] The generated voice data is transmitted to the terminal using an additional data transmission means of the server. The input is the generated voice data, and the HTTP protocol is used for transmission. The output is the transfer of the voice data to the terminal.

[1414] Step 12:

[1415] The terminal plays the received voice data via the voice output means and transmits instructions to the pet. The input is the voice data received from the server. For example, a smart speaker plays the voice "Let's go for a walk." The output is the transmission of instructions to the pet.

[1416] (Application example 1)

[1417] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1418] In today's world, pets are important members of the family, and there is a need to monitor their condition and safety while their owners are away. However, there is no effective system that analyzes a pet's facial expressions, gestures, and voice, detects their intentions and abnormal behavior in real time, and notifies the owner. As a result, if a pet is in trouble while the owner is away, it is difficult to detect the problem early and take appropriate action. There is also a lack of systems that can accurately analyze a pet's condition and issue appropriate instructions. There is a need to solve these issues.

[1419] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1420] In this invention, the server includes an abnormal behavior analysis means for analyzing abnormal behavior of a pet and generating a warning message based on the analysis, a means for sending a push notification of the generated warning message to the owner's device, and a server means for using an image recognition model and a voice recognition model to determine the pet's intentions and abnormal behavior. This allows the owner to monitor the pet's condition in real time even when the owner is away, and to be notified promptly when an abnormality is detected. Furthermore, an effective system for maintaining the safety and health of pets can be provided, providing appropriate countermeasures.

[1421] The "sensor means" is a device including a camera and a microphone for capturing the sounds, facial expressions, and behaviors of a pet.

[1422] The "pre-processing means" is a device that divides the acquired data into frames and performs pre-processing such as removing noise in order to properly analyze the data.

[1423] A "communication means" is a device that uses Wi-Fi or mobile communication to transmit preprocessed data to a server.

[1424] The "server means" is a computer system with advanced computing power for analyzing data and determining the intentions of the pet.

[1425] The "message generating means" is a device that generates a message in natural language based on the determined intention.

[1426] "User interface means" is a device for displaying the generated message on a terminal.

[1427] The "means for accepting user input" is an interface device for accepting instructions and inputs from the owner.

[1428] A "voice synthesis means" is a device for analyzing user input received by the server and generating voice instructions.

[1429] "Audio output means" refers to a device for reproducing the generated audio instructions.

[1430] The "abnormal behavior analysis means" is a device that analyzes abnormal behavior of a pet and generates a warning message based on the analysis.

[1431] The "push notification means" is a device that transmits the generated warning message to the owner's terminal in real time.

[1432] The "image recognition model" is an AI model that analyzes acquired video data and determines a pet's facial expressions and behavior.

[1433] A "voice recognition model" is an AI model that analyzes acquired voice data and identifies the voice of a pet.

[1434] This system facilitates smooth communication between pets and their owners, and also has a security function that sends instant notifications if something unusual happens to the pet. It mainly consists of the following elements:

[1435] 1. System Overview

[1436] 1.1. Sensor means

[1437] The device is equipped with a camera and microphone that captures the pet's facial expressions, movements, and voice in real time. The camera and microphone are controlled using the OpenCV library.

[1438] 1.2. Pre-processing methods

[1439] The preprocessing means in the terminal divides the acquired video data into frames and removes noise. The preprocessed data is sent to the server using the communication means described below.

[1440] 1.3. Means of communication

[1441] The preprocessed data is sent to the server via Wi-Fi or mobile communication, using the Requests library.

[1442] 1.4. Server Means

[1443] The server is a computer system for advanced analysis, analyzing data using image and voice recognition models to determine the pet's intentions and abnormal behavior. The server runs the AI ​​models using the Keras / TensorFlow library.

[1444] 1.5. Message Generation and User Interface Methods

[1445] The server generates a natural language message based on the intention or abnormal behavior determined by the server and sends it to the device, which displays the message to the owner through a user interface and prompts them to take the necessary action.

[1446] 1.6. Abnormal behavior analysis means

[1447] The system analyzes pet behavior data and generates a warning message if it detects abnormal behavior. This message is then pushed to the owner's device in real time.

[1448] 2. Processing Flow

[1449] The server performs analysis using image recognition models and voice recognition models. Based on the analysis results, it generates natural language messages and warning messages, and generates voice instructions using a voice synthesis means as needed. The generated voice instructions are played back by the voice output means of the terminal.

[1450] 3. Specific Examples

[1451] Example 1: When your pet exhibits abnormal behavior

[1452] 1. The device's camera and microphone record your pet's behavior.

[1453] 2. The preprocessing means preprocesses the data and sends it to the server.

[1454] 3. The server analyzes the data and detects any abnormal behavior.

[1455] 4. A warning message is generated and pushed to the owner's device.

[1456] Prompt Sentence Examples

[1457] "Your pet is barking anxiously. Abnormal behavior may have been detected. Owner should check immediately."

[1458] Example 2: When an owner says to their pet, "Let's go for a walk"

[1459] 1. The owner inputs the information using voice or text.

[1460] 2. The user interface means sends the input to the server.

[1461] 3. The server analyzes the data and generates voice instructions using a voice synthesis means.

[1462] 4. The generated voice instructions are played on the device.

[1463] In this way, the system of the present invention not only monitors the condition of pets in real time even when the owner is away, but also sends prompt notifications when an abnormality is detected, providing an effective way to maintain the safety and health of pets.

[1464] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1465] Step 1:

[1466] The device uses a camera and microphone to capture the pet's facial expressions, gestures, and voice in real time. The input is video frames and audio data. Specifically, the device uses the OpenCV library to control the camera and capture video. At the same time, it uses the microphone to capture audio data.

[1467] Step 2:

[1468] The preprocessing means in the terminal divides the acquired video data into frames and removes noise. The input is the video frames and audio data acquired in step 1. Specifically, the video data is resized to the required resolution and divided into frames. Noise is also removed from the audio data. The output is the preprocessed video frames and audio data.

[1469] Step 3:

[1470] The communication means sends the preprocessed data to the server. The input is the video frame and audio data preprocessed in step 2. Specifically, the device uses the Requests library to send the data to the server using an HTTP POST request. The output is the data sent to the server.

[1471] Step 4:

[1472] The server analyzes the received data and determines the pet's intention. The input is the preprocessed data sent in step 3. The server uses the Keras / TensorFlow library to execute image recognition and speech recognition models to determine the pet's intention from its facial expressions, gestures, and voice. The output is the intention data as the analysis result.

[1473] Step 5:

[1474] The server generates a natural language message based on the analysis results. The input is the intent data obtained in step 4. The server uses a generative AI model to generate a natural language message based on the intent data. The output is the generated natural language message.

[1475] Step 6:

[1476] The message generation means transmits the generated message to the terminal. The input is the natural language message generated in step 5. The message is transmitted to the owner's terminal via the communication means. The output is the message displayed on the owner's terminal.

[1477] Step 7:

[1478] The terminal displays the received message to the owner using a user interface. The input is the message received in step 6. As a specific operation, the terminal displays the message on a GUI (Graphical User Interface). The output is the message displayed to the owner.

[1479] Step 8:

[1480] The user provides input through the user interface. The input can be text or voice, such as instructions or questions. The user interface means transmits this input to the server. The output is the user input transmitted to the server.

[1481] Step 9:

[1482] The server analyzes input from the user and generates appropriate voice instructions. The input is the user input, such as the instruction or question submitted in step 8. The server analyzes the text using a natural language processing model and generates the voice instructions using a speech synthesis means. The output is the generated voice instructions.

[1483] Step 10:

[1484] The generated voice instruction is transmitted to the terminal. The input is the voice instruction generated in step 9. The voice instruction is transmitted to the terminal through the communication means. The output is the voice instruction transmitted to the terminal.

[1485] Step 11:

[1486] The terminal plays the received voice instruction using the voice output means. The input is the voice instruction received in step 10. In particular, the terminal plays the voice instruction through a speaker. The output is the played voice instruction.

[1487] Step 12:

[1488] The server periodically analyzes pet behavior data and generates a warning message if it detects abnormal behavior. The input is preprocessed video and audio data. If abnormal behavior is detected, the server immediately generates a warning message and describes it in natural language using a generative AI model. The output is a warning message based on the abnormal behavior.

[1489] Step 13:

[1490] The warning message generated by the server is sent to the owner's device using a push notification means. The input is the warning message generated in step 12. The push notification server sends it to the owner's smartphone or other device. The output is the warning message received by the owner's device.

[1491] Step 14:

[1492] The owner's device displays the received warning message on the user interface and immediately notifies the owner of the abnormal behavior. The input is the warning message received in step 13. As a specific operation, the device immediately warns the owner by using an alert, notification sound, banner display, etc. The output is the warning message displayed to the owner.

[1493] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1494] This invention is a system that analyzes a pet's facial expressions, voice, and behavior, and uses that information to enable effective communication between the owner and the pet. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, richer communication is realized. The system configuration and specific operation are explained below.

[1495] 1. System Configuration

[1496] 1.1. Sensor means

[1497] The sensor means installed in the terminal includes a camera and microphone that detect and collect the voice, facial expressions, and behavior of the pet, allowing you to obtain the pet's status in real time.

[1498] 1.2. Pre-processing methods

[1499] The preprocessing means in the terminal preprocesses the acquired data in order to properly analyze it. For example, video data is divided into frames and noise is removed. Audio data is also preprocessed in the same way.

[1500] 1.3. Means of communication

[1501] The terminal transmits the preprocessed data to the server using a communication means, which can be Wi-Fi or mobile communication.

[1502] 1.4. Server Means

[1503] The server means uses a computer system with advanced computing power, which inputs the received data into an image recognition model or a voice recognition model and analyzes the data.

[1504] 1.5. Message Creation Methods

[1505] The message generation means in the server generates a message in natural language based on the analysis results. This message is intended to convey the pet's intentions to the owner in an easy-to-understand manner.

[1506] 1.6. User Interface Methods

[1507] The terminal displays the messages sent from the server to the owner via the user interface means, allowing the owner to easily check the status and needs of their pet.

[1508] 1.7. Speech synthesis methods

[1509] The voice synthesis means in the server converts the owner's input into voice data, allowing the owner to give voice instructions to their pet.

[1510] 1.8. Audio Output Methods

[1511] The audio output means installed in the terminal reproduces the received audio data and conveys instructions to the pet.

[1512] 1.9. Emotion Engine

[1513] The emotion engine includes technology to analyze the user's voice and facial expressions and recognize their emotions, allowing the system to understand the user's emotional state and reflect it in message generation and voice instructions.

[1514] 2. System Operation

[1515] Example 1: When your pet needs to go to the toilet

[1516] 1. Device: The device's camera and microphone detect the pet's facial expressions and gestures and collect video and audio data.

[1517] 2. Terminal: The preprocessing means divides the collected video data into frames and removes noise. The audio data is also preprocessed in the same way.

[1518] 3. Terminal: Uses communication means to send preprocessed data to the server.

[1519] 4. Server: The server receives the data sent from the device and temporarily stores it in a database.

[1520] 5. Server: The received video data is input into an image recognition model (e.g., CNN) to analyze the pet's facial expressions and behaviors. Specifically, it extracts features such as the pet's ears, tail, and posture.

[1521] 6. Server: Input the audio data into a speech recognition model (e.g., RNN or LSTM) to analyze the tone and patterns of the pet's voice.

[1522] 7. Server: Integrates the results of image and voice recognition to determine the pet's intention. In this case, the intention is determined to be "I want to go to the toilet."

[1523] 8. Server: The message generation means generates a natural language message saying "I want to go to the toilet."

[1524] 9. Server: Sends the generated message to the terminal.

[1525] 10. Terminal: The terminal receives the message sent from the server and displays it on the user interface.

[1526] Example 2: When an owner says to their pet, "Let's go for a walk"

[1527] 1. User: The owner types "Let's go for a walk" in the chat on the LINE app, or gives the command "Let's go for a walk" by voice.

[1528] 2. Device: The device receives the text message and voice data and performs preprocessing. The voice data is converted to text.

[1529] 3. Terminal: Uses communication means to send the acquired data to the server.

[1530] 4. Server: The server receives the data sent from the device and begins analyzing it.

[1531] 5. Server: Analyzes the voice instruction data and understands the instruction "Let's go for a walk."

[1532] 6. Server: The speech synthesis means converts the text data into speech data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[1533] 7. Server: Sends the generated audio data to the device.

[1534] 8. Device: The device receives the voice data from the server and plays it through the speaker, instructing the pet to "go for a walk."

[1535] 9. Server: The emotion engine analyzes the user's voice and facial expressions to recognize their emotions. For example, it can distinguish between commands from a happy owner and commands from a tired owner.

[1536] 10. Server: Based on the results of the emotion engine, adjusts message generation and voice synthesis, and conveys instructions to the pet in an appropriate tone.

[1537] In this way, the system of the present invention enables smooth and intuitive communication between pets and their owners by linking the terminal and server, and also enables more effective interactions by taking the user's emotions into consideration.

[1538] The processing flow will be explained below.

[1539] Example 1: When your pet needs to go to the toilet

[1540] Step 1:

[1541] Terminal

[1542] The device's camera and microphone detect your pet's facial expressions and movements in real time, collecting video and audio data.

[1543] Step 2:

[1544] Terminal

[1545] The pre-processing means divides the collected video data into frames and removes noise, and also pre-processes the audio data in the same way.

[1546] Step 3:

[1547] Terminal

[1548] The communication means prepares the pre-processed data for transmission to the server, where it is standardized and converted into a format suitable for communication.

[1549] Step 4:

[1550] Terminal

[1551] A communication means is used to transmit the pre-processed data to a server.

[1552] Step 5:

[1553] server

[1554] The server receives the data sent from the terminal and temporarily stores it in a database.

[1555] Step 6:

[1556] server

[1557] The received video data is input into an image recognition model (e.g., CNN) to analyze the pet's facial expressions and behaviors. Specifically, features such as the pet's ears, tail, and posture are extracted.

[1558] Step 7:

[1559] server

[1560] The audio data is input into a speech recognition model (e.g., RNN or LSTM) to analyze the tone and patterns of the pet's voice.

[1561] Step 8:

[1562] server

[1563] The results of image and voice recognition are combined to comprehensively determine the pet's intention. In this case, the intention is determined to be "I want to go to the toilet."

[1564] Step 9:

[1565] server

[1566] The message generating means generates a natural language message saying "I want to go to the toilet."

[1567] Step 10:

[1568] server

[1569] Send the generated message to the terminal.

[1570] Step 11:

[1571] Terminal

[1572] The terminal receives the message sent from the server and displays it on the user interface.

[1573] Step 12:

[1574] User

[1575] The user (owner) checks the notification on the device and understands that the pet wants to go to the toilet.

[1576] Example 2: When an owner says to their pet, "Let's go for a walk"

[1577] Step 1:

[1578] User

[1579] The owner can either type "Let's go for a walk" in the chat on the LINE app or give the command "Let's go for a walk" by voice.

[1580] Step 2:

[1581] Terminal

[1582] The device receives text messages and voice data, performs preprocessing, and converts the voice data into text through speech recognition.

[1583] Step 3:

[1584] Terminal

[1585] The acquired text message and voice data are sent to a server using a communication means.

[1586] Step 4:

[1587] server

[1588] The server receives the data sent from the terminal and begins analyzing it.

[1589] Step 5:

[1590] server

[1591] It analyzes received text messages and voice data and understands the instructions such as "Let's go for a walk."

[1592] Step 6:

[1593] server

[1594] A voice synthesis means converts the text data into voice data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[1595] Step 7:

[1596] server

[1597] The generated voice data is transmitted to the terminal.

[1598] Step 8:

[1599] Terminal

[1600] The device receives voice data from the server and plays it through the speaker, instructing your pet to "go for a walk."

[1601] Step 9:

[1602] server

[1603] The emotion engine analyzes the user's voice and facial expressions to recognize their emotions. For example, it can distinguish between commands from a happy owner and commands from a tired owner.

[1604] Step 10:

[1605] server

[1606] Based on the results of the emotion engine, message generation and voice synthesis are adjusted to convey instructions to pets in an appropriate tone.

[1607] In this way, the system of the present invention enables smooth and intuitive communication between pets and their owners by linking the terminal and server, and also enables more effective interactions by taking the user's emotions into consideration.

[1608] Example 2

[1609] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1610] In today's communication between pets and their owners, it is difficult to accurately understand the pet's intentions and emotions, which often results in delayed appropriate responses. Furthermore, when an owner gives a command to a pet, the pet may not accurately understand the owner's intentions, making effective communication difficult. To solve these issues, a system is needed that can accurately detect and analyze a pet's condition and appropriately communicate the results to the owner.

[1611] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a sensor means for detecting and collecting the pet's status, a preprocessing means for dividing the acquired video data into frames and removing noise, a communication means for transmitting the preprocessed data to the server, a calculation means for analyzing the data and determining the pet's intention, a message generation means for generating a message in natural language based on the determined intention, a communication means for transmitting the generated message to the terminal, a user interface means for displaying the received message, an input means for accepting user input, a communication means for transmitting the user input to the server, a synthesis means for analyzing the user input received by the server and generating voice instructions, a communication means for transmitting the generated voice instructions to the terminal, an output means for playing back the received voice instructions, and an emotion recognition means for analyzing the user's voice and facial expressions to recognize emotions. This enables smooth and intuitive communication between the pet and its owner.

[1612] "Sensor means" refers to means that use devices such as cameras and microphones to detect and collect information about the condition of pets.

[1613] The "preprocessing means" is a means for processing the acquired data into a form that is easier to analyze. Specifically, the preprocessing means divides the video data into frames and performs processing to remove noise.

[1614] "Communication means" refers to the means for sending and receiving preprocessed data and generated messages between servers and terminals, and uses Wi-Fi or mobile communications.

[1615] The "calculation means" is a means for analyzing the received data and determining the pet's intentions, and uses an image recognition model or a voice recognition model.

[1616] The "message generating means" is a means for generating a natural language message that conveys the pet's intention based on the analysis results.

[1617] "User interface means" refers to means for displaying received messages to the owner, allowing the owner to check the status and needs of their pet.

[1618] The "input means" is a means for accepting instructions or input from the user, and accepts text or voice input.

[1619] A "synthesizer" is a means for analyzing a user's input and generating appropriate voice instructions.

[1620] The "output means" refers to a means for reproducing the generated voice instructions, and uses a device such as a speaker.

[1621] The "emotion recognition means" is a means for analyzing the user's voice and facial expression and recognizing the user's emotion.

[1622] MODE FOR CARRYING OUT THE INVENTION

[1623] The present invention is a system that analyzes a pet's facial expressions, voice, and behavior, and uses that information to enable effective communication between the owner and the pet. This system includes multiple technical elements that operate in conjunction with a terminal and a server. The system configuration and specific operation are described below.

[1624] System Configuration

[1625] Sensor means

[1626] The device is equipped with a camera and microphone, which detect and collect information about your pet's condition in real time. For example, the camera captures images of your pet's ears, tail, and posture, while the microphone records their cries and barks.

[1627] Pretreatment means

[1628] The pre-processing means within the device has the function of processing the collected data into a form that is easier to analyze. Specifically, it divides the video data into frames and removes noise. It also performs noise removal processing on audio data. For example, the device divides the video into frames every few seconds and automatically adjusts the brightness and color tone from each frame.

[1629] communication means

[1630] The device has a communication means to send the preprocessed data to the server. This communication means can be Wi-Fi or mobile communication. For example, the device checks for a Wi-Fi connection and uploads the data to the server via a stable network.

[1631] Server Means

[1632] The server is a computer system with advanced computing power that inputs the received data into an image recognition model (e.g., CNN) or a voice recognition model (e.g., RNN or LSTM) to analyze the pet's facial expressions and behaviors. Specifically, the server uses a deep learning model to detect the position of the ears and tail from the video and stress or happiness from the voice.

[1633] Message Generation Method

[1634] The message generation means in the server generates and transmits a natural language message that conveys the pet's intentions based on the analysis results, such as "I want to go to the toilet" or "I want to play."

[1635] User Interface Means

[1636] The device displays messages sent from the server to the owner through a user interface, allowing the owner to easily check the pet's status and needs. For example, messages can be displayed as a pop-up notification on a smartphone or tablet.

[1637] Voice synthesis means

[1638] The server's voice synthesis means converts user input into voice data, allowing users to give voice commands to their pets. For example, if an owner sends a message to their pet via the LINE app saying "Let's go for a walk," the server converts the message into natural-sounding voice.

[1639] Audio output means

[1640] The device's built-in voice output means plays back the received voice data to communicate instructions to your pet. For example, a natural voice will be played from the device's speaker saying, "Let's go for a walk."

[1641] emotion recognition means

[1642] The emotion recognition means includes technology that analyzes the user's voice and facial expressions and recognizes their emotions. This allows the system to understand the user's emotional state and reflect it in message generation and voice instructions. For example, the system can distinguish between instructions from an owner who is happy and instructions from an owner who is tired and communicate them to the pet in an appropriate tone.

[1643] Specific examples

[1644] Example 1: When your pet needs to go to the toilet

[1645] The device's camera and microphone detect your pet's facial expressions and gestures, collecting video and audio data.

[1646] The pre-processing means divides the collected video data into frames and removes noise, and also pre-processes the audio data in the same way.

[1647] The preprocessed data is sent to the server using a communication means.

[1648] The data received by the server is input into image and voice recognition models for analysis, and the intent is determined to be "I want to go to the toilet."

[1649] The message generating means generates a natural language message saying "I want to go to the toilet" and transmits it to the terminal.

[1650] The terminal receives the message sent from the server and displays it on the user interface.

[1651] Example 2: When an owner says to their pet, "Let's go for a walk"

[1652] The owner can either type "Let's go for a walk" in the chat on the LINE app or give the command "Let's go for a walk" by voice.

[1653] The device receives text messages and voice data, performs preprocessing, and converts the voice data into text.

[1654] The acquired data is sent to the server using a communication means.

[1655] The server analyzes the received data and understands the instruction "Let's go for a walk."

[1656] A voice synthesis means converts the text data into voice data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[1657] The generated voice data is transmitted to the terminal.

[1658] The device receives voice data from the server and plays it through the speaker, instructing your pet to "go for a walk."

[1659] Examples of prompt statements

[1660] Example inputs to a generative AI model:

[1661] "Please explain how to analyze a pet's gestures and voice to communicate its intentions to its owner."

[1662] "Please explain how the system works to notify pet owners appropriately when their pets behave in a certain way."

[1663] As described above, the system of the present invention enables smooth and effective communication between pets and their owners by linking the terminal and server, and also takes into account the user's emotions, enabling more intuitive and friendly communication.

[1664] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1665] Step 1: Collect pet status

[1666] Subject: Device

[1667] Description: The device uses a camera and microphone to detect your pet's condition (facial expressions, voice, and behavior) in real time and collects this data.

[1668] Input: Your pet's real-life facial expressions, sounds, and gestures

[1669] Output: Collected video and audio data

[1670] Specific operations: The device's camera captures your pet's ears, tail, posture, etc., and the microphone records any meows or barks.

[1671] Step 2: Preprocessing the data

[1672] Subject: Device

[1673] Description: The preprocessing means divides the collected video data into frames and removes noise. The audio data is also subjected to noise removal.

[1674] Input: Collected video and audio data

[1675] Output: Preprocessed video frames and audio data

[1676] What it does: The device splits the video into frames every few seconds, then automatically adjusts the brightness and color of each frame to remove noise.

[1677] Step 3: Sending data

[1678] Subject: Device

[1679] Description: The pre-processed data is sent to the server via a communication method, in this case Wi-Fi or mobile communication.

[1680] Input: Preprocessed video frames and audio data

[1681] Output: Data sent to the server

[1682] What happens: The device checks for a Wi-Fi connection and uploads data to the server over a stable network.

[1683] Step 4: Analyze the data

[1684] Subject: Server

[1685] Description: The server inputs the received data into an image recognition model (e.g., CNN) or a speech recognition model (e.g., RNN or LSTM) for analysis.

[1686] Input: Preprocessed data sent to the server

[1687] Output: Parsed pet's intention and state information

[1688] Specific operation: The server uses a deep learning model to detect the position of the ears and tail from video footage, and emotions and intentions such as stress or joy from audio.

[1689] Step 5: Generate a message

[1690] Subject: Server

[1691] Description: Based on the analysis results, the message generator generates a natural language message that conveys the pet's intentions.

[1692] Input: Parsed pet intent and state information

[1693] Output: Natural language message

[1694] Concrete behavior: The process of generating simple messages such as "I want to go to the toilet" or "I want to play."

[1695] Step 6: Sending a message

[1696] Subject: Server (terminal)

[1697] Description: The server generates a message and sends it to the terminal, which receives it.

[1698] Input: A natural language message

[1699] Output: Message sent to the terminal

[1700] Specific operation: The server sends a message to the terminal, and the terminal receives the message.

[1701] Step 7: Displaying messages

[1702] Subject: Device

[1703] Description: Displays received messages to the owner through user interface means.

[1704] Input: Message sent to terminal

[1705] Output: The message displayed in the user interface

[1706] What happens: A message will pop up through your smartphone or tablet's notification feature.

[1707] Step 8: Accepting User Input

[1708] Subject: User

[1709] Description: The user enters instructions in the form of speech or text.

[1710] Input: User instructions (voice or text message)

[1711] Output: User's instruction data

[1712] Specific actions: The owner types "Let's go for a walk" in the chat on the LINE app, or gives voice instructions.

[1713] Step 9: Sending User Input

[1714] Subject: Device

[1715] Description: Preprocesses input from the user and sends it to the server via a communication medium.

[1716] Input: User-specified data

[1717] Output: User instruction data sent to the server

[1718] Specific operation: The process of converting input voice data into text and sending it to the server.

[1719] Step 10: Generate voice instructions

[1720] Subject: Server

[1721] Description: The server analyzes the user's input and generates appropriate voice instructions using a speech synthesis method.

[1722] Input: User instruction data sent to the server

[1723] Output: Generated voice instruction data

[1724] Specific operation: Convert the text message "Let's go for a walk" into voice data with natural pronunciation.

[1725] Step 11: Sending voice instructions

[1726] Subject: Server (terminal)

[1727] Description: The server generates and sends voice instructions to the device, which receives them.

[1728] Input: Generated voice command data

[1729] Output: Audio instructions sent to the device

[1730] Specific operation: The server sends voice data to the terminal, and the terminal receives the data.

[1731] Step 12: Play audio instructions

[1732] Subject: Device

[1733] Description: Plays back the received audio instructions through the audio output means.

[1734] Input: Voice commands sent to the device

[1735] Output: The audio instructions played

[1736] Specific operation: "Let's go for a walk" is played in a natural voice from the device's speaker.

[1737] (Application example 2)

[1738] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1739] Currently, stores that cater to customers with pets need to quickly and accurately understand the pet's condition and respond appropriately. However, it is difficult to understand the pet's intentions in real time, which can lead to delayed responses, especially in emergencies. In addition, smooth communication between pets, owners, and store staff is required, but an efficient system to achieve this does not exist.

[1740] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for determining the pet's intention, a notification means for sending a notification to the user and staff, and a means for analyzing the pet's condition using an image recognition model and a voice recognition model. This makes it possible to monitor the pet's condition in real time and take appropriate measures, including emergency notifications. It also enables the user and staff to quickly understand the pet's condition and take appropriate measures.

[1741] "Pets" are animals kept by humans, such as dogs and cats.

[1742] "Audio" refers to the voices and sounds made by pets.

[1743] "Expression" refers to the facial expressions and changes in your pet's face.

[1744] "Gestures" refer to the body movements and actions of pets.

[1745] "Sensor means" refers to a device such as a camera or microphone for detecting the voice, facial expressions, and behavior of a pet.

[1746] "Preprocessing means" refers to means for converting collected data into a form that is easy to analyze.

[1747] "Communication means" refers to a means for transmitting pre-processed data to a server.

[1748] The "server means" refers to a means for analyzing the received data and determining the intention of the pet.

[1749] The "message generating means" refers to a means for generating a message in natural language based on the determined intention.

[1750] "User interface means" refers to means for displaying generated messages to a user.

[1751] A "voice synthesis means" is a means for analyzing input from a user and generating voice instructions.

[1752] "Audio output means" refers to means for reproducing the generated audio instructions.

[1753] "Notification means" refers to a means for determining the pet's condition and sending notifications to the user and staff.

[1754] An "image recognition model" is a machine learning model for analyzing pets' facial expressions and behaviors.

[1755] A "voice recognition model" is a machine learning model for analyzing pet voices.

[1756] This invention is a support system for customers who bring pets to physical stores, and as a specific example, we will explain a "pet support app." This application has the function of monitoring the pet's condition in real time and sending appropriate notifications to users and staff.

[1757] Hardware and software used

[1758] Hardware

[1759] Camera: Used to capture your pet's expressions and behavior in real time.

[1760] Microphone: Used to record your pet's voice.

[1761] Smartphone: The above sensors (camera and microphone) are built in.

[1762] Server: A high-performance computer system that analyzes data and stores results.

[1763] software

[1764] OpenCV: A library for acquiring and preprocessing video data from a camera.

[1765] TensorFlow / Keras: Machine learning libraries for implementing image recognition models to analyze pet facial expressions and behaviors.

[1766] SpeechRecognition: A library for converting your pet's spoken commands into text.

[1767] Requests: A library for sending preprocessed data to a server.

[1768] Data processing and calculation

[1769] Terminal handling

[1770] The device activates the camera and microphone to capture the pet's facial expressions, gestures, and voice in real time. Then, it splits the video data into frames and performs preprocessing using OpenCV. It also converts the voice data into text using SpeechRecognition. This preprocessed data is sent to the server using Requests.

[1771] Server Processing

[1772] The server analyzes the received data in real time. It uses a TensorFlow / Keras-based image recognition model to extract features from the pet's facial expressions and gestures and determine the pet's intentions. It also uses a voice recognition model to analyze the pet's voice patterns. These analysis results are combined to determine the pet's condition. The determination result is generated as a natural language message by a message generation means and sent to the terminal.

[1773] Specific examples

[1774] Example 1: When your pet needs to go to the toilet

[1775] The device's camera and microphone capture the pet's state and perform preprocessing.

[1776] The server analyzes the data and determines the intention to "go to the toilet."

[1777] The result of the judgment is notified to the user and staff, and appropriate action is taken.

[1778] Example 2: When the owner says, "Let's go for a walk"

[1779] The user types in the text "Let's go for a walk" or gives the voice command.

[1780] The device preprocesses the data and sends it to the server.

[1781] The server analyzes the instructions, generates appropriate voice instructions, and sends them to the terminal.

[1782] The device will play audio instructions and communicate them to your pet.

[1783] Prompt Sentence Examples

[1784] Message prompt to display in the user interface:

[1785] When a pet's condition is emergency, generate the following message: 'Your pet is in [condition]. Please take appropriate action.'

[1786] Staff notification message prompt:

[1787] Generate an emergency notification message to send to staff: 'Pet is in [condition]. Please respond immediately.'

[1788] In this way, the system of the present invention enables support for customers with pets by linking the server and terminals to quickly and accurately grasp the condition of pets and provide appropriate notifications to users and staff.

[1789] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1790] Step 1:

[1791] The device activates the camera and microphone to capture the pet's facial expressions, gestures, and voice in real time. The input is the camera image and the microphone audio, and the output is image data and voice data for each frame.

[1792] Step 2:

[1793] The device uses OpenCV to preprocess the video data. Specifically, it splits it into frames and performs noise reduction and resizing as needed. The input is the captured image data for each frame, and the output is the preprocessed image data.

[1794] Step 3:

[1795] The device uses SpeechRecognition to preprocess the audio data, specifically converting it to text. The input is the captured audio data, and the output is text data.

[1796] Step 4:

[1797] The terminal uses Requests to send preprocessed image data and text data to the server. The input is the preprocessed image data and text data, and the output is the data sent to the server.

[1798] Step 5:

[1799] The server analyzes the received image data using a TensorFlow / Keras-based image recognition model. Specifically, it extracts features from the pet's facial expressions and behavior. The input is the preprocessed image data, and the output is the analysis results.

[1800] Step 6:

[1801] The server uses a speech recognition model to analyze the received text data. Specifically, it analyzes the pet's voice patterns and intentions from the text data. The input is the preprocessed text data, and the output is the analysis results.

[1802] Step 7:

[1803] The server integrates the results of image and voice recognition analysis to comprehensively judge the pet's condition. The input is the results of image and voice analysis, and the output is a judgment result indicating the pet's condition.

[1804] Step 8:

[1805] The server generates a natural language message based on the judgment result. Using the message generation means, the judgment result is converted into a text message that is easy for users and staff to understand. The input is the judgment result, and the output is a natural language message.

[1806] Step 9:

[1807] The server sends the generated message to the terminal. The input is the generated natural language message, and the output is the message sent to the terminal.

[1808] Step 10:

[1809] The terminal displays the received messages to the user through a user interface means, where the input is the message sent from the server and the output is the message displayed to the user.

[1810] Step 11:

[1811] The user inputs commands to the pet through the terminal. For example, "Let's go for a walk" is given by voice or text. The input is the user's command, and the output is text or voice data.

[1812] Step 12:

[1813] The terminal sends text data and voice data to the server. The input is the text or voice data containing the user's instructions, and the output is the data sent to the server.

[1814] Step 13:

[1815] The server analyzes the received instructions and generates voice instructions for the pet using a voice synthesis means. The input is the user's instruction data, and the output is the synthesized voice instruction.

[1816] Step 14:

[1817] The server sends the synthesized voice instructions to the terminal, where the input is the generated voice instructions and the output is the voice data sent to the terminal.

[1818] Step 15:

[1819] The terminal uses the audio output means to play audio instructions to the pet, with the input being audio data sent from the server and the output being the played audio instructions.

[1820] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1821] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1822] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1823] [Fourth embodiment]

[1824] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1825] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1826] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1827] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1828] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1829] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1830] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1831] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1832] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1833] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1834] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1835] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1836] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1837] The present invention is a system that analyzes the facial expressions, sounds, and behaviors of pets and enables communication between owners and pets based on this information. The configuration and specific operation of the system are described below.

[1838] 1. System Configuration

[1839] 1.1. Sensor means

[1840] The sensor means installed in the terminal includes a camera and microphone for detecting and collecting the voice, facial expressions, and behavior of the pet, thereby enabling the status of the pet to be acquired in real time.

[1841] 1.2. Pre-processing methods

[1842] The pre-processing means in the terminal pre-processes the acquired data in order to properly analyze it, for example, by dividing the video data into frames and removing noise.

[1843] 1.3. Means of communication

[1844] The terminal transmits the preprocessed data to the server using a communication means, which can be Wi-Fi or mobile communication.

[1845] 1.4. Server Means

[1846] The server uses a computer system with advanced computing power to analyze the received data, which is then input into an image recognition model or a voice recognition model for analysis.

[1847] 1.5. Message Creation Methods

[1848] The message generation means in the server generates a message in natural language based on the analysis results. This message is intended to convey the pet's intentions to the owner in an easy-to-understand manner.

[1849] 1.6. User Interface Methods

[1850] The terminal displays the messages sent from the server to the owner via the user interface means, allowing the owner to easily check the status and needs of their pet.

[1851] 1.7. Speech synthesis methods

[1852] The voice synthesis means in the server converts the owner's input into voice data, allowing the owner to give voice instructions to their pet.

[1853] 1.8. Audio Output Methods

[1854] The audio output means installed in the terminal reproduces the received audio data and conveys instructions to the pet.

[1855] 2. System Operation

[1856] Example 1: When your pet needs to go to the toilet

[1857] 1. Device: The device's camera and microphone detect your pet's facial expressions and behaviors and collect data.

[1858] 2. Terminal: The preprocessing means preprocesses the data and prepares it for transmission to the server.

[1859] 3. Terminal: Uses communication means to send the preprocessed data to the server.

[1860] 4. Server: The server inputs the received data into image and voice recognition models to analyze the pet's intentions.

[1861] 5. Server: Based on the analysis results, generate a message saying "I want to go to the toilet."

[1862] 6. Server: Sends the generated message to the terminal.

[1863] 7. Terminal: Displays the received message to the owner using a user interface.

[1864] Example 2: When an owner says to their pet, "Let's go for a walk"

[1865] 1. User: The owner types "Let's go for a walk" in the chat on the LINE app or gives voice instructions.

[1866] 2. Terminal: Sends the owner's input text and voice data to the server.

[1867] 3. Server: Analyzes the received data and generates audio data as needed.

[1868] 4. Server: Uses a speech synthesis tool to generate the speech "Let's go for a walk."

[1869] 5. Server: Sends the generated voice data to the device.

[1870] 6. Terminal: The received voice data is played back using the voice output means, and instructions are conveyed to the pet.

[1871] As described above, the system of the present invention is designed to realize smooth and intuitive communication between pets and their owners, allowing them to quickly understand the pet's condition and needs and take the necessary measures.

[1872] The processing flow will be explained below.

[1873] Processing flow

[1874] Example 1: When your pet needs to go to the toilet

[1875] Step 1:

[1876] Terminal

[1877] The device's camera and microphone detect your pet's facial expressions and movements in real time, collecting video and audio data.

[1878] Step 2:

[1879] Terminal

[1880] The pre-processing means divides the collected video data into frames and removes noise, and also pre-processes the audio data in the same way.

[1881] Step 3:

[1882] Terminal

[1883] The communication means prepares the pre-processed data for transmission to the server, where it is standardized and converted into a format suitable for communication.

[1884] Step 4:

[1885] server

[1886] The server receives the data sent from the terminal and temporarily stores it in a database.

[1887] Step 5:

[1888] server

[1889] The received video data is input into an image recognition model (e.g., CNN) to analyze the pet's facial expressions and behaviors. Specifically, features such as the pet's ears, tail, and posture are extracted.

[1890] Step 6:

[1891] server

[1892] The audio data is input into a speech recognition model (e.g., RNN or LSTM) to analyze the tone and patterns of the pet's voice.

[1893] Step 7:

[1894] server

[1895] The results of image and voice recognition are combined to comprehensively determine the pet's intention. In this case, the intention is determined to be "I want to go to the toilet."

[1896] Step 8:

[1897] server

[1898] The message generating means generates a natural language message saying "I want to go to the toilet."

[1899] Step 9:

[1900] server

[1901] Send the generated message to the terminal.

[1902] Step 10:

[1903] Terminal

[1904] The terminal receives the message sent from the server and displays it on the user interface.

[1905] Example 2: When an owner says to their pet, "Let's go for a walk"

[1906] Step 1:

[1907] User

[1908] The owner can either type "Let's go for a walk" in the chat on the LINE app or give the command "Let's go for a walk" by voice.

[1909] Step 2:

[1910] Terminal

[1911] The device receives text messages and voice data, performs preprocessing, and converts the voice data into text.

[1912] Step 3:

[1913] Terminal

[1914] The acquired data is sent to the server using a communication means.

[1915] Step 4:

[1916] server

[1917] The server receives the data sent from the terminal and begins analyzing it.

[1918] Step 5:

[1919] server

[1920] It analyzes voice command data and understands the command "Let's go for a walk."

[1921] Step 6:

[1922] server

[1923] A voice synthesis means converts the text data into voice data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[1924] Step 7:

[1925] server

[1926] The generated voice data is transmitted to the terminal.

[1927] Step 8:

[1928] Terminal

[1929] The device receives voice data from the server and plays it through the speaker, instructing your pet to "go for a walk."

[1930] In this way, the system of the present invention allows the terminal and server to work together to realize smooth communication between pets and their owners.

[1931] Example 1

[1932] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1933] In general, communication between pets and owners is difficult to achieve quickly and accurately because it is difficult to understand the intentions and state of the pet. In particular, owners often cannot adequately meet their pets' needs because they do not have a way to accurately grasp what their pets want and communicate it in human language.

[1934] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1935] In this invention, the server includes sensor means for acquiring the voice, facial expressions, and behaviors of the pet, data processing means for preprocessing the acquired data, communication means for transmitting the preprocessed data to the server, server means for analyzing the data and determining the intention of the pet, message generation means for generating a message in natural language based on the determined intention, data transmission means for transmitting the generated message to the terminal, user interface means for displaying the received message, input means for accepting user input, additional communication means for transmitting the user input to the server, voice synthesis means for analyzing the user input received by the server and generating voice instructions, additional data transmission means for transmitting the generated voice instructions to the terminal, and voice output means for playing back the received voice instructions. This allows the owner to quickly and accurately grasp the condition and needs of the pet and take appropriate measures.

[1936] The "sensor means" is a device for acquiring the voice, facial expressions, and behavior of a pet.

[1937] "Data processing means" refers to a device or software for preprocessing acquired data.

[1938] "Communication means" is a device or software for transmitting pre-processed data to a server.

[1939] The "server means" is a system for analyzing data on a server and determining the intentions of the pet.

[1940] A "message generator" is a device or software for generating a natural language message based on the determined intent.

[1941] The "data transmission means" is a function for transmitting the generated message to the terminal.

[1942] "User interface means" is a device or software for displaying received messages.

[1943] "Input means" refers to a device or software for accepting user input.

[1944] "Additional communication means" is a function for sending input from the user to the server.

[1945] A "voice synthesis means" is a device or software for analyzing user input received at the server and generating voice instructions.

[1946] The "additional data transmission means" is a function for transmitting the generated voice instructions to the terminal.

[1947] "Audio output means" refers to a device or software for reproducing received audio instructions.

[1948] The present invention is a system for analyzing a pet's facial expressions, voice, and behavior, and facilitating smooth communication between the owner and the pet based on the results. This system is realized through the interaction of a terminal, a server, and a user.

[1949] System Configuration

[1950] Sensor means

[1951] The device is equipped with a camera and microphone to capture your pet's facial expressions, voice, and behavior in real time. The camera captures your pet's facial expressions and movements, while the microphone collects barks and other sounds. These sensory tools are important for accurately understanding your pet's condition and behavior.

[1952] Data Processing Means

[1953] The device has a data processing means that preprocesses the acquired data. Specifically, image processing libraries such as OpenCV are used to divide the video data into frames and remove noise. Additionally, collected audio data is filtered to remove noise. This preprocessing converts the data into a format that is easier to analyze.

[1954] communication means

[1955] The device then sends the pre-processed data to the server over Wi-Fi or mobile data, typically using HTTP or WebSocket protocols, which is important to ensure fast and secure data transfer.

[1956] Server Means

[1957] The server has advanced computing power to analyze the received data. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to input the data into image recognition and voice recognition models for analysis. For example, the image recognition model analyzes the pet's facial expressions and gestures, and the voice recognition model analyzes the meaning of its cries.

[1958] Message Generation Method

[1959] The server's message generation means generates natural language messages based on the analysis results. This process uses generative AI models such as GPT-3. For example, if a pet's behavior is interpreted as "I want to go to the toilet," a message to that effect is generated for the owner.

[1960] Data transmission method

[1961] The message generated by the server is sent to the terminal using the data transmission means, which uses the HTTP protocol.

[1962] User Interface Means

[1963] The device displays the received message to the user via a user interface means, for example, as a notification on a smartphone app, allowing the owner to understand the status and needs of their pet.

[1964] Input Method

[1965] The user can give instructions to their pet by voice or text using the smartphone's microphone or keyboard.

[1966] Additional Communication Methods

[1967] The device sends user input to the server using an additional communication method. For example, if an owner types "Let's go for a walk" into the LINE app, the message is sent to the server.

[1968] Voice synthesis means

[1969] The server converts the received instructions from the owner into voice data using a voice synthesis tool. Specifically, it uses a service such as Google Cloud Text-to-Speech to generate voice data such as "Let's go for a walk."

[1970] Additional Data Transmission Methods

[1971] The generated voice data is transmitted to the terminal using additional data transmission means.

[1972] Audio output means

[1973] The device then plays the received voice data via the voice output means to communicate instructions to the pet. For example, a smart speaker might play a voice saying, "Let's go for a walk."

[1974] Specific examples

[1975] Example 1: When your pet needs to go to the toilet

[1976] 1. The device's camera and microphone detect your pet's facial expressions and behaviors and collect data.

[1977] 2. The data processing means of the terminal divides the video data into frames and removes noise.

[1978] 3. The device sends the preprocessed data to the server.

[1979] 4. The server analyzes the received data and generates a message saying "I want to go to the toilet."

[1980] 5. The server sends the generated message to the terminal.

[1981] 6. The device displays the received message to the owner using the user interface.

[1982] Example 2: When an owner says to their pet, "Let's go for a walk"

[1983] 1. The user types "Let's go for a walk" in the chat on the LINE app.

[1984] 2. The device sends the owner's input text to the server.

[1985] 3. The server analyzes the received data and generates audio data.

[1986] 4. The server uses a speech synthesis means to generate the speech "Let's go for a walk."

[1987] 5. The server sends the generated voice data to the terminal.

[1988] 6. The terminal uses the audio output means to play back the received audio data and convey instructions to the pet.

[1989] Prompt Sentence Examples

[1990] Prompt: "Generate a prompt to ask a pet owner if they want to go for a walk."

[1991] Output: "Shall we go for a walk?"

[1992] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1993] Step 1:

[1994] The device uses sensors to detect your pet's facial expressions, voice, and gestures. The input is your pet's real-time movements and voice, which are captured using a camera and microphone. Specifically, the camera captures your pet's facial expressions and movements, and the microphone collects its barks and other sounds. The output is the captured video and audio data.

[1995] Step 2:

[1996] The device preprocesses the acquired data. The preprocessing means divides the video data into frames and removes noise. Similarly, audio data is filtered to remove noise. The processing involves preprocessing the video data using an image processing library such as OpenCV, and preprocessing the audio data using audio filtering techniques. The output is video and audio data in a clean, easy-to-analyze format.

[1997] Step 3:

[1998] The device sends the preprocessed data to the server via a communication means. The input is the preprocessed video and audio data, which is sent via Wi-Fi or mobile communication. The HTTP protocol or WebSocket protocol is used for data transmission. The output is the completion of data transfer to the server.

[1999] Step 4:

[2000] The server analyzes the received data. The input is pre-processed video and audio data. For the analysis, deep learning frameworks such as TensorFlow and PyTorch are used to apply image recognition and voice recognition models. Specifically, the image recognition model analyzes the pet's facial expressions and gestures, while the voice recognition model analyzes the meaning of the pet's cries. The output is an analysis result that indicates the pet's intentions.

[2001] Step 5:

[2002] The server generates a natural language message based on the analysis results. The input is the analysis result indicating the pet's intention. A generative AI model such as GPT-3 is used for this process. Specifically, for example, if the pet's gesture is interpreted as "I want to go to the toilet," a message to that effect is generated for the owner. The output is a message written in natural language.

[2003] Step 6:

[2004] The server sends the generated message to the terminal using a data transmission means. The input is the generated natural language message. The HTTP protocol is used for transmission. The output is the completion of message transfer to the terminal.

[2005] Step 7:

[2006] The terminal displays the received message to the user via the user interface means. The input is a natural language message received from the server. The message is displayed as a notification on the smartphone app. The output is a message displayed to the owner.

[2007] Step 8:

[2008] The user inputs instructions for the pet by voice or text. The smartphone's microphone or keyboard is used as the input means. The input data is text or voice instructions given by the user, such as "Let's take your pet for a walk." The output is instruction data sent from the user to the device.

[2009] Step 9:

[2010] The device sends user input to the server using additional communication means. The input is the user's instruction data (text or voice). This is sent using Wi-Fi or mobile communication. The output is the transfer of instruction data to the server.

[2011] Step 10:

[2012] The server analyzes the received user instructions and converts them into voice data using a voice synthesis means. The input is the user's instruction data. This process uses services such as Google Cloud Text-to-Speech. Specifically, the text instruction "Let's go for a walk" is converted into voice data. The output is voice data.

[2013] Step 11:

[2014] The generated voice data is transmitted to the terminal using an additional data transmission means of the server. The input is the generated voice data, and the HTTP protocol is used for transmission. The output is the transfer of the voice data to the terminal.

[2015] Step 12:

[2016] The terminal plays the received voice data via the voice output means and transmits instructions to the pet. The input is the voice data received from the server. For example, a smart speaker plays the voice "Let's go for a walk." The output is the transmission of instructions to the pet.

[2017] (Application example 1)

[2018] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2019] In today's world, pets are important members of the family, and there is a need to monitor their condition and safety while their owners are away. However, there is no effective system that analyzes a pet's facial expressions, gestures, and voice, detects their intentions and abnormal behavior in real time, and notifies the owner. As a result, if a pet is in trouble while the owner is away, it is difficult to detect the problem early and take appropriate action. There is also a lack of systems that can accurately analyze a pet's condition and issue appropriate instructions. There is a need to solve these issues.

[2020] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2021] In this invention, the server includes an abnormal behavior analysis means for analyzing abnormal behavior of a pet and generating a warning message based on the analysis, a means for sending a push notification of the generated warning message to the owner's device, and a server means for using an image recognition model and a voice recognition model to determine the pet's intentions and abnormal behavior. This allows the owner to monitor the pet's condition in real time even when the owner is away, and to be notified promptly when an abnormality is detected. Furthermore, an effective system for maintaining the safety and health of pets can be provided, providing appropriate countermeasures.

[2022] The "sensor means" is a device including a camera and a microphone for capturing the sounds, facial expressions, and behaviors of a pet.

[2023] The "pre-processing means" is a device that divides the acquired data into frames and performs pre-processing such as removing noise in order to properly analyze the data.

[2024] A "communication means" is a device that uses Wi-Fi or mobile communication to transmit preprocessed data to a server.

[2025] The "server means" is a computer system with advanced computing power for analyzing data and determining the intentions of the pet.

[2026] The "message generating means" is a device that generates a message in natural language based on the determined intention.

[2027] "User interface means" is a device for displaying the generated message on a terminal.

[2028] The "means for accepting user input" is an interface device for accepting instructions and inputs from the owner.

[2029] A "voice synthesis means" is a device for analyzing user input received by the server and generating voice instructions.

[2030] "Audio output means" refers to a device for reproducing the generated audio instructions.

[2031] The "abnormal behavior analysis means" is a device that analyzes abnormal behavior of a pet and generates a warning message based on the analysis.

[2032] The "push notification means" is a device that transmits the generated warning message to the owner's terminal in real time.

[2033] The "image recognition model" is an AI model that analyzes acquired video data and determines a pet's facial expressions and behavior.

[2034] A "voice recognition model" is an AI model that analyzes acquired voice data and identifies the voice of a pet.

[2035] This system facilitates smooth communication between pets and their owners, and also has a security function that sends instant notifications if something unusual happens to the pet. It mainly consists of the following elements:

[2036] 1. System Overview

[2037] 1.1. Sensor means

[2038] The device is equipped with a camera and microphone that captures the pet's facial expressions, movements, and voice in real time. The camera and microphone are controlled using the OpenCV library.

[2039] 1.2. Pre-processing methods

[2040] The preprocessing means in the terminal divides the acquired video data into frames and removes noise. The preprocessed data is sent to the server using the communication means described below.

[2041] 1.3. Means of communication

[2042] The preprocessed data is sent to the server via Wi-Fi or mobile communication, using the Requests library.

[2043] 1.4. Server Means

[2044] The server is a computer system for advanced analysis, analyzing data using image and voice recognition models to determine the pet's intentions and abnormal behavior. The server runs the AI ​​models using the Keras / TensorFlow library.

[2045] 1.5. Message Generation and User Interface Methods

[2046] The server generates a natural language message based on the intention or abnormal behavior determined by the server and sends it to the device, which displays the message to the owner through a user interface and prompts them to take the necessary action.

[2047] 1.6. Abnormal behavior analysis means

[2048] The system analyzes pet behavior data and generates a warning message if it detects abnormal behavior. This message is then pushed to the owner's device in real time.

[2049] 2. Processing Flow

[2050] The server performs analysis using image recognition models and voice recognition models. Based on the analysis results, it generates natural language messages and warning messages, and generates voice instructions using a voice synthesis means as needed. The generated voice instructions are played back by the voice output means of the terminal.

[2051] 3. Specific Examples

[2052] Example 1: When your pet exhibits abnormal behavior

[2053] 1. The device's camera and microphone record your pet's behavior.

[2054] 2. The preprocessing means preprocesses the data and sends it to the server.

[2055] 3. The server analyzes the data and detects any abnormal behavior.

[2056] 4. A warning message is generated and pushed to the owner's device.

[2057] Prompt Sentence Examples

[2058] "Your pet is barking anxiously. Abnormal behavior may have been detected. Owner should check immediately."

[2059] Example 2: When an owner says to their pet, "Let's go for a walk"

[2060] 1. The owner inputs the information using voice or text.

[2061] 2. The user interface means sends the input to the server.

[2062] 3. The server analyzes the data and generates voice instructions using a voice synthesis means.

[2063] 4. The generated voice instructions are played on the device.

[2064] In this way, the system of the present invention not only monitors the condition of pets in real time even when the owner is away, but also sends prompt notifications when an abnormality is detected, providing an effective way to maintain the safety and health of pets.

[2065] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2066] Step 1:

[2067] The device uses a camera and microphone to capture the pet's facial expressions, gestures, and voice in real time. The input is video frames and audio data. Specifically, the device uses the OpenCV library to control the camera and capture video. At the same time, it uses the microphone to capture audio data.

[2068] Step 2:

[2069] The preprocessing means in the terminal divides the acquired video data into frames and removes noise. The input is the video frames and audio data acquired in step 1. Specifically, the video data is resized to the required resolution and divided into frames. Noise is also removed from the audio data. The output is the preprocessed video frames and audio data.

[2070] Step 3:

[2071] The communication means sends the preprocessed data to the server. The input is the video frame and audio data preprocessed in step 2. Specifically, the device uses the Requests library to send the data to the server using an HTTP POST request. The output is the data sent to the server.

[2072] Step 4:

[2073] The server analyzes the received data and determines the pet's intention. The input is the preprocessed data sent in step 3. The server uses the Keras / TensorFlow library to execute image recognition and speech recognition models to determine the pet's intention from its facial expressions, gestures, and voice. The output is the intention data as the analysis result.

[2074] Step 5:

[2075] The server generates a natural language message based on the analysis results. The input is the intent data obtained in step 4. The server uses a generative AI model to generate a natural language message based on the intent data. The output is the generated natural language message.

[2076] Step 6:

[2077] The message generation means transmits the generated message to the terminal. The input is the natural language message generated in step 5. The message is transmitted to the owner's terminal via the communication means. The output is the message displayed on the owner's terminal.

[2078] Step 7:

[2079] The terminal displays the received message to the owner using a user interface. The input is the message received in step 6. As a specific operation, the terminal displays the message on a GUI (Graphical User Interface). The output is the message displayed to the owner.

[2080] Step 8:

[2081] The user provides input through the user interface. The input can be text or voice, such as instructions or questions. The user interface means transmits this input to the server. The output is the user input transmitted to the server.

[2082] Step 9:

[2083] The server analyzes input from the user and generates appropriate voice instructions. The input is the user input, such as the instruction or question submitted in step 8. The server analyzes the text using a natural language processing model and generates the voice instructions using a speech synthesis means. The output is the generated voice instructions.

[2084] Step 10:

[2085] The generated voice instruction is transmitted to the terminal. The input is the voice instruction generated in step 9. The voice instruction is transmitted to the terminal through the communication means. The output is the voice instruction transmitted to the terminal.

[2086] Step 11:

[2087] The terminal plays the received voice instruction using the voice output means. The input is the voice instruction received in step 10. In particular, the terminal plays the voice instruction through a speaker. The output is the played voice instruction.

[2088] Step 12:

[2089] The server periodically analyzes pet behavior data and generates a warning message if it detects abnormal behavior. The input is preprocessed video and audio data. If abnormal behavior is detected, the server immediately generates a warning message and describes it in natural language using a generative AI model. The output is a warning message based on the abnormal behavior.

[2090] Step 13:

[2091] The warning message generated by the server is sent to the owner's device using a push notification means. The input is the warning message generated in step 12. The push notification server sends it to the owner's smartphone or other device. The output is the warning message received by the owner's device.

[2092] Step 14:

[2093] The owner's device displays the received warning message on the user interface and immediately notifies the owner of the abnormal behavior. The input is the warning message received in step 13. As a specific operation, the device immediately warns the owner by using an alert, notification sound, banner display, etc. The output is the warning message displayed to the owner.

[2094] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2095] This invention is a system that analyzes a pet's facial expressions, voice, and behavior, and uses that information to enable effective communication between the owner and the pet. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, richer communication is realized. The system configuration and specific operation are explained below.

[2096] 1. System Configuration

[2097] 1.1. Sensor means

[2098] The sensor means installed in the terminal includes a camera and microphone that detect and collect the voice, facial expressions, and behavior of the pet, allowing you to obtain the pet's status in real time.

[2099] 1.2. Pre-processing methods

[2100] The preprocessing means in the terminal preprocesses the acquired data in order to properly analyze it. For example, video data is divided into frames and noise is removed. Audio data is also preprocessed in the same way.

[2101] 1.3. Means of communication

[2102] The terminal transmits the preprocessed data to the server using a communication means, which can be Wi-Fi or mobile communication.

[2103] 1.4. Server Means

[2104] The server means uses a computer system with advanced computing power, which inputs the received data into an image recognition model or a voice recognition model and analyzes the data.

[2105] 1.5. Message Creation Methods

[2106] The message generation means in the server generates a message in natural language based on the analysis results. This message is intended to convey the pet's intentions to the owner in an easy-to-understand manner.

[2107] 1.6. User Interface Methods

[2108] The terminal displays the messages sent from the server to the owner via the user interface means, allowing the owner to easily check the status and needs of their pet.

[2109] 1.7. Speech synthesis methods

[2110] The voice synthesis means in the server converts the owner's input into voice data, allowing the owner to give voice instructions to their pet.

[2111] 1.8. Audio Output Methods

[2112] The audio output means installed in the terminal reproduces the received audio data and conveys instructions to the pet.

[2113] 1.9. Emotion Engine

[2114] The emotion engine includes technology to analyze the user's voice and facial expressions and recognize their emotions, allowing the system to understand the user's emotional state and reflect it in message generation and voice instructions.

[2115] 2. System Operation

[2116] Example 1: When your pet needs to go to the toilet

[2117] 1. Device: The device's camera and microphone detect the pet's facial expressions and gestures and collect video and audio data.

[2118] 2. Terminal: The preprocessing means divides the collected video data into frames and removes noise. The audio data is also preprocessed in the same way.

[2119] 3. Terminal: Uses communication means to send preprocessed data to the server.

[2120] 4. Server: The server receives the data sent from the device and temporarily stores it in a database.

[2121] 5. Server: The received video data is input into an image recognition model (e.g., CNN) to analyze the pet's facial expressions and behaviors. Specifically, it extracts features such as the pet's ears, tail, and posture.

[2122] 6. Server: Input the audio data into a speech recognition model (e.g., RNN or LSTM) to analyze the tone and patterns of the pet's voice.

[2123] 7. Server: Integrates the results of image and voice recognition to determine the pet's intention. In this case, the intention is determined to be "I want to go to the toilet."

[2124] 8. Server: The message generation means generates a natural language message saying "I want to go to the toilet."

[2125] 9. Server: Sends the generated message to the terminal.

[2126] 10. Terminal: The terminal receives the message sent from the server and displays it on the user interface.

[2127] Example 2: When an owner says to their pet, "Let's go for a walk"

[2128] 1. User: The owner types "Let's go for a walk" in the chat on the LINE app, or gives the command "Let's go for a walk" by voice.

[2129] 2. Device: The device receives the text message and voice data and performs preprocessing. The voice data is converted to text.

[2130] 3. Terminal: Uses communication means to send the acquired data to the server.

[2131] 4. Server: The server receives the data sent from the device and begins analyzing it.

[2132] 5. Server: Analyzes the voice instruction data and understands the instruction "Let's go for a walk."

[2133] 6. Server: The speech synthesis means converts the text data into speech data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[2134] 7. Server: Sends the generated audio data to the device.

[2135] 8. Device: The device receives the voice data from the server and plays it through the speaker, instructing the pet to "go for a walk."

[2136] 9. Server: The emotion engine analyzes the user's voice and facial expressions to recognize their emotions. For example, it can distinguish between commands from a happy owner and commands from a tired owner.

[2137] 10. Server: Based on the results of the emotion engine, adjusts message generation and voice synthesis, and conveys instructions to the pet in an appropriate tone.

[2138] In this way, the system of the present invention enables smooth and intuitive communication between pets and their owners by linking the terminal and server, and also enables more effective interactions by taking the user's emotions into consideration.

[2139] The processing flow will be explained below.

[2140] Example 1: When your pet needs to go to the toilet

[2141] Step 1:

[2142] Terminal

[2143] The device's camera and microphone detect your pet's facial expressions and movements in real time, collecting video and audio data.

[2144] Step 2:

[2145] Terminal

[2146] The pre-processing means divides the collected video data into frames and removes noise, and also pre-processes the audio data in the same way.

[2147] Step 3:

[2148] Terminal

[2149] The communication means prepares the pre-processed data for transmission to the server, where it is standardized and converted into a format suitable for communication.

[2150] Step 4:

[2151] Terminal

[2152] A communication means is used to transmit the pre-processed data to a server.

[2153] Step 5:

[2154] server

[2155] The server receives the data sent from the terminal and temporarily stores it in a database.

[2156] Step 6:

[2157] server

[2158] The received video data is input into an image recognition model (e.g., CNN) to analyze the pet's facial expressions and behaviors. Specifically, features such as the pet's ears, tail, and posture are extracted.

[2159] Step 7:

[2160] server

[2161] The audio data is input into a speech recognition model (e.g., RNN or LSTM) to analyze the tone and patterns of the pet's voice.

[2162] Step 8:

[2163] server

[2164] The results of image and voice recognition are combined to comprehensively determine the pet's intention. In this case, the intention is determined to be "I want to go to the toilet."

[2165] Step 9:

[2166] server

[2167] The message generating means generates a natural language message saying "I want to go to the toilet."

[2168] Step 10:

[2169] server

[2170] Send the generated message to the terminal.

[2171] Step 11:

[2172] Terminal

[2173] The terminal receives the message sent from the server and displays it on the user interface.

[2174] Step 12:

[2175] User

[2176] The user (owner) checks the notification on the device and understands that the pet wants to go to the toilet.

[2177] Example 2: When an owner says to their pet, "Let's go for a walk"

[2178] Step 1:

[2179] User

[2180] The owner can either type "Let's go for a walk" in the chat on the LINE app or give the command "Let's go for a walk" by voice.

[2181] Step 2:

[2182] Terminal

[2183] The device receives text messages and voice data, performs preprocessing, and converts the voice data into text through speech recognition.

[2184] Step 3:

[2185] Terminal

[2186] The acquired text message and voice data are sent to a server using a communication means.

[2187] Step 4:

[2188] server

[2189] The server receives the data sent from the terminal and begins analyzing it.

[2190] Step 5:

[2191] server

[2192] It analyzes received text messages and voice data and understands the instructions such as "Let's go for a walk."

[2193] Step 6:

[2194] server

[2195] A voice synthesis means converts the text data into voice data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[2196] Step 7:

[2197] server

[2198] The generated voice data is transmitted to the terminal.

[2199] Step 8:

[2200] Terminal

[2201] The device receives voice data from the server and plays it through the speaker, instructing your pet to "go for a walk."

[2202] Step 9:

[2203] server

[2204] The emotion engine analyzes the user's voice and facial expressions to recognize their emotions. For example, it can distinguish between commands from a happy owner and commands from a tired owner.

[2205] Step 10:

[2206] server

[2207] Based on the results of the emotion engine, message generation and voice synthesis are adjusted to convey instructions to pets in an appropriate tone.

[2208] In this way, the system of the present invention enables smooth and intuitive communication between pets and their owners by linking the terminal and server, and also enables more effective interactions by taking the user's emotions into consideration.

[2209] Example 2

[2210] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2211] In today's communication between pets and their owners, it is difficult to accurately understand the pet's intentions and emotions, which often results in delayed appropriate responses. Furthermore, when an owner gives a command to a pet, the pet may not accurately understand the owner's intentions, making effective communication difficult. To solve these issues, a system is needed that can accurately detect and analyze a pet's condition and appropriately communicate the results to the owner.

[2212] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a sensor means for detecting and collecting the pet's status, a preprocessing means for dividing the acquired video data into frames and removing noise, a communication means for transmitting the preprocessed data to the server, a calculation means for analyzing the data and determining the pet's intention, a message generation means for generating a message in natural language based on the determined intention, a communication means for transmitting the generated message to the terminal, a user interface means for displaying the received message, an input means for accepting user input, a communication means for transmitting the user input to the server, a synthesis means for analyzing the user input received by the server and generating voice instructions, a communication means for transmitting the generated voice instructions to the terminal, an output means for playing back the received voice instructions, and an emotion recognition means for analyzing the user's voice and facial expressions to recognize emotions. This enables smooth and intuitive communication between the pet and its owner.

[2213] "Sensor means" refers to means that use devices such as cameras and microphones to detect and collect information about the condition of pets.

[2214] The "preprocessing means" is a means for processing the acquired data into a form that is easier to analyze. Specifically, the preprocessing means divides the video data into frames and performs processing to remove noise.

[2215] "Communication means" refers to the means for sending and receiving preprocessed data and generated messages between servers and terminals, and uses Wi-Fi or mobile communications.

[2216] The "calculation means" is a means for analyzing the received data and determining the pet's intentions, and uses an image recognition model or a voice recognition model.

[2217] The "message generating means" is a means for generating a natural language message that conveys the pet's intention based on the analysis results.

[2218] "User interface means" refers to means for displaying received messages to the owner, allowing the owner to check the status and needs of their pet.

[2219] The "input means" is a means for accepting instructions or input from the user, and accepts text or voice input.

[2220] A "synthesizer" is a means for analyzing a user's input and generating appropriate voice instructions.

[2221] The "output means" refers to a means for reproducing the generated voice instructions, and uses a device such as a speaker.

[2222] The "emotion recognition means" is a means for analyzing the user's voice and facial expression and recognizing the user's emotion.

[2223] MODE FOR CARRYING OUT THE INVENTION

[2224] The present invention is a system that analyzes a pet's facial expressions, voice, and behavior, and uses that information to enable effective communication between the owner and the pet. This system includes multiple technical elements that operate in conjunction with a terminal and a server. The system configuration and specific operation are described below.

[2225] System Configuration

[2226] Sensor means

[2227] The device is equipped with a camera and microphone, which detect and collect information about your pet's condition in real time. For example, the camera captures images of your pet's ears, tail, and posture, while the microphone records their cries and barks.

[2228] Pretreatment means

[2229] The pre-processing means within the device has the function of processing the collected data into a form that is easier to analyze. Specifically, it divides the video data into frames and removes noise. It also performs noise removal processing on audio data. For example, the device divides the video into frames every few seconds and automatically adjusts the brightness and color tone from each frame.

[2230] communication means

[2231] The device has a communication means to send the preprocessed data to the server. This communication means can be Wi-Fi or mobile communication. For example, the device checks for a Wi-Fi connection and uploads the data to the server via a stable network.

[2232] Server Means

[2233] The server is a computer system with advanced computing power that inputs the received data into an image recognition model (e.g., CNN) or a voice recognition model (e.g., RNN or LSTM) to analyze the pet's facial expressions and behaviors. Specifically, the server uses a deep learning model to detect the position of the ears and tail from the video and stress or happiness from the voice.

[2234] Message Generation Method

[2235] The message generation means in the server generates and transmits a natural language message that conveys the pet's intentions based on the analysis results, such as "I want to go to the toilet" or "I want to play."

[2236] User Interface Means

[2237] The device displays messages sent from the server to the owner through a user interface, allowing the owner to easily check the pet's status and needs. For example, messages can be displayed as a pop-up notification on a smartphone or tablet.

[2238] Voice synthesis means

[2239] The server's voice synthesis means converts user input into voice data, allowing users to give voice commands to their pets. For example, if an owner sends a message to their pet via the LINE app saying "Let's go for a walk," the server converts the message into natural-sounding voice.

[2240] Audio output means

[2241] The device's built-in voice output means plays back the received voice data to communicate instructions to your pet. For example, a natural voice will be played from the device's speaker saying, "Let's go for a walk."

[2242] emotion recognition means

[2243] The emotion recognition means includes technology that analyzes the user's voice and facial expressions and recognizes their emotions. This allows the system to understand the user's emotional state and reflect it in message generation and voice instructions. For example, the system can distinguish between instructions from an owner who is happy and instructions from an owner who is tired and communicate them to the pet in an appropriate tone.

[2244] Specific examples

[2245] Example 1: When your pet needs to go to the toilet

[2246] The device's camera and microphone detect your pet's facial expressions and gestures, collecting video and audio data.

[2247] The pre-processing means divides the collected video data into frames and removes noise, and also pre-processes the audio data in the same way.

[2248] The preprocessed data is sent to the server using a communication means.

[2249] The data received by the server is input into image and voice recognition models for analysis, and the intent is determined to be "I want to go to the toilet."

[2250] The message generating means generates a natural language message saying "I want to go to the toilet" and transmits it to the terminal.

[2251] The terminal receives the message sent from the server and displays it on the user interface.

[2252] Example 2: When an owner says to their pet, "Let's go for a walk"

[2253] The owner can either type "Let's go for a walk" in the chat on the LINE app or give the command "Let's go for a walk" by voice.

[2254] The device receives text messages and voice data, performs preprocessing, and converts the voice data into text.

[2255] The acquired data is sent to the server using a communication means.

[2256] The server analyzes the received data and understands the instruction "Let's go for a walk."

[2257] A voice synthesis means converts the text data into voice data and generates the instruction "Let's go for a walk" in a natural pronunciation.

[2258] The generated voice data is transmitted to the terminal.

[2259] The device receives voice data from the server and plays it through the speaker, instructing your pet to "go for a walk."

[2260] Examples of prompt statements

[2261] Example inputs to a generative AI model:

[2262] "Please explain how to analyze a pet's gestures and voice to communicate its intentions to its owner."

[2263] "Please explain how the system works to notify pet owners appropriately when their pets behave in a certain way."

[2264] As described above, the system of the present invention enables smooth and effective communication between pets and their owners by linking the terminal and server, and also takes into account the user's emotions, enabling more intuitive and friendly communication.

[2265] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2266] Step 1: Collect pet status

[2267] Subject: Device

[2268] Description: The device uses a camera and microphone to detect your pet's condition (facial expressions, voice, and behavior) in real time and collects this data.

[2269] Input: Your pet's real-life facial expressions, sounds, and gestures

[2270] Output: Collected video and audio data

[2271] Specific operations: The device's camera captures your pet's ears, tail, posture, etc., and the microphone records any meows or barks.

[2272] Step 2: Preprocessing the data

[2273] Subject: Device

[2274] Description: The preprocessing means divides the collected video data into frames and removes noise. The audio data is also subjected to noise removal.

[2275] Input: Collected video and audio data

[2276] Output: Preprocessed video frames and audio data

[2277] What it does: The device splits the video into frames every few seconds, then automatically adjusts the brightness and color of each frame to remove noise.

[2278] Step 3: Sending data

[2279] Subject: Device

[2280] Description: The pre-processed data is sent to the server via a communication method, in this case Wi-Fi or mobile communication.

[2281] Input: Preprocessed video frames and audio data

[2282] Output: Data sent to the server

[2283] What happens: The device checks for a Wi-Fi connection and uploads data to the server over a stable network.

[2284] Step 4: Analyze the data

[2285] Subject: Server

[2286] Description: The server inputs the received data into an image recognition model (e.g., CNN) or a speech recognition model (e.g., RNN or LSTM) for analysis.

[2287] Input: Preprocessed data sent to the server

[2288] Output: Parsed pet's intention and state information

[2289] Specific operation: The server uses a deep learning model to detect the position of the ears and tail from video footage, and emotions and intentions such as stress or joy from audio.

[2290] Step 5: Generate a message

[2291] Subject: Server

[2292] Description: Based on the analysis results, the message generator generates a natural language message that conveys the pet's intentions.

[2293] Input: Parsed pet intent and state information

[2294] Output: Natural language message

[2295] Concrete behavior: The process of generating simple messages such as "I want to go to the toilet" or "I want to play."

[2296] Step 6: Sending a message

[2297] Subject: Server (terminal)

[2298] Description: The server generates a message and sends it to the terminal, which receives it.

[2299] Input: A natural language message

[2300] Output: Message sent to the terminal

[2301] Specific operation: The server sends a message to the terminal, and the terminal receives the message.

[2302] Step 7: Displaying messages

[2303] Subject: Device

[2304] Description: Displays received messages to the owner through user interface means.

[2305] Input: Message sent to terminal

[2306] Output: The message displayed in the user interface

[2307] What happens: A message will pop up through your smartphone or tablet's notification feature.

[2308] Step 8: Accepting User Input

[2309] Subject: User

[2310] Description: The user enters instructions in the form of speech or text.

[2311] Input: User instructions (voice or text message)

[2312] Output: User's instruction data

[2313] Specific actions: The owner types "Let's go for a walk" in the chat on the LINE app, or gives voice instructions.

[2314] Step 9: Sending User Input

[2315] Subject: Device

[2316] Description: Preprocesses input from the user and sends it to the server via a communication medium.

[2317] Input: User-specified data

[2318] Output: User instruction data sent to the server

[2319] Specific operation: The process of converting input voice data into text and sending it to the server.

[2320] Step 10: Generate voice instructions

[2321] Subject: Server

[2322] Description: The server analyzes the user's input and generates appropriate voice instructions using a speech synthesis method.

[2323] Input: User instruction data sent to the server

[2324] Output: Generated voice instruction data

[2325] Specific operation: Convert the text message "Let's go for a walk" into voice data with natural pronunciation.

[2326] Step 11: Sending voice instructions

[2327] Subject: Server (terminal)

[2328] Description: The server generates and sends voice instructions to the device, which receives them.

[2329] Input: Generated voice command data

[2330] Output: Audio instructions sent to the device

[2331] Specific operation: The server sends voice data to the terminal, and the terminal receives the data.

[2332] Step 12: Play audio instructions

[2333] Subject: Device

[2334] Description: Plays back the received audio instructions through the audio output means.

[2335] Input: Voice commands sent to the device

[2336] Output: The audio instructions played

[2337] Specific operation: "Let's go for a walk" is played in a natural voice from the device's speaker.

[2338] (Application example 2)

[2339] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2340] Currently, stores that cater to customers with pets need to quickly and accurately understand the pet's condition and respond appropriately. However, it is difficult to understand the pet's intentions in real time, which can lead to delayed responses, especially in emergencies. In addition, smooth communication between pets, owners, and store staff is required, but an efficient system to achieve this does not exist.

[2341] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for determining the pet's intention, a notification means for sending a notification to the user and staff, and a means for analyzing the pet's condition using an image recognition model and a voice recognition model. This makes it possible to monitor the pet's condition in real time and take appropriate measures, including emergency notifications. It also enables the user and staff to quickly understand the pet's condition and take appropriate measures.

[2342] "Pets" are animals kept by humans, such as dogs and cats.

[2343] "Audio" refers to the voices and sounds made by pets.

[2344] "Expression" refers to the facial expressions and changes in your pet's face.

[2345] "Gestures" refer to the body movements and actions of pets.

[2346] "Sensor means" refers to a device such as a camera or microphone for detecting the voice, facial expressions, and behavior of a pet.

[2347] "Preprocessing means" refers to means for converting collected data into a form that is easy to analyze.

[2348] "Communication means" refers to a means for transmitting pre-processed data to a server.

[2349] The "server means" refers to a means for analyzing the received data and determining the intention of the pet.

[2350] The "message generating means" refers to a means for generating a message in natural language based on the determined intention.

[2351] "User interface means" refers to means for displaying generated messages to a user.

[2352] A "voice synthesis means" is a means for analyzing input from a user and generating voice instructions.

[2353] "Audio output means" refers to means for reproducing the generated audio instructions.

[2354] "Notification means" refers to a means for determining the pet's condition and sending notifications to the user and staff.

[2355] An "image recognition model" is a machine learning model for analyzing pets' facial expressions and behaviors.

[2356] A "voice recognition model" is a machine learning model for analyzing pet voices.

[2357] This invention is a support system for customers who bring pets to physical stores, and as a specific example, we will explain a "pet support app." This application has the function of monitoring the pet's condition in real time and sending appropriate notifications to users and staff.

[2358] Hardware and software used

[2359] Hardware

[2360] Camera: Used to capture your pet's expressions and behavior in real time.

[2361] Microphone: Used to record your pet's voice.

[2362] Smartphone: The above sensors (camera and microphone) are built in.

[2363] Server: A high-performance computer system that analyzes data and stores results.

[2364] software

[2365] OpenCV: A library for acquiring and preprocessing video data from a camera.

[2366] TensorFlow / Keras: Machine learning libraries for implementing image recognition models to analyze pet facial expressions and behaviors.

[2367] SpeechRecognition: A library for converting your pet's spoken commands into text.

[2368] Requests: A library for sending preprocessed data to a server.

[2369] Data processing and calculation

[2370] Terminal handling

[2371] The device activates the camera and microphone to capture the pet's facial expressions, gestures, and voice in real time. Then, it splits the vid...

Claims

1. a sensor means for acquiring the voice, facial expression, and behavior of the pet; means for pre-processing the acquired data; a communication means for transmitting the preprocessed data to a server; a server means for analyzing the data and determining the intention of the pet; a message generating means for generating a message in natural language based on the determined intention; means for transmitting the generated message to a terminal; user interface means for displaying received messages; means for accepting user input; a communication means for transmitting input from a user to a server; a speech synthesis means for analyzing user input received at the server and generating speech instructions; means for transmitting the generated voice instructions to the terminal; audio output means for playing back the received audio instructions; A system including:

2. 10. The system of claim 1, including server means for using image recognition models and voice recognition models to determine the intention of the pet.

3. 10. The system of claim 1, wherein the terminal includes means for activating a camera and a microphone to capture video and audio of the pet.

Images