system

Abstract

We provide the system. [Solution] A video acquisition means for acquiring and pre-processing video data of animals, A sound acquisition means for acquiring animal sound data and performing preprocessing, An analysis means analyzes the data acquired by the aforementioned video acquisition means and audio acquisition means to estimate the emotions and demands of the animals based on their behavior and vocalizations. A means of converting human speech into signals or sounds that animals can understand, A notification means for notifying the user of the data analyzed by the aforementioned analysis means, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional means for grasping the emotions and demands of animals are limited, and it is difficult to achieve smooth communication between animals and humans. Therefore, it is impossible to accurately understand what the animal wants to convey and respond appropriately, and situations may occur where the needs and stresses of the animal are overlooked. In addition, there is also a lack of means for making animals understand human intentions, and there is a problem that smooth two-way communication cannot be achieved.

Means for Solving the Problems

[0005] This invention provides a system that acquires and analyzes video and audio data of animals to estimate their emotions and needs in real time. The data obtained by the video and audio acquisition means is analyzed by the analysis means to estimate emotions and needs based on the animal's behavior and vocalizations. Furthermore, a conversion means converts human speech into signals or sounds that animals can understand, enabling two-way communication. The analysis results are provided to the user through a notification means, allowing the user to understand the animal's needs and respond appropriately. In addition, a learning means improves analysis accuracy by learning individual differences among animals, resulting in more accurate estimations.

[0006] "Image acquisition means" refers to a device or function that acquires image data of animals and preprocesses it into a format suitable for analysis.

[0007] "Voice acquisition means" refers to a device or function that acquires animal voice data and performs noise reduction and normalization for analysis.

[0008] "Analysis means" refers to a device or function for analyzing animal behavior and vocalizations based on acquired video and audio data, and for estimating emotions and needs.

[0009] "Conversion means" refers to a device or function for converting human speech into signals or sounds that animals can understand.

[0010] "Notification means" refers to a device or function that provides a method for communicating the analyzed results to the user.

[0011] A "learning tool" is a device or function used to learn the individual differences in the behavior and vocalizations of individual animals and to improve the accuracy of analysis. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2]This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0013] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0018] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This invention is a system that analyzes animal behavior and vocalizations to estimate the emotions and needs of animals in real time. This system consists of a user terminal and a server.

[0034] The user places a device near the animal and acquires video and audio data through the camera and microphone. The device temporarily stores the acquired data, performs noise reduction and resolution adjustments, and prepares it for data transfer to the server.

[0035] The server uses video recognition AI and voice recognition AI to analyze data received from the terminal. Based on the video data, it analyzes the animal's movements and posture, and from the voice data, it detects the characteristics of the animal's vocalizations. Based on this information, the server estimates the animal's emotions and needs.

[0036] The estimated emotions and requests are sent from the server to the terminal and notified to the user. For example, if a dog is jumping while wagging its tail, the server will determine that the dog is "happy" and convey that information to the user through the terminal.

[0037] Furthermore, when a user enters a command into the terminal, the server converts that command into a signal or sound that the animal can understand. For example, if the command "stay" is entered, the server converts it into a sound signal of a specific frequency and transmits it to the animal.

[0038] This system also has a function to improve analysis accuracy by learning individual differences in animal behavior and vocalizations through learning methods. This allows it to learn the tendencies and habits of specific animals, enabling more accurate estimation of their emotions and needs.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The device uses cameras and microphones placed near the animals to continuously acquire video and audio data. This prepares it to capture the animals' behavior and vocalizations in real time.

[0042] Step 2:

[0043] The terminal performs noise reduction and normalization processing on the acquired video and audio data to improve data quality. This processing is an important pre-processing step for improving analysis accuracy.

[0044] Step 3:

[0045] The terminal sends the pre-processed data to the server. This data transfer is an essential step for starting the analysis process.

[0046] Step 4:

[0047] The server uses video recognition AI to analyze the received video data and detect the animal's posture and movements. This makes it possible to objectively evaluate the animal's behavior.

[0048] Step 5:

[0049] The server runs a speech recognition AI to analyze the received audio data. The analysis identifies the frequency and patterns of the animal's calls and infers its emotions and needs.

[0050] Step 6:

[0051] The server integrates the video and audio analysis results and uses an AI model to estimate the animal's emotions and needs. The estimation results provide an overall assessment of the animal's condition.

[0052] Step 7:

[0053] The server sends estimated emotions and requests to the terminal and notifies the user. This allows the user to receive immediate feedback about the animal's condition.

[0054] Step 8:

[0055] When a user wants to give instructions to an animal, they input those instructions into a device. These instructions are then used for communication with the animal.

[0056] Step 9:

[0057] The server receives instructions sent by the user and converts them into signals or sounds that the animal can understand. This conversion process ensures that the user's intentions are accurately conveyed to the animal.

[0058] Step 10:

[0059] The server accumulates historical data to learn about individual differences among animals and improves the accuracy of its analysis through learning mechanisms. This continuous learning enables more accurate and reliable analysis.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] Accurately understanding animals' emotions and needs in real time has traditionally been difficult. This has led to problems such as inflicting unnecessary stress on animals or failing to provide appropriate care. Furthermore, there has been a lack of appropriate analytical methods that take individual animal differences into account, creating a need for more precise decision-making.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes an image acquisition means for acquiring and pre-processing visual information of an animal, an acoustic acquisition means for acquiring and pre-processing auditory information of an animal, and an information processing means for analyzing the information acquired by the image acquisition means and the acoustic acquisition means to estimate emotions and requests based on the animal's movements and vocalizations. This enables real-time and highly accurate analysis of emotions and requests based on the animal's behavior and vocalizations.

[0065] "Visual information acquisition means" refers to means for acquiring video data to record the movements and postures of animals and for pre-processing that data.

[0066] A "sound information acquisition means" is a means for acquiring data related to animal sounds and other sounds, and for preprocessing that data.

[0067] "Information processing means" refers to means for analyzing acquired visual and auditory information to estimate the emotions and needs of animals.

[0068] A "conversion means" is a means of converting human instructions into signals or sounds that can be recognized by animals.

[0069] A "notification means" is a means of notifying the user of the results analyzed by an information processing means.

[0070] A "generative AI model" is an artificial intelligence model used to analyze patterns of animal behavior and vocalizations and to estimate their emotions and needs.

[0071] The "learning function" is a function that learns about individual differences in animals and improves the accuracy of analysis.

[0072] This invention is a system that analyzes animal behavior and vocalizations in real time and estimates their emotions and needs. The system consists of a user, a terminal, and a server.

[0073] The user places the device close to the animal. The device is equipped with a camera and microphone, and acquires video and audio data of the animal in real time. This data is temporarily stored in the device and pre-processed, such as noise reduction and resolution adjustment.

[0074] The terminal digitizes the pre-processed data and sends it to the server. Examples of terminals include typical smartphones and dedicated IoT devices.

[0075] The server is located in the cloud and runs high-performance image recognition AI and speech recognition AI. For image recognition, software libraries such as OpenCV and TENSORFLOW® are commonly used. For speech recognition, established speech analysis technologies such as Google® Speech-to-Text API are employed. The server analyzes the received data and comprehensively estimates emotions and requests based on animal movements, postures, and vocal patterns.

[0076] This analysis process utilizes a generative AI model that learns from previously accumulated animal behavior data to achieve highly accurate estimations that take individual differences into account. The analysis results are immediately transmitted to the terminal and notified to the user.

[0077] Furthermore, users can issue commands to animals through a terminal. When a user enters a command into the terminal, the server converts that command into signals or sounds that the animals can recognize and provides them. This conversion adjusts the frequency, volume, and other parameters to be specific to each animal.

[0078] As a concrete example, consider a scenario where a user is spending time with a dog in a park. The device captures the dog wagging its tail and obtains the corresponding audio. The server analyzes this and estimates that the dog is "happy." This information is then communicated to the user, enabling a more appropriate response. Another example of a prompt for the generative AI model is a sentence like, "Detect how long the dog is barking and estimate the reason."

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1:

[0081] The user places a terminal near the animal and uses the camera and microphone to acquire visual and auditory information. The input is real-time video and audio data of the animal. The terminal temporarily stores this data and performs noise reduction and resolution adjustments. The output is pre-processed digital data. This step improves the quality of the acquired data and optimizes it for analysis on the server.

[0082] Step 2:

[0083] The terminal packets pre-processed data and sends it to the server. The input is pre-processed video and audio data. Data format conversion is performed to efficiently send the data to the server. As output, transmittable data packets are generated. This step includes implementations to ensure secure and rapid data transfer.

[0084] Step 3:

[0085] The server receives data packets sent from the terminal. The input is the transmitted data packets. After verifying the reliability and integrity of the data, the server starts the analysis. The output is data ready for analysis. Receiving and verifying the data is a crucial operation for fulfilling the prerequisites for data analysis.

[0086] Step 4:

[0087] The server performs analysis using video recognition AI and speech recognition AI based on the received data. The input is pre-processed data deployed on the server. From the video data, the movements and postures of animals are analyzed, and from the audio data, patterns of vocalizations are identified. Emotions and requests are estimated using a generative AI model. As output, data of estimated emotions and requests is obtained. The generative AI model learns individual differences, enabling highly accurate estimations.

[0088] Step 5:

[0089] The server converts the estimated results into a data format and sends it to the terminal. The input is the estimated result data. The data is converted into an appropriate format for communication with the user. The output is data that can be communicated to the user. This step includes actions to organize the information the user receives in an easy-to-understand manner.

[0090] Step 6:

[0091] The terminal analyzes data received from the server and notifies the user. The input is estimated data sent from the server. The information is displayed in an appropriate format, and alerts are given via voice or other means. The output is the information delivered to the user. In this step, feedback is provided through an interface that the user can instantly understand.

[0092] Step 7:

[0093] Users can issue commands to animals by entering specific commands into a terminal. The input consists of commands from the user. These commands are sent to a server and converted into signals or sounds that animals can understand. The converted instructions are then transmitted to the animals as output. This process is designed to facilitate communication between animals and humans.

[0094] (Application Example 1)

[0095] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0096] In modern times, a challenge in communication between animals and humans is the difficulty in accurately understanding an animal's emotions and needs and responding appropriately based on that understanding. There is also a need to maintain smooth communication with pets even when away from home or during absences. Furthermore, there is the problem of difficulty in understanding an animal's condition in real time and responding appropriately.

[0097] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0098] In this invention, the server includes means for acquiring and pre-processing video information of animals, means for acquiring and pre-processing audio information of animals, and means for analyzing the acquired data and estimating emotions and requests based on the animal's behavior and vocalizations. This makes it possible to accurately grasp the animal's emotions and requests in real time and to transmit instructions to the animal via a smart device even when away from home.

[0099] "Image acquisition means" refers to a device that has the function of acquiring image information of animals using sensors or cameras and performing preprocessing as necessary.

[0100] A "sound acquisition means" is a device that has the function of acquiring animal sound information using a microphone or sound sensor and performing preprocessing as necessary.

[0101] "Analysis means" refers to a system that includes technology for analyzing acquired video and audio information and estimating the emotions and needs of animals based on their behavior and vocalizations.

[0102] A "conversion device" is a device that has the function of converting human instructions into signals or sounds that animals can understand.

[0103] A "notification device" is a device that has the function of notifying the user of the data analyzed by the analysis device, enabling them to immediately understand the animal's condition.

[0104] "Control means" refers to technology that connects to mechanical devices for animals and plays a role in providing the necessary control to ensure smooth communication.

[0105] The system for realizing this invention analyzes animal behavior and vocalizations in real time and notifies the user of their emotions and requests. This system mainly consists of terminals and servers.

[0106] The terminal is placed near the animal and uses a camera and microphone to acquire video and audio information about the animal. The terminal temporarily stores this information and performs preprocessing such as noise reduction and resolution adjustment. After that, the terminal prepares to transfer the preprocessed data to the server.

[0107] The server receives the transferred data as described above and analyzes it using image recognition AI and speech recognition AI. In this process, it uses AI frameworks such as TensorFlow and PyTorch to analyze the behavioral and vocal characteristics of animals and estimate their emotions and requests.

[0108] The estimated data is communicated to the user via a terminal. This allows the user to monitor the animal's condition in real time and take necessary actions. Furthermore, instructions entered by the user into the terminal are converted by a server into signals or sounds that the animal can understand. These converted instructions are transmitted to the animal's mechanical devices via control means, enabling smooth communication.

[0109] As a concrete example, one day while a user is out, they receive a notification that their pet dog is in an excited state. Based on this information, the user inputs the command "sit," and this command is transmitted to the dog via the device, reducing the dog's stress.

[0110] An example of a prompt might be: "Develop an app that estimates a pet's emotions from its behavior and sounds and sends a notification to the owner. For example, if the dog wags its tail or jumps, the app would notify the owner that the pet is 'happy'."

[0111] This invention aims to improve analysis accuracy by continuously learning individual differences in animal behavior and vocalizations, thereby providing more appropriate and accurate estimation results.

[0112] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0113] Step 1:

[0114] The device uses a camera and microphone to acquire video and audio information about animals. The input is camera video and microphone audio, and the output is raw video and audio data. Specifically, it works by pointing the camera at the animal and operating the microphone near the animal to acquire information.

[0115] Step 2:

[0116] The video and audio data acquired by the device are preprocessed by noise reduction and resolution adjustment. The input is the raw data acquired in step 1, and the output is the processed video and audio data. Specifically, noise is removed using filtering technology, and adjustments are made if the image quality is not clear.

[0117] Step 3:

[0118] The terminal sends the pre-processed data to the server. The input is the processed data generated in step 2, and the output is the data transferred to the server. Specifically, the data is uploaded to the server via the network using a communication protocol.

[0119] Step 4:

[0120] The server analyzes the received data using video recognition AI and speech recognition AI. The input is processed data sent from the terminal, and the output is an estimation of emotions and requests based on the animal's behavioral and vocal characteristics. Specifically, a generative AI model is used to analyze data patterns and obtain estimation results.

[0121] Step 5:

[0122] The server notifies the terminal of the estimation results, and the information is conveyed to the user. The input is the estimation results generated by the server, and the output is the information presented to the user. Specifically, information is sent to the user's terminal using push notifications, and the user is notified visually or audibly.

[0123] Step 6:

[0124] The user inputs instructions to the terminal. The input consists of commands performed by the user, and the output is instruction data sent to the server. Specifically, instructions such as "sit" or "stay" are input through screen operations.

[0125] Step 7:

[0126] The server converts user instructions into signals or sounds that animals can understand and transmits them to the animal's mechanical device via a terminal. The input is user instruction data, and the output is the converted signals or sounds. Specifically, a generative AI model is used to create signals, which are then transmitted to the animal using control devices.

[0127] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0128] This invention combines a system that analyzes animal behavior and vocalizations with an emotion engine that recognizes user emotions, in order to establish communication between animals and humans. This system consists of a terminal operated by the user and a server that performs data analysis.

[0129] The terminal uses cameras and microphones placed near the animals to collect video and audio data of the animals in real time. The terminal preprocesses the data, removes noise, and then transmits it to a server. This data is used to analyze the animals' behavior and vocal patterns.

[0130] The server analyzes the received video and audio data to estimate the animal's emotions and needs. Based on the information obtained through the analysis, it determines whether the animal is happy, hungry, or anxious.

[0131] Furthermore, the emotion engine analyzes the user's tone of voice, word choice, and even facial expressions from video data to identify the user's emotions. This allows it to detect the user's emotional state, such as whether they are stressed or calm.

[0132] When a user gives instructions to an animal, they input the content into a terminal. The server takes into account the user's emotional state, as detected by the emotion engine, and converts the instructions into signals or sounds that the animal can understand. This process ensures that, for example, even if the user is excited and gives instructions in a strong tone, the signals are adjusted to a gentle tone so that the animal does not experience unnecessary stress.

[0133] For example, when a user gives the command "sit," if the server senses impatience in the user's voice, it will convert the tone to a calmer one before transmitting it to the animal. In this way, the emotion engine plays a crucial role in facilitating smooth two-way communication.

[0134] Through the above process, smooth communication can be established between animals and users, enabling a rapid response to the animals' needs.

[0135] The following describes the processing flow.

[0136] Step 1:

[0137] The device uses a camera and microphone to acquire video and audio data of animals. This process records the animals' movements and sounds in real time.

[0138] Step 2:

[0139] The terminal performs preprocessing on the acquired data, including noise reduction and normalization. This improves the quality of the data, which enhances the accuracy of the analysis.

[0140] Step 3:

[0141] The terminal sends the pre-processed video and audio data to the server. At this stage, the data is ready for analysis.

[0142] Step 4:

[0143] The server uses video recognition AI to analyze animal behavior and detect posture and movements. This information forms the basis for determining the animal's emotions and needs.

[0144] Step 5:

[0145] The server uses speech recognition AI to analyze audio data and identify the frequency, tone, and pattern of animal calls. This allows it to estimate the animal's needs and mood.

[0146] Step 6:

[0147] The server utilizes an emotion engine to recognize the user's emotions. It analyzes the user's tone of voice, chosen words, and facial expressions to estimate their emotional state.

[0148] Step 7:

[0149] The server integrates the results of animal behavior and voice analysis with the user's emotional state, and converts the commands to the animal into gently adjusted signals and voices.

[0150] Step 8:

[0151] The server transmits the generated signal or sound to the terminal, which then relays it to the animal. In this process, the coordinated communication is delivered to the animal.

[0152] Step 9:

[0153] The user receives feedback from the system, observes and evaluates the animal's response, and adjusts instructions as needed to determine the next action.

[0154] In this way, two-way communication between animals and users is realized, enabling appropriate responses that meet the animals' needs and emotions.

[0155] (Example 2)

[0156] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0157] Facilitating smooth communication between animals and humans is extremely difficult. Conventional systems not only struggle to accurately understand animal emotions and needs, but also transmit information unilaterally without considering human emotional states, often leading to unsuccessful communication. A solution to this problem and a means of achieving meaningful communication between animals and humans are needed.

[0158] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0159] In this invention, the server includes data collection means for collecting and preprocessing animal behavior and voice data, analysis means for analyzing emotions and requests based on animal behavior and voice using a generative AI model, and user emotion analysis means for analyzing the tone of the user's voice and facial expressions and identifying their emotional state. This facilitates two-way communication between animals and humans, enabling a more accurate understanding of each other's emotions and requests.

[0160] "Data collection means" refers to a device or system that has the function of collecting and pre-processing animal behavior and sound data in real time.

[0161] "Data transfer means" refers to a device or system having communication protocols and functions for securely and quickly transferring pre-processed data to a server.

[0162] "Generative AI models" refer to artificial intelligence technologies used to analyze animal behavior and vocal data to estimate the emotions and needs of animals.

[0163] "Analysis means" refers to software or devices that utilize generative AI models to analyze emotions and requests based on animal behavior and vocalizations.

[0164] "User emotion analysis means" refers to software or a device that has the function of analyzing the tone of a user's voice and facial expressions to identify their emotional state.

[0165] "Conversion means" refers to software or a device that has the function of converting instructions given by a user into signals or sounds that an animal can understand.

[0166] "Notification means" refers to a device or system that has the function of transmitting information about the animal's condition obtained by the analysis means to the user.

[0167] A "learning tool" is software or a system that has the function of learning individual differences in animal behavior and vocalizations and improving the accuracy of analysis.

[0168] A "feedback improvement method" is a system that has the function of adjusting the generated AI model based on user feedback to improve its accuracy and effectiveness.

[0169] This invention is a system for facilitating communication between animals and humans. The system consists of a user-operated terminal and a server that performs data analysis. The terminal uses a camera and microphone placed near the animal to collect video and audio data of the animal in real time. After preprocessing the collected data, such as noise filtering, it is transferred to the server using a secure protocol.

[0170] The server analyzes the received video and audio data. Specifically, it uses a generative AI model to infer emotions and needs from animal behavior and vocalizations. This analysis determines whether the animal is happy, hungry, or anxious. Furthermore, the server uses an emotion engine to identify the user's emotional state based on the tone of voice and facial expressions provided by the user.

[0171] For example, consider a scenario where a user gives the command "sit." When the user inputs this command into the device, the server takes into account the user's emotional state, as determined by the emotion engine, and converts the command into signals and sounds that the animal can understand. Even if the user is excited and gives the command in a strong tone, the server converts the command into a calmer tone before conveying it to the animal. In this way, the system ensures smooth communication between the user and the animal.

[0172] A concrete example of a prompt message is, "When the user commands the dog to sit, try to communicate in a calm tone." Based on this prompt message, the server performs the necessary processing, and the tone and signals transmitted to the animal are adjusted accordingly.

[0173] This system allows for accurate understanding of animals' emotions and needs, and enables information transmission that appropriately considers the user's emotional state, thereby making communication with animals more meaningful.

[0174] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0175] Step 1:

[0176] The device uses a camera and microphone placed near the animal to collect video and audio data of the animal in real time. It takes in video data from the camera and audio data from the microphone as input, and improves the quality of the input data by applying noise filtering. The data, after this preprocessing is complete, is output in a format suitable for subsequent processing steps.

[0177] Step 2:

[0178] The terminal transfers pre-processed video and audio data to the server. This step uses a data transfer protocol (e.g., HTTP / 2 or WebSocket) to ensure security and speed. The input is filtered data, and the output is data transmitted via encrypted communication.

[0179] Step 3:

[0180] The server analyzes the received video and audio data. It utilizes a generative AI model to estimate emotions and needs from animal behavior and sounds. In this process, video and audio data are received as input, the model performs data analysis, and an estimated result is output, such as whether the animal is happy, hungry, or anxious.

[0181] Step 4:

[0182] The server identifies the user's emotional state from their voice tone and facial expressions. It receives audio and video data entered by the user into the terminal and analyzes it using an emotion engine. The input is the user's voice and video, and the output is the analysis result indicating the user's emotional state.

[0183] Step 5:

[0184] The server converts user instructions into signals or sounds that animals can understand. A generative AI model considers the user's emotional state and outputs instructions in a controlled tone that is not stressful for the animal. Input is the user's instructions and the emotional analysis results, while output is signals or sounds for the animal.

[0185] Step 6:

[0186] The user observes the animal's reactions and inputs the results as feedback into the device. This feedback information is sent to a server and used to improve the generated AI model. The input is feedback data based on the animal's reactions, while the output is data used to adjust and improve the AI ​​model.

[0187] (Application Example 2)

[0188] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0189] Systems for facilitating smooth communication between animals and humans need to accurately understand the animal's emotions and needs, and ensure that human instructions are properly conveyed to the animal. However, existing technologies struggle to accurately analyze animal behavior and vocalizations, and rarely adjust to the human emotional state. This can lead to miscommunication between animals and humans.

[0190] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0191] In this invention, the server includes a video acquisition means for acquiring and pre-processing video information of animals, a voice acquisition means for acquiring and pre-processing audio information of animals, an analysis means for analyzing the acquired data and estimating the animal's emotions and requests, a conversion means for converting human speech into a form that animals can understand, and an adjustment means for analyzing the user's emotional state and adjusting signals or voice tone. This improves the accuracy of communication between animals and humans, enabling both parties to interact without stress.

[0192] "Image acquisition means" refers to equipment or devices that have the function of collecting image information of animals and performing appropriate preprocessing.

[0193] "Voice acquisition means" refers to equipment or devices that have the function of collecting animal vocal information and appropriately pre-processing it.

[0194] "Analysis means" refers to a system or process that has the function of analyzing acquired video and audio data to estimate the emotions and needs of animals.

[0195] A "conversion means" is a device or process for converting human speech into signals or sounds that animals can understand.

[0196] "Notification means" refers to the mechanisms and technologies used to communicate analyzed data to the user.

[0197] "Adjustment means" refers to a device or method that analyzes the user's emotional state and adjusts the tone of signals or voices sent to the animal based on the results.

[0198] "Learning methods" refer to techniques and methods for learning individual differences in animal behavior and vocalizations and improving the accuracy of analysis.

[0199] This invention consists of a system combining a terminal and a server to enable smooth communication between animals and users.

[0200] The terminal uses cameras and microphones placed near the animals to collect video and audio information about the animals in real time. This information is pre-processed on-site to remove noise before being transmitted to the server.

[0201] The server is equipped with analytical means to analyze received video and audio data and estimate emotions and requests based on animal behavior and vocalizations. The analytical means uses Python®-based OpenCV to analyze video data and Librosa to extract features from audio data. This information is then used with deep learning techniques utilizing TensorFlow and Keras to estimate animal emotions and requests with high accuracy.

[0202] The server also analyzes the user's emotional state using their voice and video data. To this end, an emotion engine analyzes the user's voice tone and facial expressions to determine their emotional state. For example, if the user is excited, this emotional data is used to adjust the tone of the signals transmitted to the animal. This adjustment mechanism can convert the user's commands into a gentler tone so that they are not burdensome to the animal.

[0203] For example, when a user gives the command "sit," the server will change its tone to a gentler one if it detects impatience in the user's voice. This allows the animal to understand the command without causing unnecessary stress.

[0204] Furthermore, this system can also support user interaction by generating an example prompt using the generated AI model, such as "Please tell me how to tell my pet to sit in a way that doesn't cause it stress."

[0205] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0206] Step 1:

[0207] The device uses a camera and microphone to acquire real-time video and audio information of animals. The input is a live feed of the animals, and the output is pre-processed digital data. This data is subjected to noise reduction filtering in real time.

[0208] Step 2:

[0209] The terminal transfers pre-processed video and audio information to the server. The input is denoised data, and the output is the data transferred to the server. This data includes timestamps and metadata.

[0210] Step 3:

[0211] The server analyzes the transferred video data using OpenCV. The input is video data, and the output is the result of extracting behavioral patterns. In this step, the animal's movements are identified as feature points.

[0212] Step 4:

[0213] The server uses Librosa to analyze audio data. The input is audio data, and the output is the result of extracting audio features. The audio waveform is converted into a frequency spectrum, and specific audio patterns are identified.

[0214] Step 5:

[0215] The server uses TensorFlow or Keras to estimate animal emotions based on data extracted from video and audio. The input is video and audio feature data, and the output is the estimated emotion and request results.

[0216] Step 6:

[0217] The server analyzes the user's voice and video data and uses an emotion engine to determine the user's emotional state. The input is the user's voice and video, and the output is data on the user's emotional state.

[0218] Step 7:

[0219] The server adjusts signals or sounds to the animal based on the user's emotional state. The inputs are the animal's emotional state and the user's emotional state, and the output is the adjusted communication signal.

[0220] Step 8:

[0221] The server reconstructs the analyzed animal's emotions and user instructions, and transmits them from the terminal in an animal-friendly tone. The input is a modified communication signal, and the output is a signal that the animal can easily understand.

[0222] Step 9:

[0223] The server uses a generative AI model to generate prompt messages for the user. The input is the animal's state and the user's instructions, and the output is a prompt message that assists the user's actions.

[0224] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0225] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0226] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0227] [Second Embodiment]

[0228] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0229] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0230] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0231] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0232] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0233] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0234] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0235] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0236] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0237] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0238] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0239] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0240] This invention is a system that analyzes animal behavior and vocalizations to estimate the emotions and needs of animals in real time. This system consists of a user terminal and a server.

[0241] The user places a device near the animal and acquires video and audio data through the camera and microphone. The device temporarily stores the acquired data, performs noise reduction and resolution adjustments, and prepares it for data transfer to the server.

[0242] The server uses video recognition AI and voice recognition AI to analyze data received from the terminal. Based on the video data, it analyzes the animal's movements and posture, and from the voice data, it detects the characteristics of the animal's vocalizations. Based on this information, the server estimates the animal's emotions and needs.

[0243] The estimated emotions and requests are sent from the server to the terminal and notified to the user. For example, if a dog is jumping while wagging its tail, the server will determine that the dog is "happy" and convey that information to the user through the terminal.

[0244] Furthermore, when a user enters a command into the terminal, the server converts that command into a signal or sound that the animal can understand. For example, if the command "stay" is entered, the server converts it into a sound signal of a specific frequency and transmits it to the animal.

[0245] This system also has a function to improve analysis accuracy by learning individual differences in animal behavior and vocalizations through learning methods. This allows it to learn the tendencies and habits of specific animals, enabling more accurate estimation of their emotions and needs.

[0246] The following describes the processing flow.

[0247] Step 1:

[0248] The device uses cameras and microphones placed near the animals to continuously acquire video and audio data. This prepares it to capture the animals' behavior and vocalizations in real time.

[0249] Step 2:

[0250] The terminal performs noise reduction and normalization processing on the acquired video and audio data to improve data quality. This processing is an important pre-processing step for improving analysis accuracy.

[0251] Step 3:

[0252] The terminal sends the pre-processed data to the server. This data transfer is an essential step for starting the analysis process.

[0253] Step 4:

[0254] The server uses video recognition AI to analyze the received video data and detect the animal's posture and movements. This makes it possible to objectively evaluate the animal's behavior.

[0255] Step 5:

[0256] The server runs a speech recognition AI to analyze the received audio data. The analysis identifies the frequency and patterns of the animal's calls and infers its emotions and needs.

[0257] Step 6:

[0258] The server integrates the video and audio analysis results and uses an AI model to estimate the animal's emotions and needs. The estimation results provide an overall assessment of the animal's condition.

[0259] Step 7:

[0260] The server sends estimated emotions and requests to the terminal and notifies the user. This allows the user to receive immediate feedback about the animal's condition.

[0261] Step 8:

[0262] When a user wants to give instructions to an animal, they input those instructions into a device. These instructions are then used for communication with the animal.

[0263] Step 9:

[0264] The server receives instructions sent by the user and converts them into signals or sounds that the animal can understand. This conversion process ensures that the user's intentions are accurately conveyed to the animal.

[0265] Step 10:

[0266] The server accumulates historical data to learn about individual differences among animals and improves the accuracy of its analysis through learning mechanisms. This continuous learning enables more accurate and reliable analysis.

[0267] (Example 1)

[0268] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0269] Accurately understanding animals' emotions and needs in real time has traditionally been difficult. This has led to problems such as inflicting unnecessary stress on animals or failing to provide appropriate care. Furthermore, there has been a lack of appropriate analytical methods that take individual animal differences into account, creating a need for more precise decision-making.

[0270] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0271] In this invention, the server includes an image acquisition means for acquiring and pre-processing visual information of an animal, an acoustic acquisition means for acquiring and pre-processing auditory information of an animal, and an information processing means for analyzing the information acquired by the image acquisition means and the acoustic acquisition means to estimate emotions and requests based on the animal's movements and vocalizations. This enables real-time and highly accurate analysis of emotions and requests based on the animal's behavior and vocalizations.

[0272] "Visual information acquisition means" refers to means for acquiring video data to record the movements and postures of animals and for pre-processing that data.

[0273] A "sound information acquisition means" is a means for acquiring data related to animal sounds and other sounds, and for preprocessing that data.

[0274] "Information processing means" refers to means for analyzing acquired visual and auditory information to estimate the emotions and needs of animals.

[0275] A "conversion means" is a means of converting human instructions into signals or sounds that can be recognized by animals.

[0276] A "notification means" is a means of notifying the user of the results analyzed by an information processing means.

[0277] A "generative AI model" is an artificial intelligence model used to analyze patterns of animal behavior and vocalizations and to estimate their emotions and needs.

[0278] The "learning function" is a function that learns about individual differences in animals and improves the accuracy of analysis.

[0279] This invention is a system that analyzes animal behavior and vocalizations in real time and estimates their emotions and needs. The system consists of a user, a terminal, and a server.

[0280] The user places the device close to the animal. The device is equipped with a camera and microphone, and acquires video and audio data of the animal in real time. This data is temporarily stored in the device and pre-processed, such as noise reduction and resolution adjustment.

[0281] The terminal digitizes the pre-processed data and sends it to the server. Examples of terminals include typical smartphones and dedicated IoT devices.

[0282] The server is deployed on the cloud and runs high-performance video recognition AI and speech recognition AI. Generally, software libraries such as OpenCV and TensorFlow are used for video recognition. For speech recognition, established speech analysis technologies such as the Google Speech-to-Text API are used. The server analyzes the received data and comprehensively estimates emotions and requests based on the movements, postures, and voice patterns of animals.

[0283] In this analysis process, a generative AI model is used to learn the animal behavior data accumulated in the past, realizing high-precision estimation considering individual differences. The results of the analysis are immediately sent to the terminal to inform the user.

[0284] Furthermore, the user can give instructions to the animal through the terminal. When the user inputs a command into the terminal, the server converts the instruction into a signal or voice that the animal can recognize and provides it. In this conversion, frequencies, volumes, etc. specialized for each animal are adjusted.

[0285] As a specific example, consider a situation where the user is in a park with a dog. The terminal captures the dog wagging its tail and acquires the corresponding voice. The server analyzes these and estimates that the dog is "happy." By notifying this information to the user, more appropriate responses become possible. Also, as an example of a prompt sentence for the generative AI model, a sentence such as "Detect the time the dog has been barking for a long time and estimate the reason" can be cited.

[0286] The flow of the specific process in Example 1 will be described using FIG. 11.

[0287] Step 1:

[0288] The user places a terminal near the animal and uses the camera and microphone to acquire visual and auditory information. The input is real-time video and audio data of the animal. The terminal temporarily stores this data and performs noise reduction and resolution adjustments. The output is pre-processed digital data. This step improves the quality of the acquired data and optimizes it for analysis on the server.

[0289] Step 2:

[0290] The terminal packets pre-processed data and sends it to the server. The input is pre-processed video and audio data. Data format conversion is performed to efficiently send the data to the server. As output, transmittable data packets are generated. This step includes implementations to ensure secure and rapid data transfer.

[0291] Step 3:

[0292] The server receives data packets sent from the terminal. The input is the transmitted data packets. After verifying the reliability and integrity of the data, the server starts the analysis. The output is data ready for analysis. Receiving and verifying the data is a crucial operation for fulfilling the prerequisites for data analysis.

[0293] Step 4:

[0294] The server performs analysis using video recognition AI and speech recognition AI based on the received data. The input is pre-processed data deployed on the server. From the video data, the movements and postures of animals are analyzed, and from the audio data, patterns of vocalizations are identified. Emotions and requests are estimated using a generative AI model. As output, data of estimated emotions and requests is obtained. The generative AI model learns individual differences, enabling highly accurate estimations.

[0295] Step 5:

[0296] The server converts the estimated results into a data format and sends it to the terminal. The input is the estimated result data. The data is converted into an appropriate format for communication with the user. The output is data that can be communicated to the user. This step includes actions to organize the information the user receives in an easy-to-understand manner.

[0297] Step 6:

[0298] The terminal analyzes data received from the server and notifies the user. The input is estimated data sent from the server. The information is displayed in an appropriate format, and alerts are given via voice or other means. The output is the information delivered to the user. In this step, feedback is provided through an interface that the user can instantly understand.

[0299] Step 7:

[0300] Users can issue commands to animals by entering specific commands into a terminal. The input consists of commands from the user. These commands are sent to a server and converted into signals or sounds that animals can understand. The converted instructions are then transmitted to the animals as output. This process is designed to facilitate communication between animals and humans.

[0301] (Application Example 1)

[0302] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0303] In modern times, a challenge in communication between animals and humans is the difficulty in accurately understanding an animal's emotions and needs and responding appropriately based on that understanding. There is also a need to maintain smooth communication with pets even when away from home or during absences. Furthermore, there is the problem of difficulty in understanding an animal's condition in real time and responding appropriately.

[0304] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0305] In this invention, the server includes means for acquiring video information of an animal and performing preprocessing, means for acquiring voice information of the animal and performing preprocessing, and means for analyzing the acquired data and estimating emotions and demands based on the behavior and voice of the animal. Thereby, it becomes possible to accurately grasp the emotions and demands of the animal in real time and transmit instructions to the animal via a smart device even when away from home.

[0306] The "video acquisition means" is a device having a function of acquiring video information of an animal using a sensor or a camera and performing preprocessing as necessary.

[0307] The "voice acquisition means" is a device having a function of acquiring voice information of an animal using a microphone or a voice sensor and performing preprocessing as necessary.

[0308] The "analysis means" is a system including technologies for analyzing the acquired video information and voice information and estimating the emotions and demands of the animal based on its behavior and voice.

[0309] The "conversion means" is a device having a function of converting human instructions into signals or voices that can be understood by the animal.

[0310] The "notification means" is a device having a function of notifying the user of the data analyzed by the analysis means so that the state of the animal can be grasped immediately.

[0311] The "control means" is a technology that plays a role of connecting to a mechanical device for the animal and performing control necessary to achieve smooth communication.

[0312] The system for realizing this invention analyzes the behavior and voice of an animal in real time and notifies the user of its emotions and demands. This system is mainly composed of a terminal and a server.

[0313] The terminal is placed near the animal and uses a camera and microphone to acquire video and audio information about the animal. The terminal temporarily stores this information and performs preprocessing such as noise reduction and resolution adjustment. After that, the terminal prepares to transfer the preprocessed data to the server.

[0314] The server receives the transferred data as described above and analyzes it using image recognition AI and speech recognition AI. In this process, it uses AI frameworks such as TensorFlow and PyTorch to analyze the behavioral and vocal characteristics of animals and estimate their emotions and requests.

[0315] The estimated data is communicated to the user via a terminal. This allows the user to monitor the animal's condition in real time and take necessary actions. Furthermore, instructions entered by the user into the terminal are converted by a server into signals or sounds that the animal can understand. These converted instructions are transmitted to the animal's mechanical devices via control means, enabling smooth communication.

[0316] As a concrete example, one day while a user is out, they receive a notification that their pet dog is in an excited state. Based on this information, the user inputs the command "sit," and this command is transmitted to the dog via the device, reducing the dog's stress.

[0317] An example of a prompt might be: "Develop an app that estimates a pet's emotions from its behavior and sounds and sends a notification to the owner. For example, if the dog wags its tail or jumps, the app would notify the owner that the pet is 'happy'."

[0318] This invention aims to improve analysis accuracy by continuously learning individual differences in animal behavior and vocalizations, thereby providing more appropriate and accurate estimation results.

[0319] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0320] Step 1:

[0321] The device uses a camera and microphone to acquire video and audio information about animals. The input is camera video and microphone audio, and the output is raw video and audio data. Specifically, it works by pointing the camera at the animal and operating the microphone near the animal to acquire information.

[0322] Step 2:

[0323] The video and audio data acquired by the device are preprocessed by noise reduction and resolution adjustment. The input is the raw data acquired in step 1, and the output is the processed video and audio data. Specifically, noise is removed using filtering technology, and adjustments are made if the image quality is not clear.

[0324] Step 3:

[0325] The terminal sends the pre-processed data to the server. The input is the processed data generated in step 2, and the output is the data transferred to the server. Specifically, the data is uploaded to the server via the network using a communication protocol.

[0326] Step 4:

[0327] The server analyzes the received data using video recognition AI and speech recognition AI. The input is processed data sent from the terminal, and the output is an estimation of emotions and requests based on the animal's behavioral and vocal characteristics. Specifically, a generative AI model is used to analyze data patterns and obtain estimation results.

[0328] Step 5:

[0329] The server notifies the terminal of the estimation results, and the information is conveyed to the user. The input is the estimation results generated by the server, and the output is the information presented to the user. Specifically, information is sent to the user's terminal using push notifications, and the user is notified visually or audibly.

[0330] Step 6:

[0331] The user inputs instructions to the terminal. The input consists of commands performed by the user, and the output is instruction data sent to the server. Specifically, instructions such as "sit" or "stay" are input through screen operations.

[0332] Step 7:

[0333] The server converts user instructions into signals or sounds that animals can understand and transmits them to the animal's mechanical device via a terminal. The input is user instruction data, and the output is the converted signals or sounds. Specifically, a generative AI model is used to create signals, which are then transmitted to the animal using control devices.

[0334] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0335] This invention combines a system that analyzes animal behavior and vocalizations with an emotion engine that recognizes user emotions, in order to establish communication between animals and humans. This system consists of a terminal operated by the user and a server that performs data analysis.

[0336] The terminal uses cameras and microphones placed near the animals to collect video and audio data of the animals in real time. The terminal preprocesses the data, removes noise, and then transmits it to a server. This data is used to analyze the animals' behavior and vocal patterns.

[0337] The server analyzes the received video and audio data to estimate the animal's emotions and needs. Based on the information obtained through the analysis, it determines whether the animal is happy, hungry, or anxious.

[0338] Furthermore, the emotion engine analyzes the user's tone of voice, word choice, and even facial expressions from video data to identify the user's emotions. This allows it to detect the user's emotional state, such as whether they are stressed or calm.

[0339] When a user gives instructions to an animal, they input the content into a terminal. The server takes into account the user's emotional state, as detected by the emotion engine, and converts the instructions into signals or sounds that the animal can understand. This process ensures that, for example, even if the user is excited and gives instructions in a strong tone, the signals are adjusted to a gentle tone so that the animal does not experience unnecessary stress.

[0340] For example, when a user gives the command "sit," if the server senses impatience in the user's voice, it will convert the tone to a calmer one before transmitting it to the animal. In this way, the emotion engine plays a crucial role in facilitating smooth two-way communication.

[0341] Through the above process, smooth communication can be established between animals and users, enabling a rapid response to the animals' needs.

[0342] The following describes the processing flow.

[0343] Step 1:

[0344] The device uses a camera and microphone to acquire video and audio data of animals. This process records the animals' movements and sounds in real time.

[0345] Step 2:

[0346] The terminal performs preprocessing on the acquired data, including noise reduction and normalization. This improves the quality of the data, which enhances the accuracy of the analysis.

[0347] Step 3:

[0348] The terminal sends the pre-processed video and audio data to the server. At this stage, the data is ready for analysis.

[0349] Step 4:

[0350] The server uses video recognition AI to analyze animal behavior and detect posture and movements. This information forms the basis for determining the animal's emotions and needs.

[0351] Step 5:

[0352] The server uses speech recognition AI to analyze audio data and identify the frequency, tone, and pattern of animal calls. This allows it to estimate the animal's needs and mood.

[0353] Step 6:

[0354] The server utilizes an emotion engine to recognize the user's emotions. It analyzes the user's tone of voice, chosen words, and facial expressions to estimate their emotional state.

[0355] Step 7:

[0356] The server integrates the results of animal behavior and voice analysis with the user's emotional state, and converts the commands to the animal into gently adjusted signals and voices.

[0357] Step 8:

[0358] The server transmits the generated signal or sound to the terminal, which then relays it to the animal. In this process, the coordinated communication is delivered to the animal.

[0359] Step 9:

[0360] The user receives feedback from the system, observes and evaluates the animal's response, and adjusts instructions as needed to determine the next action.

[0361] In this way, two-way communication between animals and users is realized, enabling appropriate responses that meet the animals' needs and emotions.

[0362] (Example 2)

[0363] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0364] Facilitating smooth communication between animals and humans is extremely difficult. Conventional systems not only struggle to accurately understand animal emotions and needs, but also transmit information unilaterally without considering human emotional states, often leading to unsuccessful communication. A solution to this problem and a means of achieving meaningful communication between animals and humans are needed.

[0365] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0366] In this invention, the server includes data collection means for collecting and preprocessing animal behavior and voice data, analysis means for analyzing emotions and requests based on animal behavior and voice using a generative AI model, and user emotion analysis means for analyzing the tone of the user's voice and facial expressions and identifying their emotional state. This facilitates two-way communication between animals and humans, enabling a more accurate understanding of each other's emotions and requests.

[0367] "Data collection means" refers to a device or system that has the function of collecting and pre-processing animal behavior and sound data in real time.

[0368] "Data transfer means" refers to a device or system having communication protocols and functions for securely and quickly transferring pre-processed data to a server.

[0369] "Generative AI models" refer to artificial intelligence technologies used to analyze animal behavior and vocal data to estimate the emotions and needs of animals.

[0370] "Analysis means" refers to software or devices that utilize generative AI models to analyze emotions and requests based on animal behavior and vocalizations.

[0371] "User emotion analysis means" refers to software or a device that has the function of analyzing the tone of a user's voice and facial expressions to identify their emotional state.

[0372] "Conversion means" refers to software or a device that has the function of converting instructions given by a user into signals or sounds that an animal can understand.

[0373] "Notification means" refers to a device or system that has the function of transmitting information about the animal's condition obtained by the analysis means to the user.

[0374] A "learning tool" is software or a system that has the function of learning individual differences in animal behavior and vocalizations and improving the accuracy of analysis.

[0375] A "feedback improvement method" is a system that has the function of adjusting the generated AI model based on user feedback to improve its accuracy and effectiveness.

[0376] This invention is a system for facilitating communication between animals and humans. The system consists of a user-operated terminal and a server that performs data analysis. The terminal uses a camera and microphone placed near the animal to collect video and audio data of the animal in real time. After preprocessing the collected data, such as noise filtering, it is transferred to the server using a secure protocol.

[0377] The server analyzes the received video and audio data. Specifically, it uses a generative AI model to infer emotions and needs from animal behavior and vocalizations. This analysis determines whether the animal is happy, hungry, or anxious. Furthermore, the server uses an emotion engine to identify the user's emotional state based on the tone of voice and facial expressions provided by the user.

[0378] For example, consider a scenario where a user gives the command "sit." When the user inputs this command into the device, the server takes into account the user's emotional state, as determined by the emotion engine, and converts the command into signals and sounds that the animal can understand. Even if the user is excited and gives the command in a strong tone, the server converts the command into a calmer tone before conveying it to the animal. In this way, the system ensures smooth communication between the user and the animal.

[0379] A concrete example of a prompt message is, "When the user commands the dog to sit, try to communicate in a calm tone." Based on this prompt message, the server performs the necessary processing, and the tone and signals transmitted to the animal are adjusted accordingly.

[0380] This system allows for accurate understanding of animals' emotions and needs, and enables information transmission that appropriately considers the user's emotional state, thereby making communication with animals more meaningful.

[0381] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0382] Step 1:

[0383] The device uses a camera and microphone placed near the animal to collect video and audio data of the animal in real time. It takes in video data from the camera and audio data from the microphone as input, and improves the quality of the input data by applying noise filtering. The data, after this preprocessing is complete, is output in a format suitable for subsequent processing steps.

[0384] Step 2:

[0385] The terminal transfers pre-processed video and audio data to the server. This step uses a data transfer protocol (e.g., HTTP / 2 or WebSocket) to ensure security and speed. The input is filtered data, and the output is data transmitted via encrypted communication.

[0386] Step 3:

[0387] The server analyzes the received video and audio data. It utilizes a generative AI model to estimate emotions and needs from animal behavior and sounds. In this process, video and audio data are received as input, the model performs data analysis, and an estimated result is output, such as whether the animal is happy, hungry, or anxious.

[0388] Step 4:

[0389] The server identifies the user's emotional state from their voice tone and facial expressions. It receives audio and video data entered by the user into the terminal and analyzes it using an emotion engine. The input is the user's voice and video, and the output is the analysis result indicating the user's emotional state.

[0390] Step 5:

[0391] The server converts user instructions into signals or sounds that animals can understand. A generative AI model considers the user's emotional state and outputs instructions in a controlled tone that is not stressful for the animal. Input is the user's instructions and the emotional analysis results, while output is signals or sounds for the animal.

[0392] Step 6:

[0393] The user observes the animal's reactions and inputs the results as feedback into the device. This feedback information is sent to a server and used to improve the generated AI model. The input is feedback data based on the animal's reactions, while the output is data used to adjust and improve the AI ​​model.

[0394] (Application Example 2)

[0395] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0396] Systems for facilitating smooth communication between animals and humans need to accurately understand the animal's emotions and needs, and ensure that human instructions are properly conveyed to the animal. However, existing technologies struggle to accurately analyze animal behavior and vocalizations, and rarely adjust to the human emotional state. This can lead to miscommunication between animals and humans.

[0397] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0398] In this invention, the server includes a video acquisition means for acquiring and pre-processing video information of animals, a voice acquisition means for acquiring and pre-processing audio information of animals, an analysis means for analyzing the acquired data and estimating the animal's emotions and requests, a conversion means for converting human speech into a form that animals can understand, and an adjustment means for analyzing the user's emotional state and adjusting signals or voice tone. This improves the accuracy of communication between animals and humans, enabling both parties to interact without stress.

[0399] "Image acquisition means" refers to equipment or devices that have the function of collecting image information of animals and performing appropriate preprocessing.

[0400] "Voice acquisition means" refers to equipment or devices that have the function of collecting animal vocal information and appropriately pre-processing it.

[0401] "Analysis means" refers to a system or process that has the function of analyzing acquired video and audio data to estimate the emotions and needs of animals.

[0402] A "conversion means" is a device or process for converting human speech into signals or sounds that animals can understand.

[0403] "Notification means" refers to the mechanisms and technologies used to communicate analyzed data to the user.

[0404] "Adjustment means" refers to a device or method that analyzes the user's emotional state and adjusts the tone of signals or voices sent to the animal based on the results.

[0405] "Learning methods" refer to techniques and methods for learning individual differences in animal behavior and vocalizations and improving the accuracy of analysis.

[0406] This invention consists of a system combining a terminal and a server to enable smooth communication between animals and users.

[0407] The terminal uses cameras and microphones placed near the animals to collect video and audio information about the animals in real time. This information is pre-processed on-site to remove noise before being transmitted to the server.

[0408] The server is equipped with analytical means to analyze received video and audio data and estimate emotions and requests based on animal behavior and vocalizations. The analytical means uses Python-based OpenCV to analyze video data and Librosa to extract features from audio data. This information is then used with deep learning techniques utilizing TensorFlow and Keras to estimate animal emotions and requests with high accuracy.

[0409] The server also analyzes the user's emotional state using their voice and video data. To this end, an emotion engine analyzes the user's voice tone and facial expressions to determine their emotional state. For example, if the user is excited, this emotional data is used to adjust the tone of the signals transmitted to the animal. This adjustment mechanism can convert the user's commands into a gentler tone so that they are not burdensome to the animal.

[0410] For example, when a user gives the command "sit," the server will change its tone to a gentler one if it detects impatience in the user's voice. This allows the animal to understand the command without causing unnecessary stress.

[0411] Furthermore, this system can also support user interaction by generating an example prompt using the generated AI model, such as "Please tell me how to tell my pet to sit in a way that doesn't cause it stress."

[0412] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0413] Step 1:

[0414] The device uses a camera and microphone to acquire real-time video and audio information of animals. The input is a live feed of the animals, and the output is pre-processed digital data. This data is subjected to noise reduction filtering in real time.

[0415] Step 2:

[0416] The terminal transfers pre-processed video and audio information to the server. The input is denoised data, and the output is the data transferred to the server. This data includes timestamps and metadata.

[0417] Step 3:

[0418] The server analyzes the transferred video data using OpenCV. The input is video data, and the output is the result of extracting behavioral patterns. In this step, the animal's movements are identified as feature points.

[0419] Step 4:

[0420] The server uses Librosa to analyze audio data. The input is audio data, and the output is the result of extracting audio features. The audio waveform is converted into a frequency spectrum, and specific audio patterns are identified.

[0421] Step 5:

[0422] The server uses TensorFlow or Keras to estimate animal emotions based on data extracted from video and audio. The input is video and audio feature data, and the output is the estimated emotion and request results.

[0423] Step 6:

[0424] The server analyzes the user's voice and video data and uses an emotion engine to determine the user's emotional state. The input is the user's voice and video, and the output is data on the user's emotional state.

[0425] Step 7:

[0426] The server adjusts signals or sounds to the animal based on the user's emotional state. The inputs are the animal's emotional state and the user's emotional state, and the output is the adjusted communication signal.

[0427] Step 8:

[0428] The server reconstructs the analyzed animal's emotions and user instructions, and transmits them from the terminal in an animal-friendly tone. The input is a modified communication signal, and the output is a signal that the animal can easily understand.

[0429] Step 9:

[0430] The server uses a generative AI model to generate prompt messages for the user. The input is the animal's state and the user's instructions, and the output is a prompt message that assists the user's actions.

[0431] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0432] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0433] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0434] [Third Embodiment]

[0435] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0436] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0437] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0438] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0439] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0440] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0441] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0442] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0443] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0444] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0445] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0446] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0447] This invention is a system that analyzes animal behavior and vocalizations to estimate the emotions and needs of animals in real time. This system consists of a user terminal and a server.

[0448] The user places a device near the animal and acquires video and audio data through the camera and microphone. The device temporarily stores the acquired data, performs noise reduction and resolution adjustments, and prepares it for data transfer to the server.

[0449] The server uses video recognition AI and voice recognition AI to analyze data received from the terminal. Based on the video data, it analyzes the animal's movements and posture, and from the voice data, it detects the characteristics of the animal's vocalizations. Based on this information, the server estimates the animal's emotions and needs.

[0450] The estimated emotions and requests are sent from the server to the terminal and notified to the user. For example, if a dog is jumping while wagging its tail, the server will determine that the dog is "happy" and convey that information to the user through the terminal.

[0451] Furthermore, when a user enters a command into the terminal, the server converts that command into a signal or sound that the animal can understand. For example, if the command "stay" is entered, the server converts it into a sound signal of a specific frequency and transmits it to the animal.

[0452] This system also has a function to improve analysis accuracy by learning individual differences in animal behavior and vocalizations through learning methods. This allows it to learn the tendencies and habits of specific animals, enabling more accurate estimation of their emotions and needs.

[0453] The following describes the processing flow.

[0454] Step 1:

[0455] The device uses cameras and microphones placed near the animals to continuously acquire video and audio data. This prepares it to capture the animals' behavior and vocalizations in real time.

[0456] Step 2:

[0457] The terminal performs noise reduction and normalization processing on the acquired video and audio data to improve data quality. This processing is an important pre-processing step for improving analysis accuracy.

[0458] Step 3:

[0459] The terminal sends the pre-processed data to the server. This data transfer is an essential step for starting the analysis process.

[0460] Step 4:

[0461] The server uses video recognition AI to analyze the received video data and detect the animal's posture and movements. This makes it possible to objectively evaluate the animal's behavior.

[0462] Step 5:

[0463] The server runs a speech recognition AI to analyze the received audio data. The analysis identifies the frequency and patterns of the animal's calls and infers its emotions and needs.

[0464] Step 6:

[0465] The server integrates the video and audio analysis results and uses an AI model to estimate the animal's emotions and needs. The estimation results provide an overall assessment of the animal's condition.

[0466] Step 7:

[0467] The server sends estimated emotions and requests to the terminal and notifies the user. This allows the user to receive immediate feedback about the animal's condition.

[0468] Step 8:

[0469] When a user wants to give instructions to an animal, they input those instructions into a device. These instructions are then used for communication with the animal.

[0470] Step 9:

[0471] The server receives instructions sent by the user and converts them into signals or sounds that the animal can understand. This conversion process ensures that the user's intentions are accurately conveyed to the animal.

[0472] Step 10:

[0473] The server accumulates historical data to learn about individual differences among animals and improves the accuracy of its analysis through learning mechanisms. This continuous learning enables more accurate and reliable analysis.

[0474] (Example 1)

[0475] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0476] Accurately understanding animals' emotions and needs in real time has traditionally been difficult. This has led to problems such as inflicting unnecessary stress on animals or failing to provide appropriate care. Furthermore, there has been a lack of appropriate analytical methods that take individual animal differences into account, creating a need for more precise decision-making.

[0477] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0478] In this invention, the server includes an image acquisition means for acquiring and pre-processing visual information of an animal, an acoustic acquisition means for acquiring and pre-processing auditory information of an animal, and an information processing means for analyzing the information acquired by the image acquisition means and the acoustic acquisition means to estimate emotions and requests based on the animal's movements and vocalizations. This enables real-time and highly accurate analysis of emotions and requests based on the animal's behavior and vocalizations.

[0479] "Visual information acquisition means" refers to means for acquiring video data to record the movements and postures of animals and for pre-processing that data.

[0480] A "sound information acquisition means" is a means for acquiring data related to animal sounds and other sounds, and for preprocessing that data.

[0481] "Information processing means" refers to means for analyzing acquired visual and auditory information to estimate the emotions and needs of animals.

[0482] A "conversion means" is a means of converting human instructions into signals or sounds that can be recognized by animals.

[0483] A "notification means" is a means of notifying the user of the results analyzed by an information processing means.

[0484] A "generative AI model" is an artificial intelligence model used to analyze patterns of animal behavior and vocalizations and to estimate their emotions and needs.

[0485] The "learning function" is a function that learns about individual differences in animals and improves the accuracy of analysis.

[0486] This invention is a system that analyzes animal behavior and vocalizations in real time and estimates their emotions and needs. The system consists of a user, a terminal, and a server.

[0487] The user places the device close to the animal. The device is equipped with a camera and microphone, and acquires video and audio data of the animal in real time. This data is temporarily stored in the device and pre-processed, such as noise reduction and resolution adjustment.

[0488] The terminal digitizes the pre-processed data and sends it to the server. Examples of terminals include typical smartphones and dedicated IoT devices.

[0489] The servers are located in the cloud and run high-performance image recognition and speech recognition AI. For image recognition, software libraries such as OpenCV and TensorFlow are commonly used. For speech recognition, established speech analysis technologies such as the Google Speech-to-Text API are employed. The servers analyze the received data and comprehensively estimate emotions and requests based on animal movements, postures, and vocal patterns.

[0490] This analysis process utilizes a generative AI model that learns from previously accumulated animal behavior data to achieve highly accurate estimations that take individual differences into account. The analysis results are immediately transmitted to the terminal and notified to the user.

[0491] Furthermore, users can issue commands to animals through a terminal. When a user enters a command into the terminal, the server converts that command into signals or sounds that the animals can recognize and provides them. This conversion adjusts the frequency, volume, and other parameters to be specific to each animal.

[0492] As a concrete example, consider a scenario where a user is spending time with a dog in a park. The device captures the dog wagging its tail and obtains the corresponding audio. The server analyzes this and estimates that the dog is "happy." This information is then communicated to the user, enabling a more appropriate response. Another example of a prompt for the generative AI model is a sentence like, "Detect how long the dog is barking and estimate the reason."

[0493] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0494] Step 1:

[0495] The user places a terminal near the animal and uses the camera and microphone to acquire visual and auditory information. The input is real-time video and audio data of the animal. The terminal temporarily stores this data and performs noise reduction and resolution adjustments. The output is pre-processed digital data. This step improves the quality of the acquired data and optimizes it for analysis on the server.

[0496] Step 2:

[0497] The terminal packets pre-processed data and sends it to the server. The input is pre-processed video and audio data. Data format conversion is performed to efficiently send the data to the server. As output, transmittable data packets are generated. This step includes implementations to ensure secure and rapid data transfer.

[0498] Step 3:

[0499] The server receives data packets sent from the terminal. The input is the transmitted data packets. After verifying the reliability and integrity of the data, the server starts the analysis. The output is data ready for analysis. Receiving and verifying the data is a crucial operation for fulfilling the prerequisites for data analysis.

[0500] Step 4:

[0501] The server performs analysis using video recognition AI and speech recognition AI based on the received data. The input is pre-processed data deployed on the server. From the video data, the movements and postures of animals are analyzed, and from the audio data, patterns of vocalizations are identified. Emotions and requests are estimated using a generative AI model. As output, data of estimated emotions and requests is obtained. The generative AI model learns individual differences, enabling highly accurate estimations.

[0502] Step 5:

[0503] The server converts the estimated results into a data format and sends it to the terminal. The input is the estimated result data. The data is converted into an appropriate format for communication with the user. The output is data that can be communicated to the user. This step includes actions to organize the information the user receives in an easy-to-understand manner.

[0504] Step 6:

[0505] The terminal analyzes data received from the server and notifies the user. The input is estimated data sent from the server. The information is displayed in an appropriate format, and alerts are given via voice or other means. The output is the information delivered to the user. In this step, feedback is provided through an interface that the user can instantly understand.

[0506] Step 7:

[0507] Users can issue commands to animals by entering specific commands into a terminal. The input consists of commands from the user. These commands are sent to a server and converted into signals or sounds that animals can understand. The converted instructions are then transmitted to the animals as output. This process is designed to facilitate communication between animals and humans.

[0508] (Application Example 1)

[0509] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0510] In modern times, a challenge in communication between animals and humans is the difficulty in accurately understanding an animal's emotions and needs and responding appropriately based on that understanding. There is also a need to maintain smooth communication with pets even when away from home or during absences. Furthermore, there is the problem of difficulty in understanding an animal's condition in real time and responding appropriately.

[0511] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0512] In this invention, the server includes means for acquiring and pre-processing video information of animals, means for acquiring and pre-processing audio information of animals, and means for analyzing the acquired data and estimating emotions and requests based on the animal's behavior and vocalizations. This makes it possible to accurately grasp the animal's emotions and requests in real time and to transmit instructions to the animal via a smart device even when away from home.

[0513] "Image acquisition means" refers to a device that has the function of acquiring image information of animals using sensors or cameras and performing preprocessing as necessary.

[0514] A "sound acquisition means" is a device that has the function of acquiring animal sound information using a microphone or sound sensor and performing preprocessing as necessary.

[0515] "Analysis means" refers to a system that includes technology for analyzing acquired video and audio information and estimating the emotions and needs of animals based on their behavior and vocalizations.

[0516] A "conversion device" is a device that has the function of converting human instructions into signals or sounds that animals can understand.

[0517] A "notification device" is a device that has the function of notifying the user of the data analyzed by the analysis device, enabling them to immediately understand the animal's condition.

[0518] "Control means" refers to technology that connects to mechanical devices for animals and plays a role in providing the necessary control to ensure smooth communication.

[0519] The system for realizing this invention analyzes animal behavior and vocalizations in real time and notifies the user of their emotions and requests. This system mainly consists of terminals and servers.

[0520] The terminal is placed near the animal and uses a camera and microphone to acquire video and audio information about the animal. The terminal temporarily stores this information and performs preprocessing such as noise reduction and resolution adjustment. After that, the terminal prepares to transfer the preprocessed data to the server.

[0521] The server receives the transferred data as described above and analyzes it using image recognition AI and speech recognition AI. In this process, it uses AI frameworks such as TensorFlow and PyTorch to analyze the behavioral and vocal characteristics of animals and estimate their emotions and requests.

[0522] The estimated data is communicated to the user via a terminal. This allows the user to monitor the animal's condition in real time and take necessary actions. Furthermore, instructions entered by the user into the terminal are converted by a server into signals or sounds that the animal can understand. These converted instructions are transmitted to the animal's mechanical devices via control means, enabling smooth communication.

[0523] As a concrete example, one day while a user is out, they receive a notification that their pet dog is in an excited state. Based on this information, the user inputs the command "sit," and this command is transmitted to the dog via the device, reducing the dog's stress.

[0524] An example of a prompt might be: "Develop an app that estimates a pet's emotions from its behavior and sounds and sends a notification to the owner. For example, if the dog wags its tail or jumps, the app would notify the owner that the pet is 'happy'."

[0525] This invention aims to improve analysis accuracy by continuously learning individual differences in animal behavior and vocalizations, thereby providing more appropriate and accurate estimation results.

[0526] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0527] Step 1:

[0528] The device uses a camera and microphone to acquire video and audio information about animals. The input is camera video and microphone audio, and the output is raw video and audio data. Specifically, it works by pointing the camera at the animal and operating the microphone near the animal to acquire information.

[0529] Step 2:

[0530] The video and audio data acquired by the device are preprocessed by noise reduction and resolution adjustment. The input is the raw data acquired in step 1, and the output is the processed video and audio data. Specifically, noise is removed using filtering technology, and adjustments are made if the image quality is not clear.

[0531] Step 3:

[0532] The terminal sends the pre-processed data to the server. The input is the processed data generated in step 2, and the output is the data transferred to the server. Specifically, the data is uploaded to the server via the network using a communication protocol.

[0533] Step 4:

[0534] The server analyzes the received data using video recognition AI and speech recognition AI. The input is processed data sent from the terminal, and the output is an estimation of emotions and requests based on the animal's behavioral and vocal characteristics. Specifically, a generative AI model is used to analyze data patterns and obtain estimation results.

[0535] Step 5:

[0536] The server notifies the terminal of the estimation results, and the information is conveyed to the user. The input is the estimation results generated by the server, and the output is the information presented to the user. Specifically, information is sent to the user's terminal using push notifications, and the user is notified visually or audibly.

[0537] Step 6:

[0538] The user inputs instructions to the terminal. The input consists of commands performed by the user, and the output is instruction data sent to the server. Specifically, instructions such as "sit" or "stay" are input through screen operations.

[0539] Step 7:

[0540] The server converts user instructions into signals or sounds that animals can understand and transmits them to the animal's mechanical device via a terminal. The input is user instruction data, and the output is the converted signals or sounds. Specifically, a generative AI model is used to create signals, which are then transmitted to the animal using control devices.

[0541] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0542] This invention combines a system that analyzes animal behavior and vocalizations with an emotion engine that recognizes user emotions, in order to establish communication between animals and humans. This system consists of a terminal operated by the user and a server that performs data analysis.

[0543] The terminal uses cameras and microphones placed near the animals to collect video and audio data of the animals in real time. The terminal preprocesses the data, removes noise, and then transmits it to a server. This data is used to analyze the animals' behavior and vocal patterns.

[0544] The server analyzes the received video and audio data to estimate the animal's emotions and needs. Based on the information obtained through the analysis, it determines whether the animal is happy, hungry, or anxious.

[0545] Furthermore, the emotion engine analyzes the user's tone of voice, word choice, and even facial expressions from video data to identify the user's emotions. This allows it to detect the user's emotional state, such as whether they are stressed or calm.

[0546] When a user gives instructions to an animal, they input the content into a terminal. The server takes into account the user's emotional state, as detected by the emotion engine, and converts the instructions into signals or sounds that the animal can understand. This process ensures that, for example, even if the user is excited and gives instructions in a strong tone, the signals are adjusted to a gentle tone so that the animal does not experience unnecessary stress.

[0547] For example, when a user gives the command "sit," if the server senses impatience in the user's voice, it will convert the tone to a calmer one before transmitting it to the animal. In this way, the emotion engine plays a crucial role in facilitating smooth two-way communication.

[0548] Through the above process, smooth communication can be established between animals and users, enabling a rapid response to the animals' needs.

[0549] The following describes the processing flow.

[0550] Step 1:

[0551] The device uses a camera and microphone to acquire video and audio data of animals. This process records the animals' movements and sounds in real time.

[0552] Step 2:

[0553] The terminal performs preprocessing on the acquired data, including noise reduction and normalization. This improves the quality of the data, which enhances the accuracy of the analysis.

[0554] Step 3:

[0555] The terminal sends the pre-processed video and audio data to the server. At this stage, the data is ready for analysis.

[0556] Step 4:

[0557] The server uses video recognition AI to analyze animal behavior and detect posture and movements. This information forms the basis for determining the animal's emotions and needs.

[0558] Step 5:

[0559] The server uses speech recognition AI to analyze audio data and identify the frequency, tone, and pattern of animal calls. This allows it to estimate the animal's needs and mood.

[0560] Step 6:

[0561] The server utilizes an emotion engine to recognize the user's emotions. It analyzes the user's tone of voice, chosen words, and facial expressions to estimate their emotional state.

[0562] Step 7:

[0563] The server integrates the results of animal behavior and voice analysis with the user's emotional state, and converts the commands to the animal into gently adjusted signals and voices.

[0564] Step 8:

[0565] The server transmits the generated signal or sound to the terminal, which then relays it to the animal. In this process, the coordinated communication is delivered to the animal.

[0566] Step 9:

[0567] The user receives feedback from the system, observes and evaluates the animal's response, and adjusts instructions as needed to determine the next action.

[0568] In this way, two-way communication between animals and users is realized, enabling appropriate responses that meet the animals' needs and emotions.

[0569] (Example 2)

[0570] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0571] Facilitating smooth communication between animals and humans is extremely difficult. Conventional systems not only struggle to accurately understand animal emotions and needs, but also transmit information unilaterally without considering human emotional states, often leading to unsuccessful communication. A solution to this problem and a means of achieving meaningful communication between animals and humans are needed.

[0572] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0573] In this invention, the server includes data collection means for collecting and preprocessing animal behavior and voice data, analysis means for analyzing emotions and requests based on animal behavior and voice using a generative AI model, and user emotion analysis means for analyzing the tone of the user's voice and facial expressions and identifying their emotional state. This facilitates two-way communication between animals and humans, enabling a more accurate understanding of each other's emotions and requests.

[0574] "Data collection means" refers to a device or system that has the function of collecting and pre-processing animal behavior and sound data in real time.

[0575] "Data transfer means" refers to a device or system having communication protocols and functions for securely and quickly transferring pre-processed data to a server.

[0576] "Generative AI models" refer to artificial intelligence technologies used to analyze animal behavior and vocal data to estimate the emotions and needs of animals.

[0577] "Analysis means" refers to software or devices that utilize generative AI models to analyze emotions and requests based on animal behavior and vocalizations.

[0578] "User emotion analysis means" refers to software or a device that has the function of analyzing the tone of a user's voice and facial expressions to identify their emotional state.

[0579] "Conversion means" refers to software or a device that has the function of converting instructions given by a user into signals or sounds that an animal can understand.

[0580] "Notification means" refers to a device or system that has the function of transmitting information about the animal's condition obtained by the analysis means to the user.

[0581] A "learning tool" is software or a system that has the function of learning individual differences in animal behavior and vocalizations and improving the accuracy of analysis.

[0582] A "feedback improvement method" is a system that has the function of adjusting the generated AI model based on user feedback to improve its accuracy and effectiveness.

[0583] This invention is a system for facilitating communication between animals and humans. The system consists of a user-operated terminal and a server that performs data analysis. The terminal uses a camera and microphone placed near the animal to collect video and audio data of the animal in real time. After preprocessing the collected data, such as noise filtering, it is transferred to the server using a secure protocol.

[0584] The server analyzes the received video and audio data. Specifically, it uses a generative AI model to infer emotions and needs from animal behavior and vocalizations. This analysis determines whether the animal is happy, hungry, or anxious. Furthermore, the server uses an emotion engine to identify the user's emotional state based on the tone of voice and facial expressions provided by the user.

[0585] For example, consider a scenario where a user gives the command "sit." When the user inputs this command into the device, the server takes into account the user's emotional state, as determined by the emotion engine, and converts the command into signals and sounds that the animal can understand. Even if the user is excited and gives the command in a strong tone, the server converts the command into a calmer tone before conveying it to the animal. In this way, the system ensures smooth communication between the user and the animal.

[0586] A concrete example of a prompt message is, "When the user commands the dog to sit, try to communicate in a calm tone." Based on this prompt message, the server performs the necessary processing, and the tone and signals transmitted to the animal are adjusted accordingly.

[0587] This system allows for accurate understanding of animals' emotions and needs, and enables information transmission that appropriately considers the user's emotional state, thereby making communication with animals more meaningful.

[0588] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0589] Step 1:

[0590] The device uses a camera and microphone placed near the animal to collect video and audio data of the animal in real time. It takes in video data from the camera and audio data from the microphone as input, and improves the quality of the input data by applying noise filtering. The data, after this preprocessing is complete, is output in a format suitable for subsequent processing steps.

[0591] Step 2:

[0592] The terminal transfers pre-processed video and audio data to the server. This step uses a data transfer protocol (e.g., HTTP / 2 or WebSocket) to ensure security and speed. The input is filtered data, and the output is data transmitted via encrypted communication.

[0593] Step 3:

[0594] The server analyzes the received video and audio data. It utilizes a generative AI model to estimate emotions and needs from animal behavior and sounds. In this process, video and audio data are received as input, the model performs data analysis, and an estimated result is output, such as whether the animal is happy, hungry, or anxious.

[0595] Step 4:

[0596] The server identifies the user's emotional state from their voice tone and facial expressions. It receives audio and video data entered by the user into the terminal and analyzes it using an emotion engine. The input is the user's voice and video, and the output is the analysis result indicating the user's emotional state.

[0597] Step 5:

[0598] The server converts user instructions into signals or sounds that animals can understand. A generative AI model considers the user's emotional state and outputs instructions in a controlled tone that is not stressful for the animal. Input is the user's instructions and the emotional analysis results, while output is signals or sounds for the animal.

[0599] Step 6:

[0600] The user observes the animal's reactions and inputs the results as feedback into the device. This feedback information is sent to a server and used to improve the generated AI model. The input is feedback data based on the animal's reactions, while the output is data used to adjust and improve the AI ​​model.

[0601] (Application Example 2)

[0602] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0603] Systems for facilitating smooth communication between animals and humans need to accurately understand the animal's emotions and needs, and ensure that human instructions are properly conveyed to the animal. However, existing technologies struggle to accurately analyze animal behavior and vocalizations, and rarely adjust to the human emotional state. This can lead to miscommunication between animals and humans.

[0604] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0605] In this invention, the server includes a video acquisition means for acquiring and pre-processing video information of animals, a voice acquisition means for acquiring and pre-processing audio information of animals, an analysis means for analyzing the acquired data and estimating the animal's emotions and requests, a conversion means for converting human speech into a form that animals can understand, and an adjustment means for analyzing the user's emotional state and adjusting signals or voice tone. This improves the accuracy of communication between animals and humans, enabling both parties to interact without stress.

[0606] "Image acquisition means" refers to equipment or devices that have the function of collecting image information of animals and performing appropriate preprocessing.

[0607] "Voice acquisition means" refers to equipment or devices that have the function of collecting animal vocal information and appropriately pre-processing it.

[0608] "Analysis means" refers to a system or process that has the function of analyzing acquired video and audio data to estimate the emotions and needs of animals.

[0609] A "conversion means" is a device or process for converting human speech into signals or sounds that animals can understand.

[0610] "Notification means" refers to the mechanisms and technologies used to communicate analyzed data to the user.

[0611] "Adjustment means" refers to a device or method that analyzes the user's emotional state and adjusts the tone of signals or voices sent to the animal based on the results.

[0612] "Learning methods" refer to techniques and methods for learning individual differences in animal behavior and vocalizations and improving the accuracy of analysis.

[0613] This invention consists of a system combining a terminal and a server to enable smooth communication between animals and users.

[0614] The terminal uses cameras and microphones placed near the animals to collect video and audio information about the animals in real time. This information is pre-processed on-site to remove noise before being transmitted to the server.

[0615] The server is equipped with analytical means to analyze received video and audio data and estimate emotions and requests based on animal behavior and vocalizations. The analytical means uses Python-based OpenCV to analyze video data and Librosa to extract features from audio data. This information is then used with deep learning techniques utilizing TensorFlow and Keras to estimate animal emotions and requests with high accuracy.

[0616] The server also analyzes the user's emotional state using their voice and video data. To this end, an emotion engine analyzes the user's voice tone and facial expressions to determine their emotional state. For example, if the user is excited, this emotional data is used to adjust the tone of the signals transmitted to the animal. This adjustment mechanism can convert the user's commands into a gentler tone so that they are not burdensome to the animal.

[0617] For example, when a user gives the command "sit," the server will change its tone to a gentler one if it detects impatience in the user's voice. This allows the animal to understand the command without causing unnecessary stress.

[0618] Furthermore, this system can also support user interaction by generating an example prompt using the generated AI model, such as "Please tell me how to tell my pet to sit in a way that doesn't cause it stress."

[0619] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0620] Step 1:

[0621] The device uses a camera and microphone to acquire real-time video and audio information of animals. The input is a live feed of the animals, and the output is pre-processed digital data. This data is subjected to noise reduction filtering in real time.

[0622] Step 2:

[0623] The terminal transfers pre-processed video and audio information to the server. The input is denoised data, and the output is the data transferred to the server. This data includes timestamps and metadata.

[0624] Step 3:

[0625] The server analyzes the transferred video data using OpenCV. The input is video data, and the output is the result of extracting behavioral patterns. In this step, the animal's movements are identified as feature points.

[0626] Step 4:

[0627] The server uses Librosa to analyze audio data. The input is audio data, and the output is the result of extracting audio features. The audio waveform is converted into a frequency spectrum, and specific audio patterns are identified.

[0628] Step 5:

[0629] The server uses TensorFlow or Keras to estimate animal emotions based on data extracted from video and audio. The input is video and audio feature data, and the output is the estimated emotion and request results.

[0630] Step 6:

[0631] The server analyzes the user's voice and video data and uses an emotion engine to determine the user's emotional state. The input is the user's voice and video, and the output is data on the user's emotional state.

[0632] Step 7:

[0633] The server adjusts signals or sounds to the animal based on the user's emotional state. The inputs are the animal's emotional state and the user's emotional state, and the output is the adjusted communication signal.

[0634] Step 8:

[0635] The server reconstructs the analyzed animal's emotions and user instructions, and transmits them from the terminal in an animal-friendly tone. The input is a modified communication signal, and the output is a signal that the animal can easily understand.

[0636] Step 9:

[0637] The server uses a generative AI model to generate prompt messages for the user. The input is the animal's state and the user's instructions, and the output is a prompt message that assists the user's actions.

[0638] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0639] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0640] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0641] [Fourth Embodiment]

[0642] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0643] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0644] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0645] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0646] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0647] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0648] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0649] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0650] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0651] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0652] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0653] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0654] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0655] This invention is a system that analyzes animal behavior and vocalizations to estimate the emotions and needs of animals in real time. This system consists of a user terminal and a server.

[0656] The user places a device near the animal and acquires video and audio data through the camera and microphone. The device temporarily stores the acquired data, performs noise reduction and resolution adjustments, and prepares it for data transfer to the server.

[0657] The server uses video recognition AI and voice recognition AI to analyze data received from the terminal. Based on the video data, it analyzes the animal's movements and posture, and from the voice data, it detects the characteristics of the animal's vocalizations. Based on this information, the server estimates the animal's emotions and needs.

[0658] The estimated emotions and requests are sent from the server to the terminal and notified to the user. For example, if a dog is jumping while wagging its tail, the server will determine that the dog is "happy" and convey that information to the user through the terminal.

[0659] Furthermore, when a user enters a command into the terminal, the server converts that command into a signal or sound that the animal can understand. For example, if the command "stay" is entered, the server converts it into a sound signal of a specific frequency and transmits it to the animal.

[0660] This system also has a function to improve analysis accuracy by learning individual differences in animal behavior and vocalizations through learning methods. This allows it to learn the tendencies and habits of specific animals, enabling more accurate estimation of their emotions and needs.

[0661] The following describes the processing flow.

[0662] Step 1:

[0663] The device uses cameras and microphones placed near the animals to continuously acquire video and audio data. This prepares it to capture the animals' behavior and vocalizations in real time.

[0664] Step 2:

[0665] The terminal performs noise reduction and normalization processing on the acquired video and audio data to improve data quality. This processing is an important pre-processing step for improving analysis accuracy.

[0666] Step 3:

[0667] The terminal sends the pre-processed data to the server. This data transfer is an essential step for starting the analysis process.

[0668] Step 4:

[0669] The server uses video recognition AI to analyze the received video data and detect the animal's posture and movements. This makes it possible to objectively evaluate the animal's behavior.

[0670] Step 5:

[0671] The server runs a speech recognition AI to analyze the received audio data. The analysis identifies the frequency and patterns of the animal's calls and infers its emotions and needs.

[0672] Step 6:

[0673] The server integrates the video and audio analysis results and uses an AI model to estimate the animal's emotions and needs. The estimation results provide an overall assessment of the animal's condition.

[0674] Step 7:

[0675] The server sends estimated emotions and requests to the terminal and notifies the user. This allows the user to receive immediate feedback about the animal's condition.

[0676] Step 8:

[0677] When a user wants to give instructions to an animal, they input those instructions into a device. These instructions are then used for communication with the animal.

[0678] Step 9:

[0679] The server receives instructions sent by the user and converts them into signals or sounds that the animal can understand. This conversion process ensures that the user's intentions are accurately conveyed to the animal.

[0680] Step 10:

[0681] The server accumulates historical data to learn about individual differences among animals and improves the accuracy of its analysis through learning mechanisms. This continuous learning enables more accurate and reliable analysis.

[0682] (Example 1)

[0683] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0684] Accurately understanding animals' emotions and needs in real time has traditionally been difficult. This has led to problems such as inflicting unnecessary stress on animals or failing to provide appropriate care. Furthermore, there has been a lack of appropriate analytical methods that take individual animal differences into account, creating a need for more precise decision-making.

[0685] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0686] In this invention, the server includes an image acquisition means for acquiring and pre-processing visual information of an animal, an acoustic acquisition means for acquiring and pre-processing auditory information of an animal, and an information processing means for analyzing the information acquired by the image acquisition means and the acoustic acquisition means to estimate emotions and requests based on the animal's movements and vocalizations. This enables real-time and highly accurate analysis of emotions and requests based on the animal's behavior and vocalizations.

[0687] "Visual information acquisition means" refers to means for acquiring video data to record the movements and postures of animals and for pre-processing that data.

[0688] A "sound information acquisition means" is a means for acquiring data related to animal sounds and other sounds, and for preprocessing that data.

[0689] "Information processing means" refers to means for analyzing acquired visual and auditory information to estimate the emotions and needs of animals.

[0690] A "conversion means" is a means of converting human instructions into signals or sounds that can be recognized by animals.

[0691] A "notification means" is a means of notifying the user of the results analyzed by an information processing means.

[0692] A "generative AI model" is an artificial intelligence model used to analyze patterns of animal behavior and vocalizations and to estimate their emotions and needs.

[0693] The "learning function" is a function that learns about individual differences in animals and improves the accuracy of analysis.

[0694] This invention is a system that analyzes animal behavior and vocalizations in real time and estimates their emotions and needs. The system consists of a user, a terminal, and a server.

[0695] The user places the device close to the animal. The device is equipped with a camera and microphone, and acquires video and audio data of the animal in real time. This data is temporarily stored in the device and pre-processed, such as noise reduction and resolution adjustment.

[0696] The terminal digitizes the pre-processed data and sends it to the server. Examples of terminals include typical smartphones and dedicated IoT devices.

[0697] The servers are located in the cloud and run high-performance image recognition and speech recognition AI. For image recognition, software libraries such as OpenCV and TensorFlow are commonly used. For speech recognition, established speech analysis technologies such as the Google Speech-to-Text API are employed. The servers analyze the received data and comprehensively estimate emotions and requests based on animal movements, postures, and vocal patterns.

[0698] This analysis process utilizes a generative AI model that learns from previously accumulated animal behavior data to achieve highly accurate estimations that take individual differences into account. The analysis results are immediately transmitted to the terminal and notified to the user.

[0699] Furthermore, users can issue commands to animals through a terminal. When a user enters a command into the terminal, the server converts that command into signals or sounds that the animals can recognize and provides them. This conversion adjusts the frequency, volume, and other parameters to be specific to each animal.

[0700] As a concrete example, consider a scenario where a user is spending time with a dog in a park. The device captures the dog wagging its tail and obtains the corresponding audio. The server analyzes this and estimates that the dog is "happy." This information is then communicated to the user, enabling a more appropriate response. Another example of a prompt for the generative AI model is a sentence like, "Detect how long the dog is barking and estimate the reason."

[0701] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0702] Step 1:

[0703] The user places a terminal near the animal and uses the camera and microphone to acquire visual and auditory information. The input is real-time video and audio data of the animal. The terminal temporarily stores this data and performs noise reduction and resolution adjustments. The output is pre-processed digital data. This step improves the quality of the acquired data and optimizes it for analysis on the server.

[0704] Step 2:

[0705] The terminal packets pre-processed data and sends it to the server. The input is pre-processed video and audio data. Data format conversion is performed to efficiently send the data to the server. As output, transmittable data packets are generated. This step includes implementations to ensure secure and rapid data transfer.

[0706] Step 3:

[0707] The server receives data packets sent from the terminal. The input is the transmitted data packets. After verifying the reliability and integrity of the data, the server starts the analysis. The output is data ready for analysis. Receiving and verifying the data is a crucial operation for fulfilling the prerequisites for data analysis.

[0708] Step 4:

[0709] The server performs analysis using video recognition AI and speech recognition AI based on the received data. The input is pre-processed data deployed on the server. From the video data, the movements and postures of animals are analyzed, and from the audio data, patterns of vocalizations are identified. Emotions and requests are estimated using a generative AI model. As output, data of estimated emotions and requests is obtained. The generative AI model learns individual differences, enabling highly accurate estimations.

[0710] Step 5:

[0711] The server converts the estimated results into a data format and sends it to the terminal. The input is the estimated result data. The data is converted into an appropriate format for communication with the user. The output is data that can be communicated to the user. This step includes actions to organize the information the user receives in an easy-to-understand manner.

[0712] Step 6:

[0713] The terminal analyzes data received from the server and notifies the user. The input is estimated data sent from the server. The information is displayed in an appropriate format, and alerts are given via voice or other means. The output is the information delivered to the user. In this step, feedback is provided through an interface that the user can instantly understand.

[0714] Step 7:

[0715] Users can issue commands to animals by entering specific commands into a terminal. The input consists of commands from the user. These commands are sent to a server and converted into signals or sounds that animals can understand. The converted instructions are then transmitted to the animals as output. This process is designed to facilitate communication between animals and humans.

[0716] (Application Example 1)

[0717] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0718] In modern times, a challenge in communication between animals and humans is the difficulty in accurately understanding an animal's emotions and needs and responding appropriately based on that understanding. There is also a need to maintain smooth communication with pets even when away from home or during absences. Furthermore, there is the problem of difficulty in understanding an animal's condition in real time and responding appropriately.

[0719] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0720] In this invention, the server includes means for acquiring and pre-processing video information of animals, means for acquiring and pre-processing audio information of animals, and means for analyzing the acquired data and estimating emotions and requests based on the animal's behavior and vocalizations. This makes it possible to accurately grasp the animal's emotions and requests in real time and to transmit instructions to the animal via a smart device even when away from home.

[0721] "Image acquisition means" refers to a device that has the function of acquiring image information of animals using sensors or cameras and performing preprocessing as necessary.

[0722] A "sound acquisition means" is a device that has the function of acquiring animal sound information using a microphone or sound sensor and performing preprocessing as necessary.

[0723] "Analysis means" refers to a system that includes technology for analyzing acquired video and audio information and estimating the emotions and needs of animals based on their behavior and vocalizations.

[0724] A "conversion device" is a device that has the function of converting human instructions into signals or sounds that animals can understand.

[0725] A "notification device" is a device that has the function of notifying the user of the data analyzed by the analysis device, enabling them to immediately understand the animal's condition.

[0726] "Control means" refers to technology that connects to mechanical devices for animals and plays a role in providing the necessary control to ensure smooth communication.

[0727] The system for realizing this invention analyzes animal behavior and vocalizations in real time and notifies the user of their emotions and requests. This system mainly consists of terminals and servers.

[0728] The terminal is placed near the animal and uses a camera and microphone to acquire video and audio information about the animal. The terminal temporarily stores this information and performs preprocessing such as noise reduction and resolution adjustment. After that, the terminal prepares to transfer the preprocessed data to the server.

[0729] The server receives the transferred data as described above and analyzes it using image recognition AI and speech recognition AI. In this process, it uses AI frameworks such as TensorFlow and PyTorch to analyze the behavioral and vocal characteristics of animals and estimate their emotions and requests.

[0730] The estimated data is communicated to the user via a terminal. This allows the user to monitor the animal's condition in real time and take necessary actions. Furthermore, instructions entered by the user into the terminal are converted by a server into signals or sounds that the animal can understand. These converted instructions are transmitted to the animal's mechanical devices via control means, enabling smooth communication.

[0731] As a concrete example, one day while a user is out, they receive a notification that their pet dog is in an excited state. Based on this information, the user inputs the command "sit," and this command is transmitted to the dog via the device, reducing the dog's stress.

[0732] An example of a prompt might be: "Develop an app that estimates a pet's emotions from its behavior and sounds and sends a notification to the owner. For example, if the dog wags its tail or jumps, the app would notify the owner that the pet is 'happy'."

[0733] This invention aims to improve analysis accuracy by continuously learning individual differences in animal behavior and vocalizations, thereby providing more appropriate and accurate estimation results.

[0734] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0735] Step 1:

[0736] The device uses a camera and microphone to acquire video and audio information about animals. The input is camera video and microphone audio, and the output is raw video and audio data. Specifically, it works by pointing the camera at the animal and operating the microphone near the animal to acquire information.

[0737] Step 2:

[0738] The video and audio data acquired by the device are preprocessed by noise reduction and resolution adjustment. The input is the raw data acquired in step 1, and the output is the processed video and audio data. Specifically, noise is removed using filtering technology, and adjustments are made if the image quality is not clear.

[0739] Step 3:

[0740] The terminal sends the pre-processed data to the server. The input is the processed data generated in step 2, and the output is the data transferred to the server. Specifically, the data is uploaded to the server via the network using a communication protocol.

[0741] Step 4:

[0742] The server analyzes the received data using video recognition AI and speech recognition AI. The input is processed data sent from the terminal, and the output is an estimation of emotions and requests based on the animal's behavioral and vocal characteristics. Specifically, a generative AI model is used to analyze data patterns and obtain estimation results.

[0743] Step 5:

[0744] The server notifies the terminal of the estimation results, and the information is conveyed to the user. The input is the estimation results generated by the server, and the output is the information presented to the user. Specifically, information is sent to the user's terminal using push notifications, and the user is notified visually or audibly.

[0745] Step 6:

[0746] The user inputs instructions to the terminal. The input consists of commands performed by the user, and the output is instruction data sent to the server. Specifically, instructions such as "sit" or "stay" are input through screen operations.

[0747] Step 7:

[0748] The server converts user instructions into signals or sounds that animals can understand and transmits them to the animal's mechanical device via a terminal. The input is user instruction data, and the output is the converted signals or sounds. Specifically, a generative AI model is used to create signals, which are then transmitted to the animal using control devices.

[0749] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0750] This invention combines a system that analyzes animal behavior and vocalizations with an emotion engine that recognizes user emotions, in order to establish communication between animals and humans. This system consists of a terminal operated by the user and a server that performs data analysis.

[0751] The terminal uses cameras and microphones placed near the animals to collect video and audio data of the animals in real time. The terminal preprocesses the data, removes noise, and then transmits it to a server. This data is used to analyze the animals' behavior and vocal patterns.

[0752] The server analyzes the received video and audio data to estimate the animal's emotions and needs. Based on the information obtained through the analysis, it determines whether the animal is happy, hungry, or anxious.

[0753] Furthermore, the emotion engine analyzes the user's tone of voice, word choice, and even facial expressions from video data to identify the user's emotions. This allows it to detect the user's emotional state, such as whether they are stressed or calm.

[0754] When a user gives instructions to an animal, they input the content into a terminal. The server takes into account the user's emotional state, as detected by the emotion engine, and converts the instructions into signals or sounds that the animal can understand. This process ensures that, for example, even if the user is excited and gives instructions in a strong tone, the signals are adjusted to a gentle tone so that the animal does not experience unnecessary stress.

[0755] For example, when a user gives the command "sit," if the server senses impatience in the user's voice, it will convert the tone to a calmer one before transmitting it to the animal. In this way, the emotion engine plays a crucial role in facilitating smooth two-way communication.

[0756] Through the above process, smooth communication can be established between animals and users, enabling a rapid response to the animals' needs.

[0757] The following describes the processing flow.

[0758] Step 1:

[0759] The device uses a camera and microphone to acquire video and audio data of animals. This process records the animals' movements and sounds in real time.

[0760] Step 2:

[0761] The terminal performs preprocessing on the acquired data, including noise reduction and normalization. This improves the quality of the data, which enhances the accuracy of the analysis.

[0762] Step 3:

[0763] The terminal sends the pre-processed video and audio data to the server. At this stage, the data is ready for analysis.

[0764] Step 4:

[0765] The server uses video recognition AI to analyze animal behavior and detect posture and movements. This information forms the basis for determining the animal's emotions and needs.

[0766] Step 5:

[0767] The server uses speech recognition AI to analyze audio data and identify the frequency, tone, and pattern of animal calls. This allows it to estimate the animal's needs and mood.

[0768] Step 6:

[0769] The server utilizes an emotion engine to recognize the user's emotions. It analyzes the user's tone of voice, chosen words, and facial expressions to estimate their emotional state.

[0770] Step 7:

[0771] The server integrates the results of animal behavior and voice analysis with the user's emotional state, and converts the commands to the animal into gently adjusted signals and voices.

[0772] Step 8:

[0773] The server transmits the generated signal or sound to the terminal, which then relays it to the animal. In this process, the coordinated communication is delivered to the animal.

[0774] Step 9:

[0775] The user receives feedback from the system, observes and evaluates the animal's response, and adjusts instructions as needed to determine the next action.

[0776] In this way, two-way communication between animals and users is realized, enabling appropriate responses that meet the animals' needs and emotions.

[0777] (Example 2)

[0778] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0779] Facilitating smooth communication between animals and humans is extremely difficult. Conventional systems not only struggle to accurately understand animal emotions and needs, but also transmit information unilaterally without considering human emotional states, often leading to unsuccessful communication. A solution to this problem and a means of achieving meaningful communication between animals and humans are needed.

[0780] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0781] In this invention, the server includes data collection means for collecting and preprocessing animal behavior and voice data, analysis means for analyzing emotions and requests based on animal behavior and voice using a generative AI model, and user emotion analysis means for analyzing the tone of the user's voice and facial expressions and identifying their emotional state. This facilitates two-way communication between animals and humans, enabling a more accurate understanding of each other's emotions and requests.

[0782] "Data collection means" refers to a device or system that has the function of collecting and pre-processing animal behavior and sound data in real time.

[0783] "Data transfer means" refers to a device or system having communication protocols and functions for securely and quickly transferring pre-processed data to a server.

[0784] "Generative AI models" refer to artificial intelligence technologies used to analyze animal behavior and vocal data to estimate the emotions and needs of animals.

[0785] "Analysis means" refers to software or devices that utilize generative AI models to analyze emotions and requests based on animal behavior and vocalizations.

[0786] "User emotion analysis means" refers to software or a device that has the function of analyzing the tone of a user's voice and facial expressions to identify their emotional state.

[0787] "Conversion means" refers to software or a device that has the function of converting instructions given by a user into signals or sounds that an animal can understand.

[0788] "Notification means" refers to a device or system that has the function of transmitting information about the animal's condition obtained by the analysis means to the user.

[0789] A "learning tool" is software or a system that has the function of learning individual differences in animal behavior and vocalizations and improving the accuracy of analysis.

[0790] A "feedback improvement method" is a system that has the function of adjusting the generated AI model based on user feedback to improve its accuracy and effectiveness.

[0791] This invention is a system for facilitating communication between animals and humans. The system consists of a user-operated terminal and a server that performs data analysis. The terminal uses a camera and microphone placed near the animal to collect video and audio data of the animal in real time. After preprocessing the collected data, such as noise filtering, it is transferred to the server using a secure protocol.

[0792] The server analyzes the received video and audio data. Specifically, it uses a generative AI model to infer emotions and needs from animal behavior and vocalizations. This analysis determines whether the animal is happy, hungry, or anxious. Furthermore, the server uses an emotion engine to identify the user's emotional state based on the tone of voice and facial expressions provided by the user.

[0793] For example, consider a scenario where a user gives the command "sit." When the user inputs this command into the device, the server takes into account the user's emotional state, as determined by the emotion engine, and converts the command into signals and sounds that the animal can understand. Even if the user is excited and gives the command in a strong tone, the server converts the command into a calmer tone before conveying it to the animal. In this way, the system ensures smooth communication between the user and the animal.

[0794] A concrete example of a prompt message is, "When the user commands the dog to sit, try to communicate in a calm tone." Based on this prompt message, the server performs the necessary processing, and the tone and signals transmitted to the animal are adjusted accordingly.

[0795] This system allows for accurate understanding of animals' emotions and needs, and enables information transmission that appropriately considers the user's emotional state, thereby making communication with animals more meaningful.

[0796] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0797] Step 1:

[0798] The device uses a camera and microphone placed near the animal to collect video and audio data of the animal in real time. It takes in video data from the camera and audio data from the microphone as input, and improves the quality of the input data by applying noise filtering. The data, after this preprocessing is complete, is output in a format suitable for subsequent processing steps.

[0799] Step 2:

[0800] The terminal transfers pre-processed video and audio data to the server. This step uses a data transfer protocol (e.g., HTTP / 2 or WebSocket) to ensure security and speed. The input is filtered data, and the output is data transmitted via encrypted communication.

[0801] Step 3:

[0802] The server analyzes the received video and audio data. It utilizes a generative AI model to estimate emotions and needs from animal behavior and sounds. In this process, video and audio data are received as input, the model performs data analysis, and an estimated result is output, such as whether the animal is happy, hungry, or anxious.

[0803] Step 4:

[0804] The server identifies the user's emotional state from their voice tone and facial expressions. It receives audio and video data entered by the user into the terminal and analyzes it using an emotion engine. The input is the user's voice and video, and the output is the analysis result indicating the user's emotional state.

[0805] Step 5:

[0806] The server converts user instructions into signals or sounds that animals can understand. A generative AI model considers the user's emotional state and outputs instructions in a controlled tone that is not stressful for the animal. Input is the user's instructions and the emotional analysis results, while output is signals or sounds for the animal.

[0807] Step 6:

[0808] The user observes the animal's reactions and inputs the results as feedback into the device. This feedback information is sent to a server and used to improve the generated AI model. The input is feedback data based on the animal's reactions, while the output is data used to adjust and improve the AI ​​model.

[0809] (Application Example 2)

[0810] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0811] Systems for facilitating smooth communication between animals and humans need to accurately understand the animal's emotions and needs, and ensure that human instructions are properly conveyed to the animal. However, existing technologies struggle to accurately analyze animal behavior and vocalizations, and rarely adjust to the human emotional state. This can lead to miscommunication between animals and humans.

[0812] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0813] In this invention, the server includes a video acquisition means for acquiring and pre-processing video information of animals, a voice acquisition means for acquiring and pre-processing audio information of animals, an analysis means for analyzing the acquired data and estimating the animal's emotions and requests, a conversion means for converting human speech into a form that animals can understand, and an adjustment means for analyzing the user's emotional state and adjusting signals or voice tone. This improves the accuracy of communication between animals and humans, enabling both parties to interact without stress.

[0814] "Image acquisition means" refers to equipment or devices that have the function of collecting image information of animals and performing appropriate preprocessing.

[0815] "Voice acquisition means" refers to equipment or devices that have the function of collecting animal vocal information and appropriately pre-processing it.

[0816] "Analysis means" refers to a system or process that has the function of analyzing acquired video and audio data to estimate the emotions and needs of animals.

[0817] A "conversion means" is a device or process for converting human speech into signals or sounds that animals can understand.

[0818] "Notification means" refers to the mechanisms and technologies used to communicate analyzed data to the user.

[0819] "Adjustment means" refers to a device or method that analyzes the user's emotional state and adjusts the tone of signals or voices sent to the animal based on the results.

[0820] "Learning methods" refer to techniques and methods for learning individual differences in animal behavior and vocalizations and improving the accuracy of analysis.

[0821] This invention consists of a system combining a terminal and a server to enable smooth communication between animals and users.

[0822] The terminal uses cameras and microphones placed near the animals to collect video and audio information about the animals in real time. This information is pre-processed on-site to remove noise before being transmitted to the server.

[0823] The server is equipped with analytical means to analyze received video and audio data and estimate emotions and requests based on animal behavior and vocalizations. The analytical means uses Python-based OpenCV to analyze video data and Librosa to extract features from audio data. This information is then used with deep learning techniques utilizing TensorFlow and Keras to estimate animal emotions and requests with high accuracy.

[0824] The server also analyzes the user's emotional state using their voice and video data. To this end, an emotion engine analyzes the user's voice tone and facial expressions to determine their emotional state. For example, if the user is excited, this emotional data is used to adjust the tone of the signals transmitted to the animal. This adjustment mechanism can convert the user's commands into a gentler tone so that they are not burdensome to the animal.

[0825] For example, when a user gives the command "sit," the server will change its tone to a gentler one if it detects impatience in the user's voice. This allows the animal to understand the command without causing unnecessary stress.

[0826] Furthermore, this system can also support user interaction by generating an example prompt using the generated AI model, such as "Please tell me how to tell my pet to sit in a way that doesn't cause it stress."

[0827] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0828] Step 1:

[0829] The device uses a camera and microphone to acquire real-time video and audio information of animals. The input is a live feed of the animals, and the output is pre-processed digital data. This data is subjected to noise reduction filtering in real time.

[0830] Step 2:

[0831] The terminal transfers pre-processed video and audio information to the server. The input is denoised data, and the output is the data transferred to the server. This data includes timestamps and metadata.

[0832] Step 3:

[0833] The server analyzes the transferred video data using OpenCV. The input is video data, and the output is the result of extracting behavioral patterns. In this step, the animal's movements are identified as feature points.

[0834] Step 4:

[0835] The server uses Librosa to analyze audio data. The input is audio data, and the output is the result of extracting audio features. The audio waveform is converted into a frequency spectrum, and specific audio patterns are identified.

[0836] Step 5:

[0837] The server uses TensorFlow or Keras to estimate animal emotions based on data extracted from video and audio. The input is video and audio feature data, and the output is the estimated emotion and request results.

[0838] Step 6:

[0839] The server analyzes the user's voice and video data and uses an emotion engine to determine the user's emotional state. The input is the user's voice and video, and the output is data on the user's emotional state.

[0840] Step 7:

[0841] The server adjusts signals or sounds to the animal based on the user's emotional state. The inputs are the animal's emotional state and the user's emotional state, and the output is the adjusted communication signal.

[0842] Step 8:

[0843] The server reconstructs the analyzed animal's emotions and user instructions, and transmits them from the terminal in an animal-friendly tone. The input is a modified communication signal, and the output is a signal that the animal can easily understand.

[0844] Step 9:

[0845] The server uses a generative AI model to generate prompt messages for the user. The input is the animal's state and the user's instructions, and the output is a prompt message that assists the user's actions.

[0846] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0847] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0848] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0849] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0850] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0851] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0852] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0853] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0854] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0855] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0856] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0857] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0858] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0859] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0860] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0861] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0862] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0863] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0864] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0865] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0866] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0867] The following is further disclosed regarding the embodiments described above.

[0868] (Claim 1)

[0869] A video acquisition means for acquiring and pre-processing video data of animals,

[0870] A sound acquisition means for acquiring animal sound data and performing preprocessing,

[0871] An analysis means analyzes the data acquired by the aforementioned video acquisition means and audio acquisition means to estimate the emotions and demands of the animals based on their behavior and vocalizations.

[0872] A means of converting human speech into signals or sounds that animals can understand,

[0873] A notification means for notifying the user of the data analyzed by the aforementioned analysis means,

[0874] A system that includes this.

[0875] (Claim 2)

[0876] The system according to claim 1, further comprising a learning means for learning individual differences in animal behavior and vocalizations and improving the accuracy of analysis.

[0877] (Claim 3)

[0878] The system according to claim 1, wherein the analysis means has the function of identifying patterns of animal behavior and vocalizations and identifying the needs of the animals based thereon.

[0879] "Example 1"

[0880] (Claim 1)

[0881] Image acquisition means for acquiring and preprocessing visual information of animals,

[0882] Acoustic acquisition means for acquiring animal sound information and performing preprocessing,

[0883] Information processing means that analyzes the information acquired by the image acquisition means and sound acquisition means and estimates the emotions and requests of the animal based on its movements and vocalizations,

[0884] A means of converting human instructions into signals or sounds that animals can understand,

[0885] A notification means for notifying the user of the information analyzed by the aforementioned information processing means,

[0886] A means that uses a generative AI model to analyze patterns of animal behavior and vocalizations, and has the function of highly estimating emotions and demands,

[0887] A system that includes this.

[0888] (Claim 2)

[0889] The system according to claim 1, further comprising a learning function that learns individual differences in animal behavior and sound and improves the accuracy of analysis.

[0890] (Claim 3)

[0891] The system according to claim 1, wherein the information processing means has the function of identifying patterns of animal behavior and vocalizations and identifying the needs of the animals with high accuracy based thereon.

[0892] "Application Example 1"

[0893] (Claim 1)

[0894] A video acquisition means for acquiring and pre-processing video information of animals,

[0895] A sound acquisition means for acquiring and pre-processing animal sound information,

[0896] An analysis means analyzes the data acquired by the aforementioned video acquisition means and audio acquisition means to estimate the emotions and demands of the animals based on their behavior and vocalizations.

[0897] A means of converting human instructions into signals or sounds that animals can understand,

[0898] A notification means for notifying the user of the data analyzed by the aforementioned analysis means,

[0899] To enable communication with animals, a control means for connecting to animal-use mechanical devices,

[0900] A system that includes this.

[0901] (Claim 2)

[0902] The system according to claim 1, further comprising a learning means for learning individual differences in animal behavior and vocalizations and improving the accuracy of analysis.

[0903] (Claim 3)

[0904] The system according to claim 1, wherein the analysis means has the function of identifying patterns of animal behavior and vocalizations, and based thereon identifying the needs of the animals, and further includes the function of transmitting instructions via a mechanical device for animals.

[0905] "Example 2 of combining an emotion engine"

[0906] (Claim 1)

[0907] A data collection means for collecting and pre-processing animal behavior and sound data,

[0908] A data transfer means for transferring data acquired by the data collection means to a server,

[0909] An analytical means that uses a generative AI model to analyze emotions and requests based on animal behavior and voice,

[0910] A user emotion analysis means that analyzes the tone of the user's voice and facial expressions to identify their emotional state,

[0911] A conversion means that converts user instructions into signals or sounds that the animal can understand, and takes into account the animal's emotional state,

[0912] Based on the data obtained by the analysis means, a notification means for notifying the user of the animal's condition,

[0913] A system that includes this.

[0914] (Claim 2)

[0915] The system according to claim 1, further comprising a learning means for learning individual differences in animal behavior and vocalizations and improving analysis accuracy, and a feedback improvement means for adjusting the generated AI model based on user feedback.

[0916] (Claim 3)

[0917] The system according to claim 1, wherein the analysis means has a function to facilitate two-way communication by taking into account the emotional changes of the animal and the user.

[0918] "Application example 2 when combining with an emotional engine"

[0919] (Claim 1)

[0920] A video acquisition means for acquiring and pre-processing video information of animals,

[0921] A sound acquisition means for acquiring and pre-processing animal sound information,

[0922] An analysis means analyzes the data acquired by the aforementioned video acquisition means and audio acquisition means and estimates emotions and requests based on the animal's behavior and voice,

[0923] A means of converting human speech into signals or sounds that animals can understand,

[0924] A notification means for notifying the user of the data analyzed by the aforementioned analysis means,

[0925] An adjustment means that analyzes the user's emotional state and adjusts the tone of signals or voices to the animal based on that information,

[0926] A system that includes this.

[0927] (Claim 2)

[0928] The system according to claim 1, further comprising a learning means for learning individual differences in animal behavior and vocalizations and improving the accuracy of analysis.

[0929] (Claim 3)

[0930] The system according to claim 1, wherein the analysis means has the function of identifying patterns of animal behavior and vocalizations and identifying the needs of the animals based thereon. [Explanation of symbols]

[0931] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A video acquisition means for acquiring and pre-processing video data of animals, A sound acquisition means for acquiring animal sound data and performing preprocessing, An analysis means analyzes the data acquired by the aforementioned video acquisition means and audio acquisition means to estimate the emotions and demands of the animals based on their behavior and vocalizations. A means of converting human speech into signals or sounds that animals can understand, A notification means for notifying the user of the data analyzed by the aforementioned analysis means, A system that includes this.

2. The system according to claim 1, further comprising a learning means for learning individual differences in animal behavior and vocalizations and improving the accuracy of analysis.

3. The system according to claim 1, wherein the analysis means has the function of identifying patterns of animal behavior and vocalizations and identifying the needs of the animals based thereon.

Images