System

Abstract

A system is provided.SOLUTION: A system including a data receiving unit configured to analyze a pet's cry, image, or video, a data preprocessing unit configured to preprocess the received data into a format that can be analyzed, an analysis unit configured to determine a pet's emotion or health condition using a generated AI model based on the preprocessed data, a message generation unit configured to generate a message explaining in a natural language based on the analysis result, and a transmission unit configured to transmit the generated message and the generated pet's cry or image to a user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The challenges are that it is difficult to understand the sounds and behavior of pets, and there are limited means to accurately grasp the health condition of pets. This makes it difficult to properly understand the signals that pets give off, and therefore it is difficult to respond to the pet's needs promptly. Furthermore, it is difficult for owners to detect abnormalities in their pets early, making it difficult to manage the health of their pets. [Means for solving the problem]

[0005] The present invention is a system including a data receiving means for analyzing the sounds, images, and videos of a pet, a data preprocessing means for preprocessing the received data into an analyzable format, an analysis means for determining the pet's emotions and health condition using a generative AI model based on the preprocessed data, a message generating means for generating an explanatory message in natural language based on the analysis results, and a transmission means for transmitting the generated message and the generated sounds and images of the pet to a user's terminal. This makes it easier for owners to understand the pet's condition and take prompt and appropriate action.

[0006] The "data receiving means" is a communication means for transmitting sounds, images, and videos of pets from the user's terminal to the server.

[0007] "Data pre-processing means" refers to processing means for converting received data into an analyzable format.

[0008] "Analysis means" means a means for determining the emotions and health status of a pet using a generative AI model based on pre-processed data.

[0009] The "message generation means" is a means for generating a message explaining the results of the analysis in natural language.

[0010] The "transmission means" is a means for transmitting the generated message and the generated pet's cry and image to the user's terminal. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0012] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0013] First, the terms used in the following description will be explained.

[0014] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0015] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0016] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0017] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0019] [First embodiment]

[0020] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0021] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0024] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0027] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0031] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0032] This invention relates to a system that analyzes pet sounds, images, and videos and provides information to pet owners in an easy-to-understand format. This system receives data sent from a user's device, preprocesses it, analyzes it using a generative AI model, and provides the results to the pet owner.

[0033] System Configuration

[0034] 1. Data Receiving Method

[0035] The user uses a dedicated app to record and take pictures and videos of their pets, and the device sends the data to the server. This data receiving means is the part where the server receives and stores the data.

[0036] 2. Data preprocessing methods

[0037] The server converts the received data into an analyzable format. Specifically, it performs noise reduction on audio data and resizes and converts the format of image and video data. This data preprocessing method enables efficient analysis.

[0038] 3. Analysis method

[0039] The server uses a generative AI model to analyze the preprocessed data. For example, if it is audio data, it will assign an emotional label to the sound of the animal's bark (such as "joy" or "anxiety"), and if it is image or video data, it will determine the pet's health condition (such as "stress" or "health"). This analysis method allows for a detailed understanding of the pet's condition.

[0040] 4. Message Generation Methods

[0041] Based on the analysis results, the server generates a message in natural language explaining the pet's condition. Furthermore, it uses a generation AI to create virtual pet sounds and facial expressions. This message generation method makes it easier for owners to intuitively understand their pet's condition.

[0042] 5. Transmission Method

[0043] The server sends the generated message and the generated pet's bark and image to the device, which then notifies the user via the app, allowing them to quickly update the pet's status.

[0044] Specific Examples

[0045] Example 1: Dog bark analysis

[0046] Users record their dog's barking using a dedicated app.

[0047] The device sends the recorded data to the server.

[0048] The server receives the audio data and performs noise reduction.

[0049] The server uses a generative AI model to analyze the sound and determine that it is "anxious."

[0050] The server generates a message saying "The dog is anxious" and generates a virtual anxious bark.

[0051] The terminal receives the data from the server and notifies the user.

[0052] Users can check the notification in the app for more details.

[0053] Example 2: Cat image analysis

[0054] Users take photos of their cats using a dedicated app.

[0055] The terminal transmits the image data to the server.

[0056] The server receives the image data and performs any necessary pre-processing.

[0057] The server uses a generative AI model to analyze the image and determine whether it is "stressed."

[0058] The server generates a message saying "The cat is stressed" and an image of a facial expression showing stress.

[0059] The terminal receives the data from the server and notifies the user.

[0060] Users can check the notification in the app for more details.

[0061] In this way, the system of the present invention provides owners with an easy-to-understand view of their pet's condition, making it easier to meet their pet's needs and improving their pet's happiness and health.

[0062] The processing flow will be explained below.

[0063] Example 1: Dog bark analysis

[0064] Step 1:

[0065] Users use a dedicated app to record their dog's barking.

[0066] The device temporarily stores the recorded data.

[0067] Step 2:

[0068] The device sends the recording data to a server via an internet connection.

[0069] The server receives the recording data and stores it in a database.

[0070] Step 3:

[0071] The server performs pre-processing to remove noise from the received audio data.

[0072] The denoised audio data is converted into an analyzable format.

[0073] Step 4:

[0074] The server inputs the preprocessed audio data into a generative AI model.

[0075] The generative AI model analyzes the emotion of the bird's cry from the audio data and determines it to be "anxiety."

[0076] Step 5:

[0077] Based on the analysis results, the server creates a natural language message saying, "The dog is feeling anxious."

[0078] Additionally, it generates virtual cries that indicate anxiety.

[0079] Step 6:

[0080] The server sends the generated message and sound to the device.

[0081] Step 7:

[0082] The terminal receives the data from the server.

[0083] The device notifies the user of the received data via the app.

[0084] Step 8:

[0085] The user opens the app and sees the analysis results provided and the generated sounds.

[0086] Example 2: Cat image analysis

[0087] Step 1:

[0088] Users take photos of their cats using a dedicated app.

[0089] The device temporarily stores the captured data.

[0090] Step 2:

[0091] The device sends the photo data to a server via an Internet connection.

[0092] The server receives the photo data and stores it in a database.

[0093] Step 3:

[0094] The server preprocesses the received image data into an analyzable format.

[0095] Performs format conversion such as adjusting resolution and color.

[0096] Step 4:

[0097] The server inputs the preprocessed image data into the generative AI model.

[0098] The generative AI model analyzes the cat's facial expression from image data and determines its "stress."

[0099] Step 5:

[0100] Based on the analysis results, the server creates a natural language message saying, "The cat is feeling stressed."

[0101] Furthermore, a virtual facial expression image showing stress is generated.

[0102] Step 6:

[0103] The server sends the generated message and image to the terminal.

[0104] Step 7:

[0105] The terminal receives the data from the server.

[0106] The device notifies the user of the received data via the app.

[0107] Step 8:

[0108] The user opens the app and checks the analysis results provided and the generated facial expression image.

[0109] In this way, the specific processing of the system is carried out to analyze the pet's condition in detail through each step and provide the results to the owner.

[0110] Example 1

[0111] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0112] In recent years, there has been an increasing demand for pet health management and emotional understanding, but it is often difficult for pet owners to accurately grasp their pet's condition. With current technology, analyzing a pet's cries and facial expressions requires a high level of specialized knowledge, making it difficult for the average pet owner to access. To solve this problem, a system that allows owners to easily and intuitively understand their pet's condition is needed.

[0113] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0114] In this invention, the server includes means for transmitting data recorded or photographed by a user's device through a dedicated app, means for the server to receive the data and perform preprocessing such as noise removal and format conversion, means for determining emotion labels and health status using a generative AI model based on the preprocessed data, and means for generating virtual pet cries and facial expression images based on the analysis results, thereby enabling owners to easily understand their pet's emotions and health status through the dedicated app.

[0115] The "data receiving means" is a means for transmitting sounds, images, and videos of pets from the user's terminal to the server.

[0116] The "data preprocessing means" is a means for converting the data received by the server into an analyzable format, and includes processes such as noise removal, format conversion, and resizing.

[0117] "Analysis means" means a means for determining the emotions and health status of a pet using a generative AI model based on pre-processed data.

[0118] The "message generating means" is a means for generating a message that explains the state of the pet in natural language based on the analysis results.

[0119] The "transmission means" is a means for transmitting the generated message and the sound and image of the virtual pet to the user's terminal.

[0120] A "dedicated app" is an application program that users use to record and take pictures and videos of their pets and send them to a server.

[0121] "Noise reduction" is the process of removing unnecessary background sounds from audio data.

[0122] "Format conversion" refers to the process of converting data into a format that is easy to analyze.

[0123] A "generative AI model" is an artificial intelligence model that determines a pet's emotional label and health condition based on preprocessed data and outputs the results.

[0124] An "emotion label" is an emotion category (such as "joy" or "anxiety") that is inferred from a pet's sounds and behavior.

[0125] "Health status" refers to the health status (such as "stress" or "health") estimated from image and video data of a pet.

[0126] "Natural language" refers to words and sentences that humans use on a daily basis, and is used in this system to convey analysis results to users in an intuitively easy-to-understand format.

[0127] "Virtual pet cries" refer to the artificially generated cries of pets based on the analysis results.

[0128] A "virtual facial expression image" is an image of a pet's facial expression that is artificially generated based on the analysis results.

[0129] "User's terminal" refers to an electronic device (smartphone, tablet, etc.) used by a user on which a dedicated app is installed.

[0130] This invention relates to a system that allows users to analyze their pet's cries, images, and videos to more accurately and intuitively understand their pet's emotions and health condition. This system uses the user's device, a server, and a generative AI model to collect and analyze data and provide the results to the user.

[0131] Hardware and Software

[0132] Hardware:

[0133] Server: A central device that receives, stores, and analyzes data such as bird sounds, images, and videos.

[0134] Device: An electronic device (smartphone, tablet, etc.) used by a user to record and capture sounds, images, and videos of pets and send them to a server.

[0135] software:

[0136] Dedicated app: An application installed by the user that records and photographs data, sends it to the server, and receives and displays the analysis results.

[0137] OpenSMILE: Software used for noise reduction and preprocessing of audio data.

[0138] OpenCV: Software that performs preprocessing such as resizing and format conversion of image data.

[0139] Generative AI model: An artificial intelligence model (e.g., GPT-4, ResNet) that analyzes voice data and assigns emotion labels (happiness, anxiety, etc.), or analyzes image data and determines health status (stress, health, etc.).

[0140] Server application: A program that runs on the server and manages data reception, preprocessing, analysis, and transmission of generated data.

[0141] Program flow

[0142] 1. Data collection and transmission

[0143] Users use a dedicated app to record and take pictures and videos of their pets' cries.

[0144] The device sends the recorded and photographed data to a server using Wi-Fi or mobile data networks.

[0145] 2. Data Preprocessing

[0146] The server loads the received data, performs noise reduction on the audio data, and resizes and converts the image data. OpenSMILE is used for preprocessing the audio data, and OpenCV is used for preprocessing the image data.

[0147] 3. Data Analysis

[0148] The server inputs the preprocessed data into a generative AI model, determining emotion labels for bird sounds and health status for image data. GPT-4 and ResNet are used for analysis.

[0149] 4. Message Creation

[0150] Based on the analysis results, the server generates a natural language message explaining the pet's condition. It also generates the virtual pet's cries and facial expressions. It uses GPT-4 for message generation, Tacotron for voice generation, and DALL-E for image generation.

[0151] 5. Transmission and Notification

[0152] The server transmits the generated message and virtual data to the terminal.

[0153] The device notifies the user of the received data and allows them to check the analysis results using a dedicated app.

[0154] Specific examples

[0155] Example 1: Dog bark analysis

[0156] Users record their dog's barking using a dedicated app.

[0157] The device sends the recorded data to the server.

[0158] The server receives the audio data and uses OpenSMILE to perform noise reduction.

[0159] The server analyzes the sound using a generative AI model and determines that it is "anxious" (model used: GPT-4).

[0160] The server generates a message saying "The dog is anxious" and the Tacotron generates a virtual anxious bark.

[0161] The terminal receives the data from the server and notifies the user.

[0162] Prompt Sentence Examples

[0163] Enter your dog's bark and analyze its emotions.

[0164] Input audio: Woof woof woof (audio data)

[0165] Example 2: Cat image analysis

[0166] Users take photos of their cats using a dedicated app.

[0167] The terminal transmits the image data to the server.

[0168] The server receives the image data and uses OpenCV to resize and convert the format.

[0169] The server analyzes the image using a generative AI model and determines that it is "stressed" (model used: ResNet).

[0170] The server generates a message saying "The cat is feeling stressed," and DALL-E generates an image of a facial expression showing stress.

[0171] The terminal receives the data from the server and notifies the user.

[0172] Prompt Sentence Examples

[0173] Enter a photo of your cat and rate its health.

[0174] Input image: (image data of a cat)

[0175] This system allows users to quickly and accurately understand the condition of their pet and take appropriate action.

[0176] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0177] Step 1:

[0178] Users use a dedicated app to record and take pictures and videos of their pets' cries.

[0179] Specifically, the user presses the app's "record" button to record the dog's bark.

[0180] Input: Pet sounds, images, videos

[0181] Output: Audio, image, and video files stored on the user's device

[0182] Step 2:

[0183] The device sends the recorded and photographed data to the server.

[0184] Specifically, the device uses Wi-Fi or a cellular data network to send data to the server as an HTTP POST request.

[0185] Input: User recorded and photographed data

[0186] Output: Data uploaded to the server

[0187] Step 3:

[0188] The server stores the received data.

[0189] Specifically, the server receives the request and stores it in a database. For example, it stores the audio file in cloud storage and the metadata in a database.

[0190] Input: Data sent from the terminal

[0191] Output: Data stored on the server

[0192] Step 4:

[0193] The server pre-processes the audio data.

[0194] Specifically, OpenSMILE is used to remove noise from audio data.

[0195] Input: Stored audio data

[0196] Output: Denoised audio data

[0197] Step 5:

[0198] The server pre-processes the image data.

[0199] Specifically, OpenCV is used to resize and convert the format of image data.

[0200] Input: Saved image data

[0201] Output: Resized and formatted image data

[0202] Step 6:

[0203] The server inputs the preprocessed data into the generative AI model for analysis.

[0204] Specifically, it uses pre-trained generative AI models (e.g., GPT-4 and ResNet) to determine emotion labels from audio data and health status from image data.

[0205] Input: Preprocessed audio data, image data

[0206] Output: Emotion label (e.g., happy, anxious), health status (e.g., stress, healthy)

[0207] Step 7:

[0208] The server generates a message based on the analysis results.

[0209] Specifically, it uses a generative AI model (e.g., GPT-4) to generate analysis results as natural language messages, as well as virtual pet sounds and facial expressions.

[0210] Input: emotion label, health status

[0211] Output: Natural language message (e.g., the dog is anxious), virtual barks and facial expressions

[0212] Step 8:

[0213] The server sends the generated message and data to the terminal.

[0214] As a specific operation, an HTTP response including the generated message and data is created and sent to the terminal.

[0215] Input: Natural language messages, virtual sounds and facial expressions

[0216] Output: Messages and data sent to the terminal

[0217] Step 9:

[0218] The terminal notifies the user of the received data.

[0219] Specifically, the terminal uses a notification function to notify the user that "there is a new message."

[0220] Input: Message and data sent from the server

[0221] Output: User notification

[0222] Step 10:

[0223] Users can check the notifications through the app and understand their pet's condition.

[0224] Specifically, the user opens the app and views detailed messages and virtual data.

[0225] Input: Notification from device

[0226] Output: Analysis results and virtual data displayed through the app

[0227] This series of processes allows the user to quickly and accurately grasp the condition of the pet and take appropriate action.

[0228] (Application example 1)

[0229] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0230] Pet owners need specialized knowledge to understand their pets' emotions and health conditions, and this can lead to inadequate care. Pet shops and retailers are also required to quickly and easily assess the condition of their pets and provide appropriate advice, but there is currently no efficient way to meet this need.

[0231] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0232] In this invention, the server includes a data receiving means for analyzing the pet's cries, images, and videos, a data preprocessing means for preprocessing the received data into an analyzable format, an analysis means for determining the pet's emotions and health condition using a generative AI model based on the preprocessed data, a message generating means for generating an explanatory message in natural language based on the analysis results, a transmission means for transmitting the generated message and the generated pet's cries and images to a user's terminal, and an information output means for providing pet care information within a physical retail environment accessed by the user. This makes it possible to intuitively and quickly grasp the pet's emotions and health condition and efficiently provide appropriate care and product recommendations.

[0233] The "data receiving means" is a means by which a user can send sounds, images, and videos of their pet to the server via the terminal.

[0234] The "data preprocessing means" is a means for converting received data into an analyzable format, and performs noise removal, resizing, and format conversion.

[0235] "Analysis means" refers to a means for determining a pet's emotions and health status using a generative AI model based on pre-processed data.

[0236] The "message generation means" is a means for generating an explanatory message in natural language based on the analysis results, to clearly communicate the pet's condition.

[0237] The "transmission means" is a means for transmitting the generated message and the sound and image of the pet to the user's terminal.

[0238] "Information output means" refers to means for providing pet care information within the physical retail environment accessed by the user, and includes displays, audio guidance devices, and the like.

[0239] The present invention is a system for understanding the condition of pets and providing appropriate care, and is effective in pet shops and physical retail environments. Specific embodiments for implementing the system will be described below.

[0240] 1. System Configuration

[0241] This system consists of the following elements:

[0242] Data Receiving Method

[0243] Users use their smartphones or other devices to record and take pictures and videos of their pets, and then send the data to a server, where it is collected and analyzed in a central system.

[0244] Data preprocessing measures

[0245] The server converts the received data into an analyzable format. Specific processing includes noise reduction for audio data and resizing and format conversion for image and video data. This processing allows for efficient data analysis.

[0246] Analysis means

[0247] The server uses a generative AI model to analyze the preprocessed data. For example, if it is audio data, it can determine the emotion of the animal's voice (such as "joy" or "anxiety"), and if it is image or video data, it can determine the pet's health status (such as "stress" or "health"). This analysis uses libraries such as TensorFlow and OpenCV.

[0248] Message Generation Method

[0249] Based on the analysis results, the server generates a message in natural language explaining the pet's condition, and then uses generative AI to create virtual sounds and facial expressions, using OpenAI's GPT-3 or GPT-4.

[0250] Transmission method

[0251] The server sends the generated message, along with the pet's sounds and images, to the user's device, who receives this information through a dedicated app.

[0252] Information output means

[0253] In the physical retail environment accessed by the user (e.g., a pet shop), displays and robots installed in the store display the analysis results and provide customers with information on the condition of their pets and care advice.

[0254] 2. Hardware and Software Used

[0255] Hardware: Smartphones (iOS / Android), in-store robots, cloud servers (AWS, Google Cloud, etc.)

[0256] Software: Flask (a Python web application framework), TensorFlow (a machine learning framework), OpenCV (an image analysis library), and OpenAI's GPT-3 or GPT-4

[0257] 3. Specific Examples

[0258] Example 1: Pet emotion analysis by in-store robot

[0259] A robot installed inside a pet shop will record the sounds and images of customers' pets.

[0260] The robot sends the data to a server and receives the analysis results, which provide advice such as, "This dog seems a little anxious. Why don't you try a soothing toy?"

[0261] Based on the information provided in the app, pet store staff will recommend the right products.

[0262] Example 2: Checking health status using a smartphone app

[0263] The user takes a photo of their pet using a smartphone app and sends the data to the server.

[0264] The server analyzes the image and determines that the cat is stressed. Along with the analysis result, it generates an image of the cat's facial expression that indicates stress.

[0265] Users will receive notifications in the app and can check the details to provide appropriate care.

[0266] 4. Examples of prompts

[0267] An example prompt has the following format:

[0268] Generate a message to communicate the pet's status:

[0269] Take your pet's sound, image, or any media and explain the analysis results in natural language, for example, "My dog ​​is feeling anxious."

[0270] This system makes it possible to intuitively and quickly grasp a pet's emotional and health status, and efficiently provide appropriate care and product recommendations.

[0271] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0272] Step 1:

[0273] Users can use their smartphones or in-store robots to record and capture sounds, images, and videos of their pets, and then send the data. The input includes sounds, images, and video data of the pets. This data is then sent from the device to the server.

[0274] Step 2:

[0275] The server receives the data sent from the user using the data receiving means. The input includes the pet's cry, image, and video data sent in step 1. The received data is stored on the server.

[0276] Step 3:

[0277] The server preprocesses the received data using the data preprocessing means. Specifically, it performs noise removal on audio data, resizing and format conversion on image and video data. The input is the data received in step 2, and the output is the data after noise removal and resizing processing.

[0278] Step 4:

[0279] The server uses analytical means to analyze the preprocessed data with a generative AI model. For audio data, the emotion of the animal's cry (such as "joy" or "anxiety") is determined, and for images and videos, the pet's health status (such as "stress" or "health") is determined. The input is the preprocessed data, and the output is the analysis results, such as an emotion label or a health status determination.

[0280] Step 5:

[0281] Based on the analysis results, the server uses a message generation means to generate a message explaining the pet's condition in natural language. Furthermore, it uses a generation AI to create the virtual pet's cries and facial expressions. The input is the analysis results, and the output is the generated natural language message and the virtual pet's cries and facial expressions.

[0282] Step 6:

[0283] The server transmits the generated message and the generated pet sound and image to the user's terminal using a transmission means. The input is the message and image generated in step 5, and the output is a notification to the user's terminal.

[0284] Step 7:

[0285] The user receives messages and images sent from the server through a device (smartphone or in-store robot) and checks the status of their pet. The input is the sent message and image, and the output is the user's understanding and execution of appropriate action.

[0286] Step 8:

[0287] Using information output means, pet care information is provided in the physical retail environment (pet shop) accessed by the user. The input is analysis results and messages sent from the server, and the output is information provided by in-store displays and robots. Users and staff use this information to provide pet care and product recommendations.

[0288] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0289] This invention relates to a system that analyzes pet sounds, images, and videos, and combines them with an emotion engine that recognizes the user's emotions to provide information to pet owners in an easy-to-understand format. This system receives data sent from the user's device, preprocesses it, and uses a generative AI model to analyze the pet and user's emotions, and provides the results to the owner.

[0290] System Configuration

[0291] 1. Data Receiving Method

[0292] The user uses a dedicated app to record and take pictures and videos of their pets, and the device sends the data to the server. This data receiving means is the part where the server receives and stores the data.

[0293] 2. Data preprocessing methods

[0294] The server converts the received data into an analyzable format. Specifically, it performs noise reduction on audio data and resizes and converts the format of image and video data. This data preprocessing method enables efficient analysis.

[0295] 3. Analysis method

[0296] The server uses a generative AI model to analyze the preprocessed data. For example, if it is audio data, it will assign an emotional label to the sound of the animal's bark (such as "joy" or "anxiety"), and if it is image or video data, it will determine the pet's health condition (such as "stress" or "health"). This analysis method allows for a detailed understanding of the pet's condition.

[0297] 4. Emotion Engine

[0298] The server separately receives the user's camera footage and audio, and uses an emotion engine to analyze the user's emotions. The analysis results are displayed as emotion labels such as "happiness," "sadness," and "surprise." This allows the owner's psychological state to be understood at the same time.

[0299] 5. Message Generation Methods

[0300] Based on the analysis results, the server generates a message that comprehensively explains the status of the pet and the user. Furthermore, it uses a generation AI to create virtual pet sounds and facial expressions. This message generation method makes it easier for owners to intuitively understand the status of their pet and themselves.

[0301] 6. Transmission Method

[0302] The server sends the generated message and the generated pet's sound and image to the device, which then notifies the user via the app, allowing the user to quickly update the status of their pet and themselves.

[0303] Specific Examples

[0304] Example 1: Dog barking and user emotion analysis

[0305] Users record their dog's bark using a dedicated app and capture their own facial expressions using the camera.

[0306] The terminal transmits the recording data and image data to the server.

[0307] The server receives the audio data and image data and performs noise removal and preprocessing.

[0308] The server analyzes the sound using a generative AI model and determines it as "anxiety." At the same time, it analyzes the user's emotions using an emotion engine and determines it as "surprise."

[0309] The server generates a message saying "The dog is anxious and the user is surprised" and generates a virtual anxious cry and an image of a surprised expression.

[0310] The terminal receives the data from the server and notifies the user.

[0311] The user checks the notification in the app and sees the analysis results and generated data.

[0312] Example 2: Cat images and user sentiment analysis

[0313] Users take photos of their cats using a dedicated app and then use the camera to capture their own images.

[0314] The terminal transmits the photo data and image data to the server.

[0315] The server receives the photo data and image data and performs any necessary pre-processing.

[0316] The server analyzes the image using a generative AI model and determines whether it is "stressed." At the same time, it analyzes the user's emotions using an emotion engine and determines whether it is "joy."

[0317] The server generates a message saying "The cat is stressed, and the user is happy," and generates an image of a stressed expression and an image of a happy expression.

[0318] The terminal receives the data from the server and notifies the user.

[0319] The user checks the notification in the app and sees the analysis results and generated data.

[0320] In this way, the system of the present invention provides an easy-to-understand picture of the pet and user's condition, making it easier to meet the pet's needs and improving the happiness and health of both the pet and the owner.

[0321] The processing flow will be explained below.

[0322] Example 1: Dog barking and user emotion analysis

[0323] Step 1:

[0324] Users use a dedicated app to record their dog's barking.

[0325] At the same time, the user captures their own facial expressions using the app's camera function.

[0326] Step 2:

[0327] The device temporarily stores the audio recording data and facial expression image data.

[0328] Step 3:

[0329] The device transmits the audio recording data and facial expression image data to a server via an internet connection.

[0330] The server receives the audio recording data and facial expression image data and stores them in a database.

[0331] Step 4:

[0332] The server performs noise reduction on the received audio data.

[0333] The server adjusts the layout and converts the format of the image data.

[0334] Step 5:

[0335] The server inputs the preprocessed audio data into a generative AI model.

[0336] The generative AI model determines the emotion of a dog's bark from audio data as "anxiety."

[0337] Step 6:

[0338] The server inputs the preprocessed facial expression image data into the emotion engine.

[0339] The emotion engine determines the user's emotion as "surprise" from the facial expression image.

[0340] Step 7:

[0341] Based on the analysis results, the server generates a message saying, "The dog is feeling anxious and the user is surprised."

[0342] In addition, it generates virtual cries that indicate anxiety and images of surprised facial expressions.

[0343] Step 8:

[0344] The server sends the generated message, the bird's cry, and an image of its facial expression to the device.

[0345] Step 9:

[0346] The terminal receives the data from the server.

[0347] The device notifies the user of the received data via the app.

[0348] Step 10:

[0349] The user checks the notification in the app and sees the analysis results provided and the generated sound and facial expression images.

[0350] Example 2: Cat images and user sentiment analysis

[0351] Step 1:

[0352] Users take photos of their cats using a dedicated app.

[0353] At the same time, the user captures an image of themselves using the app's camera function.

[0354] Step 2:

[0355] The device temporarily stores the photograph data and facial expression image data.

[0356] Step 3:

[0357] The device transmits the photograph data and facial expression image data to a server via an internet connection.

[0358] The server receives the photograph data and facial expression image data and stores them in a database.

[0359] Step 4:

[0360] The server adjusts the layout and corrects the color of the image data received.

[0361] Step 5:

[0362] The server inputs the preprocessed image data into the generative AI model.

[0363] The generative AI model determines from image data that the cat's facial expression is "stressed."

[0364] Step 6:

[0365] The server inputs the preprocessed facial expression image data into the emotion engine.

[0366] The emotion engine determines the user's emotion as "joy" from the facial expression image.

[0367] Step 7:

[0368] Based on the analysis results, the server generates a message saying, "The cat is stressed, and the user is happy."

[0369] Furthermore, virtual facial expression images showing stress and happy expressions are generated.

[0370] Step 8:

[0371] The server sends the generated message and image to the terminal.

[0372] Step 9:

[0373] The terminal receives the data from the server.

[0374] The device notifies the user of the received data via the app.

[0375] Step 10:

[0376] The user checks the notification in the app and sees the analysis results and generated facial expression images.

[0377] In this way, the system of the present invention provides an easy-to-understand picture of the pet and user's condition, making it easier to meet the pet's needs and improving the happiness and health of both the pet and the owner.

[0378] Example 2

[0379] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0380] Previous systems that analyzed pet sounds, images, and videos were limited to understanding the pet's emotions and health status, but were unable to consider the owner's emotional state. This made it difficult to comprehensively understand the relationship between pet and owner, and meant that the systems were unable to adequately meet the pet's needs.

[0381] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a data receiving means for analyzing the pet's cries, images, and videos, a data preprocessing means for preprocessing the received data into an analyzable format, an analysis means for determining the pet's emotions and health condition using a generative AI model based on the preprocessed data, a means for receiving the user's camera video and audio and analyzing the user's emotions using an emotion engine, a message generating means for generating an explanatory message in natural language based on the analysis results, and a transmission means for transmitting the generated message and the generated pet's cries and images to the user's terminal. This makes it possible to comprehensively analyze the emotional states of the pet and the user and provide information to the owner in a format that is easy to understand.

[0382] The "data receiving means" is a mechanism for receiving pet sounds, images, and videos from the user's device.

[0383] The "data preprocessing means" is a mechanism that performs noise removal, resizing, and format conversion to convert received data into an analyzable format.

[0384] A "generative AI model" is a pre-trained artificial intelligence algorithm used to determine a pet's emotions and health status.

[0385] The "analysis means" is a mechanism that uses a generative AI model based on pre-processed data to determine the pet's emotions and health status.

[0386] "Means for receiving camera footage and audio from a user and analyzing the user's emotions using an emotion engine" is a mechanism for receiving camera footage and audio from a user and analyzing the user's emotions using an emotion engine.

[0387] The "message generation means" is a mechanism for generating an explanatory message in natural language based on the analysis results of the pet and the user.

[0388] The "transmission means" is a mechanism for transmitting the generated message and the generated pet's cry and image to the user's terminal.

[0389] This invention relates to a system that analyzes pet sounds, images, and videos, and combines them with an emotion engine that recognizes the user's emotions to provide information to pet owners in an easy-to-understand format. This system receives data sent from the user's device, preprocesses it, and uses a generative AI model to analyze the emotions of the pet and the user, and provides the results to the owner.

[0390] System Configuration

[0391] Data Receiving Method

[0392] The user uses a dedicated app to record and take pictures and videos of their pets, and the device sends the data to the server. This data receiving means is the part where the server receives and stores the data.

[0393] Data preprocessing measures

[0394] The server converts the received data into an analyzable format. Specifically, it performs noise removal on audio data and resizes and converts the format of image and video data. As data preprocessing methods, it uses FFT-based filtering for audio processing and the OpenCV library for image processing.

[0395] Analysis means

[0396] The server analyzes the preprocessed data using a generative AI model. TensorFlow or PyTorch are used for the generative AI model. For example, if the data is audio, it can assign an emotional label to the sound of the animal's bark (such as "joy" or "anxiety"), and if the data is image or video, it can determine the pet's health status (such as "stress" or "health"). This allows for a detailed understanding of the pet's condition.

[0397] Emotion Engine

[0398] The server separately receives the user's camera footage and audio, and uses an emotion engine to analyze the user's emotions. The analysis results are displayed as emotion labels such as "happiness," "sadness," and "surprise." The emotion engine uses Microsoft Azure Cognitive Services, which allows the owner's psychological state to be grasped at the same time.

[0399] Message Generation Method

[0400] Based on the analysis results, the server generates a message that comprehensively describes the status of the pet and the user. A natural language generation engine (e.g., GPT-4) is used to generate the message. Generative AI is also used to create virtual pet sounds and facial expressions. The DALL-E model is used to generate the virtual data.

[0401] Transmission method

[0402] The server sends the generated message and the generated pet's sound and image to the device, which then notifies the user via the app, allowing the user to quickly update the status of their pet and themselves.

[0403] Specific Examples

[0404] Example 1: Dog barking and user emotion analysis

[0405] Users record their dog's bark using a dedicated app and capture their own facial expressions using the camera.

[0406] The terminal transmits the recording data and image data to the server.

[0407] The server receives the audio data and performs noise reduction, and resizes the image data.

[0408] The server analyzes the sound using a generative AI model and determines it as "anxiety." At the same time, it analyzes the user's emotions using an emotion engine and determines it as "surprise."

[0409] The server generates a message saying "The dog is anxious and the user is surprised" and generates a virtual anxious cry and an image of a surprised expression.

[0410] The terminal receives the data from the server and notifies the user.

[0411] The user can check the analysis results and generated data in the app.

[0412] Example 2: Cat images and user sentiment analysis

[0413] Users take photos of their cats using a dedicated app and then use the camera to capture their own images.

[0414] The terminal transmits the photo data and image data to the server.

[0415] The server receives the photo data and image data and performs any necessary pre-processing.

[0416] The server analyzes the image using a generative AI model and determines whether it is "stressed." At the same time, it analyzes the user's emotions using an emotion engine and determines whether it is "joy."

[0417] The server generates a message saying "The cat is stressed, and the user is happy," and generates an image of a stressed expression and an image of a happy expression.

[0418] The terminal receives the data from the server and notifies the user.

[0419] The user checks the notification in the app and sees the analysis results and generated data.

[0420] In this way, the system of the present invention provides an easy-to-understand picture of the pet and user's condition, making it easier to meet the pet's needs and improving the happiness and health of both the pet and the owner.

[0421] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0422] Step 1: Getting and sending data

[0423] The user launches a dedicated app installed on their smartphone. They tap the record button in the app to record their pet's cries, or tap the camera button to take photos or videos of their pet. When recording or taking a photo, the user's facial expressions are also captured by the camera.

[0424] Input: Pet sounds, images, videos, user face images

[0425] Output: Audio files, image files, video files, user face image data

[0426] The terminal sends this data to the server using the HTTP protocol.

[0427] Step 2: Receiving and storing data

[0428] The server receives data sent from the device via a RESTful API. The received data is stored in local storage or cloud storage (e.g., AWS S3). Once stored, metadata is recorded in a database (e.g., MySQL) for data management.

[0429] Input: Audio files, image files, video files, user face image data

[0430] Output: Saved files, metadata

[0431] Step 3: Preprocessing the data

[0432] The server picks up the stored audio data and applies noise reduction algorithms (e.g., FFT-based filtering). For image and video data, it resizes and converts the format using the OpenCV library. Specifically, images are resized to 640x480 pixels and videos are encoded in MP4 format.

[0433] Input: saved audio files, image files, video files

[0434] Output: Pre-processed audio data, resized and formatted image and video data

[0435] Step 4: Analyze the data

[0436] The server inputs the preprocessed data into a generative AI model (e.g., TensorFlow model) for analysis. An emotion analysis model is applied to the audio data to assign labels such as "joy," "anxiety," and "anger," and an object detection model is applied to the image and video data to determine the pet's health status (e.g., "stressed," "relaxed," etc.).

[0437] Input: Preprocessed audio data, image and video data

[0438] Output: Audio data with emotion labels, image and video data with health status labels

[0439] Step 5: Analyzing user sentiment

[0440] The server receives the user's camera footage and audio data and analyzes it using an emotion engine (e.g., Microsoft Azure Cognitive Services). As a result, it obtains the user's emotion label (e.g., "joy," "sadness," "surprise," etc.).

[0441] Input: User's face image data, voice data

[0442] Output: User data with emotion labels

[0443] Step 6: Message Generation

[0444] The server generates messages using a natural language generation engine (e.g., GPT-4) based on the analysis results of the pet and the user. It also uses generative AI to create virtual pet sounds and facial expressions. The DALL-E model is used to generate the virtual data.

[0445] Input: Analysis results of pet and user (emotion labels, health status labels, etc.)

[0446] Output: Generated natural language messages, virtual sounds and facial expressions

[0447] Step 7: Data submission and notification

[0448] The server sends the generated message and virtual data to the API endpoint of the dedicated app.

[0449] The device receives the data from the server, stores it locally, and displays a notification to the user through the app. When the user opens the app, they can view the analysis results and the generated virtual data.

[0450] Input: Generated natural language messages, virtual sounds, and facial expressions

[0451] Output: Notification to user terminal, display of analysis results and generated data

[0452] This system can comprehensively analyze the emotional states of both the pet and the user, and provide information in a format that is easy for the owner to understand.

[0453] (Application example 2)

[0454] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0455] Pet owners spend a lot of time and effort understanding their pets' emotions and health conditions and providing appropriate care. It is also difficult to find products and services that suit their pets' conditions. As a result, pet needs cannot be met quickly and accurately, which can lead to a decline in the happiness and health of both the pet and the owner. To solve this problem, a system is needed that analyzes the conditions of both the pet and the owner and suggests appropriate products and services based on that analysis.

[0456] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data receiving means for analyzing the pet's cries, images, and videos; a data preprocessing means for preprocessing the received data into an analyzable format; an analysis means for determining the pet's emotions and health condition using a generative AI model based on the preprocessed data; a proposal means for proposing products and services based on the pet's condition and the user's emotions determined by the analysis means; a message generation means for generating an explanatory message in natural language based on the analysis results; and a transmission means for transmitting the generated message and the generated pet's cries and images to the user's terminal. This allows the owner to quickly understand the pet's condition and easily select and purchase appropriate products and services based on that information.

[0457] The "data receiving means" is a device or system for receiving data related to the sounds, images, and videos of pets, as well as the user's emotions.

[0458] "Data preprocessing means" refers to processing devices and algorithms that convert received data into an analyzable format. This means performs preprocessing such as noise removal and format conversion.

[0459] "Analysis means" refers to a device or system that uses a generative AI model to determine the pet's emotions and health status based on pre-processed data.

[0460] The "suggestion means" is a device or system that suggests appropriate products or services based on the state of the pet and the user's emotions determined by the analysis means.

[0461] The "message generating means" is a device or system that generates a message explaining the state of the pet and the user in natural language based on the analysis results.

[0462] The "transmission means" is a device or system that transmits the generated message and the sounds and images of the pet to the user's terminal.

[0463] A "generative AI model" is an artificial intelligence model used to analyze a pet's emotions and health condition from received data.

[0464] "Natural language" refers to a language format that humans use on a daily basis, and is used to provide analysis results to users in an intuitively understandable format.

[0465] "Products and services" is a general term for items and support activities suggested depending on the pet's condition and the user's emotions.

[0466] The present invention provides a system for analyzing the state of a pet and the emotions of a user, and for proposing appropriate products and services based on the results. This system has the following configuration.

[0467] First, the user installs a dedicated application on their smartphone and captures their pet's cries, images, and videos. This activates the data receiving means, which then transmits the captured data to the server. This data receiving means includes a standard smartphone camera and microphone.

[0468] Next, the server preprocesses the received data using a data preprocessing means. This preprocessing means includes, for example, noise removal from audio data, resizing of image data, and format conversion. This process typically uses a programming language such as Python, with libraries such as OpenCV or Librosa.

[0469] Once preprocessed, the data is analyzed using a generative AI model. The analysis methods include analyzing pet sounds to determine emotions, and analyzing images and videos to determine health and stress levels. The generative AI model is built using deep learning frameworks such as TensorFlow and PyTorch.

[0470] Furthermore, the analysis method analyzes not only the pet's condition but also the user's emotions. The user captures their own facial expressions with their smartphone camera, and the data is sent to the server. The emotion engine determines the user's emotions based on this data. This analysis uses Facial Recognition API and natural language processing engines (NLP engines), among others.

[0471] Based on the analysis results, the suggestion unit operates to suggest products and services according to the condition of the pet and the user. These suggestions include items to reduce stress for pets and relaxation items for owners.

[0472] The message generation means then generates a message in natural language based on the analysis results. For example, "If your pet is feeling stressed, why not try purchasing some relaxation products?" This generated message, along with the sound and image of the pet, is then sent back to the user's smartphone via the transmission means. The user receives this as a notification and can check it on the application.

[0473] For example, the following prompts might be fed into a generative AI model:

[0474] Output analysis results based on the pet's image and the user's emotion. For example, if the pet is stressed and the user is surprised, return information including the emotion label. Also, suggest products (such as relaxation goods) appropriate for the pet's condition.

[0475] This system allows users to quickly understand the condition of their pets and easily select and purchase appropriate products and services based on that information, ultimately improving the happiness and health of both pets and their owners.

[0476] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0477] Step 1:

[0478] The user uses a smartphone to capture the sound, images, and videos of their pet, and then sends the data to the server through a dedicated application. The input data is the sound, images, and videos of the pet, and the output is the raw data sent to the server. This step uses the smartphone's built-in camera and microphone.

[0479] Step 2:

[0480] The server receives data sent by the user using a data receiving means. The input data is the unprocessed data sent from the smartphone, and the output is the received data. In this step, communication means such as HTTP requests and WebSockets are used.

[0481] Step 3:

[0482] The server uses data preprocessing to convert the received data into an analyzable format. The input data is the received raw data, and the output is the preprocessed data. This step involves noise removal for audio data, resizing of image data, and format conversion. Specifically, the Python libraries OpenCV and Librosa are used.

[0483] Step 4:

[0484] The server analyzes the preprocessed data using a generative AI model. The input data is the preprocessed data, and the output is the analysis results showing the pet's emotions and health status. In this step, a generative AI model built using TensorFlow and PyTorch is used. The model assigns emotion labels such as "joy" or "anxiety" to the pet's cries, and determines its state, such as "stress" or "health," from images and videos.

[0485] Step 5:

[0486] The server analyzes the user's emotions using an emotion engine. The input data is the user's facial expression data captured by the smartphone camera, and the output is an analysis result indicating the user's emotions. In this step, the Facial Recognition API and a natural language processing engine (NLP engine) are used.

[0487] Step 6:

[0488] The server runs a suggestion unit that suggests products and services based on the analysis results. The input data are the analysis results of the pet and the user, and the output is a list of suggested products and services. In this step, a recommendation engine is used to select products and services suitable for the user. If the pet is feeling stressed, for example, relaxation goods will be suggested.

[0489] Step 7:

[0490] The server uses a message generation means to generate a message in natural language. The input data is the analysis results and a list of suggestions, and the output is a message to be sent to the user. The generated message will explain the pet's condition and the suggested products and services. In this step, an NLP engine is used to generate an intuitive and easy-to-understand message.

[0491] Step 8:

[0492] The server operates a sending means that sends the generated message and the generated pet sound and image to the user's device. The input data is the generated message and the pet sound and image, and the output is a notification that arrives on the user's smartphone. In this step, a push notification service (e.g., Firebase Cloud Messaging) is used.

[0493] Step 9:

[0494] The user checks the notification through the application and views the analysis results and suggested products and services. The input data is the notification sent from the server, and the output is the information provided to the user. In this step, a dedicated application on the smartphone is used, allowing the user to easily check the information and select and purchase the suggested products and services.

[0495] By linking these processing steps, the user can quickly grasp the condition of their pet and provide appropriate care based on that information.

[0496] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0497] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0498] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0499] [Second embodiment]

[0500] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0501] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0502] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0503] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0504] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0505] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0506] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0507] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0508] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0509] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0510] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0511] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0512] This invention relates to a system that analyzes pet sounds, images, and videos and provides information to pet owners in an easy-to-understand format. This system receives data sent from a user's device, preprocesses it, analyzes it using a generative AI model, and provides the results to the pet owner.

[0513] System Configuration

[0514] 1. Data Receiving Method

[0515] The user uses a dedicated app to record and take pictures and videos of their pets, and the device sends the data to the server. This data receiving means is the part where the server receives and stores the data.

[0516] 2. Data preprocessing methods

[0517] The server converts the received data into an analyzable format. Specifically, it performs noise reduction on audio data and resizes and converts the format of image and video data. This data preprocessing method enables efficient analysis.

[0518] 3. Analysis method

[0519] The server uses a generative AI model to analyze the preprocessed data. For example, if it is audio data, it will assign an emotional label to the sound of the animal's bark (such as "joy" or "anxiety"), and if it is image or video data, it will determine the pet's health condition (such as "stress" or "health"). This analysis method allows for a detailed understanding of the pet's condition.

[0520] 4. Message Generation Methods

[0521] Based on the analysis results, the server generates a message in natural language explaining the pet's condition. Furthermore, it uses a generation AI to create virtual pet sounds and facial expressions. This message generation method makes it easier for owners to intuitively understand their pet's condition.

[0522] 5. Transmission Method

[0523] The server sends the generated message and the generated pet's bark and image to the device, which then notifies the user via the app, allowing them to quickly update the pet's status.

[0524] Specific Examples

[0525] Example 1: Dog bark analysis

[0526] Users record their dog's barking using a dedicated app.

[0527] The device sends the recorded data to the server.

[0528] The server receives the audio data and performs noise reduction.

[0529] The server uses a generative AI model to analyze the sound and determine that it is "anxious."

[0530] The server generates a message saying "The dog is anxious" and generates a virtual anxious bark.

[0531] The terminal receives the data from the server and notifies the user.

[0532] Users can check the notification in the app for more details.

[0533] Example 2: Cat image analysis

[0534] Users take photos of their cats using a dedicated app.

[0535] The terminal transmits the image data to the server.

[0536] The server receives the image data and performs any necessary pre-processing.

[0537] The server uses a generative AI model to analyze the image and determine whether it is "stressed."

[0538] The server generates a message saying "The cat is stressed" and an image of a facial expression showing stress.

[0539] The terminal receives the data from the server and notifies the user.

[0540] Users can check the notification in the app for more details.

[0541] In this way, the system of the present invention provides owners with an easy-to-understand view of their pet's condition, making it easier to meet their pet's needs and improving their pet's happiness and health.

[0542] The processing flow will be explained below.

[0543] Example 1: Dog bark analysis

[0544] Step 1:

[0545] Users use a dedicated app to record their dog's barking.

[0546] The device temporarily stores the recorded data.

[0547] Step 2:

[0548] The device sends the recording data to a server via an internet connection.

[0549] The server receives the recording data and stores it in a database.

[0550] Step 3:

[0551] The server performs pre-processing to remove noise from the received audio data.

[0552] The denoised audio data is converted into an analyzable format.

[0553] Step 4:

[0554] The server inputs the preprocessed audio data into a generative AI model.

[0555] The generative AI model analyzes the emotion of the bird's cry from the audio data and determines it to be "anxiety."

[0556] Step 5:

[0557] Based on the analysis results, the server creates a natural language message saying, "The dog is feeling anxious."

[0558] Additionally, it generates virtual cries that indicate anxiety.

[0559] Step 6:

[0560] The server sends the generated message and sound to the device.

[0561] Step 7:

[0562] The terminal receives the data from the server.

[0563] The device notifies the user of the received data via the app.

[0564] Step 8:

[0565] The user opens the app and sees the analysis results provided and the generated sounds.

[0566] Example 2: Cat image analysis

[0567] Step 1:

[0568] Users take photos of their cats using a dedicated app.

[0569] The device temporarily stores the captured data.

[0570] Step 2:

[0571] The device sends the photo data to a server via an Internet connection.

[0572] The server receives the photo data and stores it in a database.

[0573] Step 3:

[0574] The server preprocesses the received image data into an analyzable format.

[0575] Performs format conversion such as adjusting resolution and color.

[0576] Step 4:

[0577] The server inputs the preprocessed image data into the generative AI model.

[0578] The generative AI model analyzes the cat's facial expression from image data and determines its "stress."

[0579] Step 5:

[0580] Based on the analysis results, the server creates a natural language message saying, "The cat is feeling stressed."

[0581] Furthermore, a virtual facial expression image showing stress is generated.

[0582] Step 6:

[0583] The server sends the generated message and image to the terminal.

[0584] Step 7:

[0585] The terminal receives the data from the server.

[0586] The device notifies the user of the received data via the app.

[0587] Step 8:

[0588] The user opens the app and checks the analysis results provided and the generated facial expression image.

[0589] In this way, the specific processing of the system is carried out to analyze the pet's condition in detail through each step and provide the results to the owner.

[0590] Example 1

[0591] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0592] In recent years, there has been an increasing demand for pet health management and emotional understanding, but it is often difficult for pet owners to accurately grasp their pet's condition. With current technology, analyzing a pet's cries and facial expressions requires a high level of specialized knowledge, making it difficult for the average pet owner to access. To solve this problem, a system that allows owners to easily and intuitively understand their pet's condition is needed.

[0593] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0594] In this invention, the server includes means for transmitting data recorded or photographed by a user's device through a dedicated app, means for the server to receive the data and perform preprocessing such as noise removal and format conversion, means for determining emotion labels and health status using a generative AI model based on the preprocessed data, and means for generating virtual pet cries and facial expression images based on the analysis results, thereby enabling owners to easily understand their pet's emotions and health status through the dedicated app.

[0595] The "data receiving means" is a means for transmitting sounds, images, and videos of pets from the user's terminal to the server.

[0596] The "data preprocessing means" is a means for converting the data received by the server into an analyzable format, and includes processes such as noise removal, format conversion, and resizing.

[0597] "Analysis means" means a means for determining the emotions and health status of a pet using a generative AI model based on pre-processed data.

[0598] The "message generating means" is a means for generating a message that explains the state of the pet in natural language based on the analysis results.

[0599] The "transmission means" is a means for transmitting the generated message and the sound and image of the virtual pet to the user's terminal.

[0600] A "dedicated app" is an application program that users use to record and take pictures and videos of their pets and send them to a server.

[0601] "Noise reduction" is the process of removing unnecessary background sounds from audio data.

[0602] "Format conversion" refers to the process of converting data into a format that is easy to analyze.

[0603] A "generative AI model" is an artificial intelligence model that determines a pet's emotional label and health condition based on preprocessed data and outputs the results.

[0604] An "emotion label" is an emotion category (such as "joy" or "anxiety") that is inferred from a pet's sounds and behavior.

[0605] "Health status" refers to the health status (such as "stress" or "health") estimated from image and video data of a pet.

[0606] "Natural language" refers to words and sentences that humans use on a daily basis, and is used in this system to convey analysis results to users in an intuitively easy-to-understand format.

[0607] "Virtual pet cries" refer to the artificially generated cries of pets based on the analysis results.

[0608] A "virtual facial expression image" is an image of a pet's facial expression that is artificially generated based on the analysis results.

[0609] "User's terminal" refers to an electronic device (smartphone, tablet, etc.) used by a user on which a dedicated app is installed.

[0610] This invention relates to a system that allows users to analyze their pet's cries, images, and videos to more accurately and intuitively understand their pet's emotions and health condition. This system uses the user's device, a server, and a generative AI model to collect and analyze data and provide the results to the user.

[0611] Hardware and Software

[0612] Hardware:

[0613] Server: A central device that receives, stores, and analyzes data such as bird sounds, images, and videos.

[0614] Device: An electronic device (smartphone, tablet, etc.) used by a user to record and capture sounds, images, and videos of pets and send them to a server.

[0615] software:

[0616] Dedicated app: An application installed by the user that records and photographs data, sends it to the server, and receives and displays the analysis results.

[0617] OpenSMILE: Software used for noise reduction and preprocessing of audio data.

[0618] OpenCV: Software that performs preprocessing such as resizing and format conversion of image data.

[0619] Generative AI model: An artificial intelligence model (e.g., GPT-4, ResNet) that analyzes voice data and assigns emotion labels (happiness, anxiety, etc.), or analyzes image data and determines health status (stress, health, etc.).

[0620] Server application: A program that runs on the server and manages data reception, preprocessing, analysis, and transmission of generated data.

[0621] Program flow

[0622] 1. Data collection and transmission

[0623] Users use a dedicated app to record and take pictures and videos of their pets' cries.

[0624] The device sends the recorded and photographed data to a server using Wi-Fi or mobile data networks.

[0625] 2. Data Preprocessing

[0626] The server loads the received data, performs noise reduction on the audio data, and resizes and converts the image data. OpenSMILE is used for preprocessing the audio data, and OpenCV is used for preprocessing the image data.

[0627] 3. Data Analysis

[0628] The server inputs the preprocessed data into a generative AI model, determining emotion labels for bird sounds and health status for image data. GPT-4 and ResNet are used for analysis.

[0629] 4. Message Creation

[0630] Based on the analysis results, the server generates a natural language message explaining the pet's condition. It also generates the virtual pet's cries and facial expressions. It uses GPT-4 for message generation, Tacotron for voice generation, and DALL-E for image generation.

[0631] 5. Transmission and Notification

[0632] The server transmits the generated message and virtual data to the terminal.

[0633] The device notifies the user of the received data and allows them to check the analysis results using a dedicated app.

[0634] Specific examples

[0635] Example 1: Dog bark analysis

[0636] Users record their dog's barking using a dedicated app.

[0637] The device sends the recorded data to the server.

[0638] The server receives the audio data and uses OpenSMILE to perform noise reduction.

[0639] The server analyzes the sound using a generative AI model and determines that it is "anxious" (model used: GPT-4).

[0640] The server generates a message saying "The dog is anxious" and the Tacotron generates a virtual anxious bark.

[0641] The terminal receives the data from the server and notifies the user.

[0642] Prompt Sentence Examples

[0643] Enter your dog's bark and analyze its emotions.

[0644] Input audio: Woof woof woof (audio data)

[0645] Example 2: Cat image analysis

[0646] Users take photos of their cats using a dedicated app.

[0647] The terminal transmits the image data to the server.

[0648] The server receives the image data and uses OpenCV to resize and convert the format.

[0649] The server analyzes the image using a generative AI model and determines that it is "stressed" (model used: ResNet).

[0650] The server generates a message saying "The cat is feeling stressed," and DALL-E generates an image of a facial expression showing stress.

[0651] The terminal receives the data from the server and notifies the user.

[0652] Prompt Sentence Examples

[0653] Enter a photo of your cat and rate its health.

[0654] Input image: (image data of a cat)

[0655] This system allows users to quickly and accurately understand the condition of their pet and take appropriate action.

[0656] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0657] Step 1:

[0658] Users use a dedicated app to record and take pictures and videos of their pets' cries.

[0659] Specifically, the user presses the app's "record" button to record the dog's bark.

[0660] Input: Pet sounds, images, videos

[0661] Output: Audio, image, and video files stored on the user's device

[0662] Step 2:

[0663] The device sends the recorded and photographed data to the server.

[0664] Specifically, the device uses Wi-Fi or a cellular data network to send data to the server as an HTTP POST request.

[0665] Input: User recorded and photographed data

[0666] Output: Data uploaded to the server

[0667] Step 3:

[0668] The server stores the received data.

[0669] Specifically, the server receives the request and stores it in a database. For example, it stores the audio file in cloud storage and the metadata in a database.

[0670] Input: Data sent from the terminal

[0671] Output: Data stored on the server

[0672] Step 4:

[0673] The server pre-processes the audio data.

[0674] Specifically, OpenSMILE is used to remove noise from audio data.

[0675] Input: Stored audio data

[0676] Output: Denoised audio data

[0677] Step 5:

[0678] The server pre-processes the image data.

[0679] Specifically, OpenCV is used to resize and convert the format of image data.

[0680] Input: Saved image data

[0681] Output: Resized and formatted image data

[0682] Step 6:

[0683] The server inputs the preprocessed data into the generative AI model for analysis.

[0684] Specifically, it uses pre-trained generative AI models (e.g., GPT-4 and ResNet) to determine emotion labels from audio data and health status from image data.

[0685] Input: Preprocessed audio data, image data

[0686] Output: Emotion label (e.g., happy, anxious), health status (e.g., stress, healthy)

[0687] Step 7:

[0688] The server generates a message based on the analysis results.

[0689] Specifically, it uses a generative AI model (e.g., GPT-4) to generate analysis results as natural language messages, as well as virtual pet sounds and facial expressions.

[0690] Input: emotion label, health status

[0691] Output: Natural language message (e.g., the dog is anxious), virtual barks and facial expressions

[0692] Step 8:

[0693] The server sends the generated message and data to the terminal.

[0694] As a specific operation, an HTTP response including the generated message and data is created and sent to the terminal.

[0695] Input: Natural language messages, virtual sounds and facial expressions

[0696] Output: Messages and data sent to the terminal

[0697] Step 9:

[0698] The terminal notifies the user of the received data.

[0699] Specifically, the terminal uses a notification function to notify the user that "there is a new message."

[0700] Input: Message and data sent from the server

[0701] Output: User notification

[0702] Step 10:

[0703] Users can check the notifications through the app and understand their pet's condition.

[0704] Specifically, the user opens the app and views detailed messages and virtual data.

[0705] Input: Notification from device

[0706] Output: Analysis results and virtual data displayed through the app

[0707] This series of processes allows the user to quickly and accurately grasp the condition of the pet and take appropriate action.

[0708] (Application example 1)

[0709] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0710] Pet owners need specialized knowledge to understand their pets' emotions and health conditions, and this can lead to inadequate care. Pet shops and retailers are also required to quickly and easily assess the condition of their pets and provide appropriate advice, but there is currently no efficient way to meet this need.

[0711] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0712] In this invention, the server includes a data receiving means for analyzing the pet's cries, images, and videos, a data preprocessing means for preprocessing the received data into an analyzable format, an analysis means for determining the pet's emotions and health condition using a generative AI model based on the preprocessed data, a message generating means for generating an explanatory message in natural language based on the analysis results, a transmission means for transmitting the generated message and the generated pet's cries and images to a user's terminal, and an information output means for providing pet care information within a physical retail environment accessed by the user. This makes it possible to intuitively and quickly grasp the pet's emotions and health condition and efficiently provide appropriate care and product recommendations.

[0713] The "data receiving means" is a means by which a user can send sounds, images, and videos of their pet to the server via the terminal.

[0714] The "data preprocessing means" is a means for converting received data into an analyzable format, and performs noise removal, resizing, and format conversion.

[0715] "Analysis means" refers to a means for determining a pet's emotions and health status using a generative AI model based on pre-processed data.

[0716] The "message generation means" is a means for generating an explanatory message in natural language based on the analysis results, to clearly communicate the pet's condition.

[0717] The "transmission means" is a means for transmitting the generated message and the sound and image of the pet to the user's terminal.

[0718] "Information output means" refers to means for providing pet care information within the physical retail environment accessed by the user, and includes displays, audio guidance devices, and the like.

[0719] The present invention is a system for understanding the condition of pets and providing appropriate care, and is effective in pet shops and physical retail environments. Specific embodiments for implementing the system will be described below.

[0720] 1. System Configuration

[0721] This system consists of the following elements:

[0722] Data Receiving Method

[0723] Users use their smartphones or other devices to record and take pictures and videos of their pets, and then send the data to a server, where it is collected and analyzed in a central system.

[0724] Data preprocessing measures

[0725] The server converts the received data into an analyzable format. Specific processing includes noise reduction for audio data and resizing and format conversion for image and video data. This processing allows for efficient data analysis.

[0726] Analysis means

[0727] The server uses a generative AI model to analyze the preprocessed data. For example, if it is audio data, it can determine the emotion of the animal's voice (such as "joy" or "anxiety"), and if it is image or video data, it can determine the pet's health status (such as "stress" or "health"). This analysis uses libraries such as TensorFlow and OpenCV.

[0728] Message Generation Method

[0729] Based on the analysis results, the server generates a message in natural language explaining the pet's condition, and then uses generative AI to create virtual sounds and facial expressions, using OpenAI's GPT-3 or GPT-4.

[0730] Transmission method

[0731] The server sends the generated message, along with the pet's sounds and images, to the user's device, who receives this information through a dedicated app.

[0732] Information output means

[0733] In the physical retail environment accessed by the user (e.g., a pet shop), displays and robots installed in the store display the analysis results and provide customers with information on the condition of their pets and care advice.

[0734] 2. Hardware and Software Used

[0735] Hardware: Smartphones (iOS / Android), in-store robots, cloud servers (AWS, Google Cloud, etc.)

[0736] Software: Flask (a Python web application framework), TensorFlow (a machine learning framework), OpenCV (an image analysis library), and OpenAI's GPT-3 or GPT-4

[0737] 3. Specific Examples

[0738] Example 1: Pet emotion analysis by in-store robot

[0739] A robot installed inside a pet shop will record the sounds and images of customers' pets.

[0740] The robot sends the data to a server and receives the analysis results, which provide advice such as, "This dog seems a little anxious. Why don't you try a soothing toy?"

[0741] Based on the information provided in the app, pet store staff will recommend the right products.

[0742] Example 2: Checking health status using a smartphone app

[0743] The user takes a photo of their pet using a smartphone app and sends the data to the server.

[0744] The server analyzes the image and determines that the cat is stressed. Along with the analysis result, it generates an image of the cat's facial expression that indicates stress.

[0745] Users will receive notifications in the app and can check the details to provide appropriate care.

[0746] 4. Examples of prompts

[0747] An example prompt has the following format:

[0748] Generate a message to communicate the pet's status:

[0749] Take your pet's sound, image, or any media and explain the analysis results in natural language, for example, "My dog ​​is feeling anxious."

[0750] This system makes it possible to intuitively and quickly grasp a pet's emotional and health status, and efficiently provide appropriate care and product recommendations.

[0751] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0752] Step 1:

[0753] Users can use their smartphones or in-store robots to record and capture sounds, images, and videos of their pets, and then send the data. The input includes sounds, images, and video data of the pets. This data is then sent from the device to the server.

[0754] Step 2:

[0755] The server receives the data sent from the user using the data receiving means. The input includes the pet's cry, image, and video data sent in step 1. The received data is stored on the server.

[0756] Step 3:

[0757] The server preprocesses the received data using the data preprocessing means. Specifically, it performs noise removal on audio data, resizing and format conversion on image and video data. The input is the data received in step 2, and the output is the data after noise removal and resizing processing.

[0758] Step 4:

[0759] The server uses analytical means to analyze the preprocessed data with a generative AI model. For audio data, the emotion of the animal's cry (such as "joy" or "anxiety") is determined, and for images and videos, the pet's health status (such as "stress" or "health") is determined. The input is the preprocessed data, and the output is the analysis results, such as an emotion label or a health status determination.

[0760] Step 5:

[0761] Based on the analysis results, the server uses a message generation means to generate a message explaining the pet's condition in natural language. Furthermore, it uses a generation AI to create the virtual pet's cries and facial expressions. The input is the analysis results, and the output is the generated natural language message and the virtual pet's cries and facial expressions.

[0762] Step 6:

[0763] The server transmits the generated message and the generated pet sound and image to the user's terminal using a transmission means. The input is the message and image generated in step 5, and the output is a notification to the user's terminal.

[0764] Step 7:

[0765] The user receives messages and images sent from the server through a device (smartphone or in-store robot) and checks the status of their pet. The input is the sent message and image, and the output is the user's understanding and execution of appropriate action.

[0766] Step 8:

[0767] Using information output means, pet care information is provided in the physical retail environment (pet shop) accessed by the user. The input is analysis results and messages sent from the server, and the output is information provided by in-store displays and robots. Users and staff use this information to provide pet care and product recommendations.

[0768] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0769] This invention relates to a system that analyzes pet sounds, images, and videos, and combines them with an emotion engine that recognizes the user's emotions to provide information to pet owners in an easy-to-understand format. This system receives data sent from the user's device, preprocesses it, and uses a generative AI model to analyze the pet and user's emotions, and provides the results to the owner.

[0770] System Configuration

[0771] 1. Data Receiving Method

[0772] The user uses a dedicated app to record and take pictures and videos of their pets, and the device sends the data to the server. This data receiving means is the part where the server receives and stores the data.

[0773] 2. Data preprocessing methods

[0774] The server converts the received data into an analyzable format. Specifically, it performs noise reduction on audio data and resizes and converts the format of image and video data. This data preprocessing method enables efficient analysis.

[0775] 3. Analysis method

[0776] The server uses a generative AI model to analyze the preprocessed data. For example, if it is audio data, it will assign an emotional label to the sound of the animal's bark (such as "joy" or "anxiety"), and if it is image or video data, it will determine the pet's health condition (such as "stress" or "health"). This analysis method allows for a detailed understanding of the pet's condition.

[0777] 4. Emotion Engine

[0778] The server separately receives the user's camera footage and audio, and uses an emotion engine to analyze the user's emotions. The analysis results are displayed as emotion labels such as "happiness," "sadness," and "surprise." This allows the owner's psychological state to be understood at the same time.

[0779] 5. Message Generation Methods

[0780] Based on the analysis results, the server generates a message that comprehensively explains the status of the pet and the user. Furthermore, it uses a generation AI to create virtual pet sounds and facial expressions. This message generation method makes it easier for owners to intuitively understand the status of their pet and themselves.

[0781] 6. Transmission Method

[0782] The server sends the generated message and the generated pet's sound and image to the device, which then notifies the user via the app, allowing the user to quickly update the status of their pet and themselves.

[0783] Specific Examples

[0784] Example 1: Dog barking and user emotion analysis

[0785] Users record their dog's bark using a dedicated app and capture their own facial expressions using the camera.

[0786] The terminal transmits the recording data and image data to the server.

[0787] The server receives the audio data and image data and performs noise removal and preprocessing.

[0788] The server analyzes the sound using a generative AI model and determines it as "anxiety." At the same time, it analyzes the user's emotions using an emotion engine and determines it as "surprise."

[0789] The server generates a message saying "The dog is anxious and the user is surprised" and generates a virtual anxious cry and an image of a surprised expression.

[0790] The terminal receives the data from the server and notifies the user.

[0791] The user checks the notification in the app and sees the analysis results and generated data.

[0792] Example 2: Cat images and user sentiment analysis

[0793] Users take photos of their cats using a dedicated app and then use the camera to capture their own images.

[0794] The terminal transmits the photo data and image data to the server.

[0795] The server receives the photo data and image data and performs any necessary pre-processing.

[0796] The server analyzes the image using a generative AI model and determines whether it is "stressed." At the same time, it analyzes the user's emotions using an emotion engine and determines whether it is "joy."

[0797] The server generates a message saying "The cat is stressed, and the user is happy," and generates an image of a stressed expression and an image of a happy expression.

[0798] The terminal receives the data from the server and notifies the user.

[0799] The user checks the notification in the app and sees the analysis results and generated data.

[0800] In this way, the system of the present invention provides an easy-to-understand picture of the pet and user's condition, making it easier to meet the pet's needs and improving the happiness and health of both the pet and the owner.

[0801] The processing flow will be explained below.

[0802] Example 1: Dog barking and user emotion analysis

[0803] Step 1:

[0804] Users use a dedicated app to record their dog's barking.

[0805] At the same time, the user captures their own facial expressions using the app's camera function.

[0806] Step 2:

[0807] The device temporarily stores the audio recording data and facial expression image data.

[0808] Step 3:

[0809] The device transmits the audio recording data and facial expression image data to a server via an internet connection.

[0810] The server receives the audio recording data and facial expression image data and stores them in a database.

[0811] Step 4:

[0812] The server performs noise reduction on the received audio data.

[0813] The server adjusts the layout and converts the format of the image data.

[0814] Step 5:

[0815] The server inputs the preprocessed audio data into a generative AI model.

[0816] The generative AI model determines the emotion of a dog's bark from audio data as "anxiety."

[0817] Step 6:

[0818] The server inputs the preprocessed facial expression image data into the emotion engine.

[0819] The emotion engine determines the user's emotion as "surprise" from the facial expression image.

[0820] Step 7:

[0821] Based on the analysis results, the server generates a message saying, "The dog is feeling anxious and the user is surprised."

[0822] In addition, it generates virtual cries that indicate anxiety and images of surprised facial expressions.

[0823] Step 8:

[0824] The server sends the generated message, the bird's cry, and an image of its facial expression to the device.

[0825] Step 9:

[0826] The terminal receives the data from the server.

[0827] The device notifies the user of the received data via the app.

[0828] Step 10:

[0829] The user checks the notification in the app and sees the analysis results provided and the generated sound and facial expression images.

[0830] Example 2: Cat images and user sentiment analysis

[0831] Step 1:

[0832] Users take photos of their cats using a dedicated app.

[0833] At the same time, the user captures an image of themselves using the app's camera function.

[0834] Step 2:

[0835] The device temporarily stores the photograph data and facial expression image data.

[0836] Step 3:

[0837] The device transmits the photograph data and facial expression image data to a server via an internet connection.

[0838] The server receives the photograph data and facial expression image data and stores them in a database.

[0839] Step 4:

[0840] The server adjusts the layout and corrects the color of the image data received.

[0841] Step 5:

[0842] The server inputs the preprocessed image data into the generative AI model.

[0843] The generative AI model determines from image data that the cat's facial expression is "stressed."

[0844] Step 6:

[0845] The server inputs the preprocessed facial expression image data into the emotion engine.

[0846] The emotion engine determines the user's emotion as "joy" from the facial expression image.

[0847] Step 7:

[0848] Based on the analysis results, the server generates a message saying, "The cat is stressed, and the user is happy."

[0849] Furthermore, virtual facial expression images showing stress and happy expressions are generated.

[0850] Step 8:

[0851] The server sends the generated message and image to the terminal.

[0852] Step 9:

[0853] The terminal receives the data from the server.

[0854] The device notifies the user of the received data via the app.

[0855] Step 10:

[0856] The user checks the notification in the app and sees the analysis results and generated facial expression images.

[0857] In this way, the system of the present invention provides an easy-to-understand picture of the pet and user's condition, making it easier to meet the pet's needs and improving the happiness and health of both the pet and the owner.

[0858] Example 2

[0859] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0860] Previous systems that analyzed pet sounds, images, and videos were limited to understanding the pet's emotions and health status, but were unable to consider the owner's emotional state. This made it difficult to comprehensively understand the relationship between pet and owner, and meant that the systems were unable to adequately meet the pet's needs.

[0861] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a data receiving means for analyzing the pet's cries, images, and videos, a data preprocessing means for preprocessing the received data into an analyzable format, an analysis means for determining the pet's emotions and health condition using a generative AI model based on the preprocessed data, a means for receiving the user's camera video and audio and analyzing the user's emotions using an emotion engine, a message generating means for generating an explanatory message in natural language based on the analysis results, and a transmission means for transmitting the generated message and the generated pet's cries and images to the user's terminal. This makes it possible to comprehensively analyze the emotional states of the pet and the user and provide information to the owner in a format that is easy to understand.

[0862] The "data receiving means" is a mechanism for receiving pet sounds, images, and videos from the user's device.

[0863] The "data preprocessing means" is a mechanism that performs noise removal, resizing, and format conversion to convert received data into an analyzable format.

[0864] A "generative AI model" is a pre-trained artificial intelligence algorithm used to determine a pet's emotions and health status.

[0865] The "analysis means" is a mechanism that uses a generative AI model based on pre-processed data to determine the pet's emotions and health status.

[0866] "Means for receiving camera footage and audio from a user and analyzing the user's emotions using an emotion engine" is a mechanism for receiving camera footage and audio from a user and analyzing the user's emotions using an emotion engine.

[0867] The "message generation means" is a mechanism for generating an explanatory message in natural language based on the analysis results of the pet and the user.

[0868] The "transmission means" is a mechanism for transmitting the generated message and the generated pet's cry and image to the user's terminal.

[0869] This invention relates to a system that analyzes pet sounds, images, and videos, and combines them with an emotion engine that recognizes the user's emotions to provide information to pet owners in an easy-to-understand format. This system receives data sent from the user's device, preprocesses it, and uses a generative AI model to analyze the emotions of the pet and the user, and provides the results to the owner.

[0870] System Configuration

[0871] Data Receiving Method

[0872] The user uses a dedicated app to record and take pictures and videos of their pets, and the device sends the data to the server. This data receiving means is the part where the server receives and stores the data.

[0873] Data preprocessing measures

[0874] The server converts the received data into an analyzable format. Specifically, it performs noise removal on audio data and resizes and converts the format of image and video data. As data preprocessing methods, it uses FFT-based filtering for audio processing and the OpenCV library for image processing.

[0875] Analysis means

[0876] The server analyzes the preprocessed data using a generative AI model. TensorFlow or PyTorch are used for the generative AI model. For example, if the data is audio, it can assign an emotional label to the sound of the animal's bark (such as "joy" or "anxiety"), and if the data is image or video, it can determine the pet's health status (such as "stress" or "health"). This allows for a detailed understanding of the pet's condition.

[0877] Emotion Engine

[0878] The server separately receives the user's camera footage and audio, and uses an emotion engine to analyze the user's emotions. The analysis results are displayed as emotion labels such as "happiness," "sadness," and "surprise." The emotion engine uses Microsoft Azure Cognitive Services, which allows the owner's psychological state to be grasped at the same time.

[0879] Message Generation Method

[0880] Based on the analysis results, the server generates a message that comprehensively describes the status of the pet and the user. A natural language generation engine (e.g., GPT-4) is used to generate the message. Generative AI is also used to create virtual pet sounds and facial expressions. The DALL-E model is used to generate the virtual data.

[0881] Transmission method

[0882] The server sends the generated message and the generated pet's sound and image to the device, which then notifies the user via the app, allowing the user to quickly update the status of their pet and themselves.

[0883] Specific Examples

[0884] Example 1: Dog barking and user emotion analysis

[0885] Users record their dog's bark using a dedicated app and capture their own facial expressions using the camera.

[0886] The terminal transmits the recording data and image data to the server.

[0887] The server receives the audio data and performs noise reduction, and resizes the image data.

[0888] The server analyzes the sound using a generative AI model and determines it as "anxiety." At the same time, it analyzes the user's emotions using an emotion engine and determines it as "surprise."

[0889] The server generates a message saying "The dog is anxious and the user is surprised" and generates a virtual anxious cry and an image of a surprised expression.

[0890] The terminal receives the data from the server and notifies the user.

[0891] The user can check the analysis results and generated data in the app.

[0892] Example 2: Cat images and user sentiment analysis

[0893] Users take photos of their cats using a dedicated app and then use the camera to capture their own images.

[0894] The terminal transmits the photo data and image data to the server.

[0895] The server receives the photo data and image data and performs any necessary pre-processing.

[0896] The server analyzes the image using a generative AI model and determines whether it is "stressed." At the same time, it analyzes the user's emotions using an emotion engine and determines whether it is "joy."

[0897] The server generates a message saying "The cat is stressed, and the user is happy," and generates an image of a stressed expression and an image of a happy expression.

[0898] The terminal receives the data from the server and notifies the user.

[0899] The user checks the notification in the app and sees the analysis results and generated data.

[0900] In this way, the system of the present invention provides an easy-to-understand picture of the pet and user's condition, making it easier to meet the pet's needs and improving the happiness and health of both the pet and the owner.

[0901] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0902] Step 1: Getting and sending data

[0903] The user launches a dedicated app installed on their smartphone. They tap the record button in the app to record their pet's cries, or tap the camera button to take photos or videos of their pet. When recording or taking a photo, the user's facial expressions are also captured by the camera.

[0904] Input: Pet sounds, images, videos, user face images

[0905] Output: Audio files, image files, video files, user face image data

[0906] The terminal sends this data to the server using the HTTP protocol.

[0907] Step 2: Receiving and storing data

[0908] The server receives data sent from the device via a RESTful API. The received data is stored in local storage or cloud storage (e.g., AWS S3). Once stored, metadata is recorded in a database (e.g., MySQL) for data management.

[0909] Input: Audio files, image files, video files, user face image data

[0910] Output: Saved files, metadata

[0911] Step 3: Preprocessing the data

[0912] The server picks up the stored audio data and applies noise reduction algorithms (e.g., FFT-based filtering). For image and video data, it resizes and converts the format using the OpenCV library. Specifically, images are resized to 640x480 pixels and videos are encoded in MP4 format.

[0913] Input: saved audio files, image files, video files

[0914] Output: Pre-processed audio data, resized and formatted image and video data

[0915] Step 4: Analyze the data

[0916] The server inputs the preprocessed data into a generative AI model (e.g., TensorFlow model) for analysis. An emotion analysis model is applied to the audio data to assign labels such as "joy," "anxiety," and "anger," and an object detection model is applied to the image and video data to determine the pet's health status (e.g., "stressed," "relaxed," etc.).

[0917] Input: Preprocessed audio data, image and video data

[0918] Output: Audio data with emotion labels, image and video data with health status labels

[0919] Step 5: Analyzing user sentiment

[0920] The server receives the user's camera footage and audio data and analyzes it using an emotion engine (e.g., Microsoft Azure Cognitive Services). As a result, it obtains the user's emotion label (e.g., "joy," "sadness," "surprise," etc.).

[0921] Input: User's face image data, voice data

[0922] Output: User data with emotion labels

[0923] Step 6: Message Generation

[0924] The server generates messages using a natural language generation engine (e.g., GPT-4) based on the analysis results of the pet and the user. It also uses generative AI to create virtual pet sounds and facial expressions. The DALL-E model is used to generate the virtual data.

[0925] Input: Analysis results of pet and user (emotion labels, health status labels, etc.)

[0926] Output: Generated natural language messages, virtual sounds and facial expressions

[0927] Step 7: Data submission and notification

[0928] The server sends the generated message and virtual data to the API endpoint of the dedicated app.

[0929] The device receives the data from the server, stores it locally, and displays a notification to the user through the app. When the user opens the app, they can view the analysis results and the generated virtual data.

[0930] Input: Generated natural language messages, virtual sounds, and facial expressions

[0931] Output: Notification to user terminal, display of analysis results and generated data

[0932] This system can comprehensively analyze the emotional states of both the pet and the user, and provide information in a format that is easy for the owner to understand.

[0933] (Application example 2)

[0934] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0935] Pet owners spend a lot of time and effort understanding their pets' emotions and health conditions and providing appropriate care. It is also difficult to find products and services that suit their pets' conditions. As a result, pet needs cannot be met quickly and accurately, which can lead to a decline in the happiness and health of both the pet and the owner. To solve this problem, a system is needed that analyzes the conditions of both the pet and the owner and suggests appropriate products and services based on that analysis.

[0936] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data receiving means for analyzing the pet's cries, images, and videos; a data preprocessing means for preprocessing the received data into an analyzable format; an analysis means for determining the pet's emotions and health condition using a generative AI model based on the preprocessed data; a proposal means for proposing products and services based on the pet's condition and the user's emotions determined by the analysis means; a message generation means for generating an explanatory message in natural language based on the analysis results; and a transmission means for transmitting the generated message and the generated pet's cries and images to the user's terminal. This allows the owner to quickly understand the pet's condition and easily select and purchase appropriate products and services based on that information.

[0937] The "data receiving means" is a device or system for receiving data related to the sounds, images, and videos of pets, as well as the user's emotions.

[0938] "Data preprocessing means" refers to processing devices and algorithms that convert received data into an analyzable format. This means performs preprocessing such as noise removal and format conversion.

[0939] "Analysis means" refers to a device or system that uses a generative AI model to determine the pet's emotions and health status based on pre-processed data.

[0940] The "suggestion means" is a device or system that suggests appropriate products or services based on the state of the pet and the user's emotions determined by the analysis means.

[0941] The "message generating means" is a device or system that generates a message explaining the state of the pet and the user in natural language based on the analysis results.

[0942] The "transmission means" is a device or system that transmits the generated message and the sounds and images of the pet to the user's terminal.

[0943] A "generative AI model" is an artificial intelligence model used to analyze a pet's emotions and health condition from received data.

[0944] "Natural language" refers to a language format that humans use on a daily basis, and is used to provide analysis results to users in an intuitively understandable format.

[0945] "Products and services" is a general term for items and support activities suggested depending on the pet's condition and the user's emotions.

[0946] The present invention provides a system for analyzing the state of a pet and the emotions of a user, and for proposing appropriate products and services based on the results. This system has the following configuration.

[0947] First, the user installs a dedicated application on their smartphone and captures their pet's cries, images, and videos. This activates the data receiving means, which then transmits the captured data to the server. This data receiving means includes a standard smartphone camera and microphone.

[0948] Next, the server preprocesses the received data using a data preprocessing means. This preprocessing means includes, for example, noise removal from audio data, resizing of image data, and format conversion. This process typically uses a programming language such as Python, with libraries such as OpenCV or Librosa.

[0949] Once preprocessed, the data is analyzed using a generative AI model. The analysis methods include analyzing pet sounds to determine emotions, and analyzing images and videos to determine health and stress levels. The generative AI model is built using deep learning frameworks such as TensorFlow and PyTorch.

[0950] Furthermore, the analysis method analyzes not only the pet's condition but also the user's emotions. The user captures their own facial expressions with their smartphone camera, and the data is sent to the server. The emotion engine determines the user's emotions based on this data. This analysis uses Facial Recognition API and natural language processing engines (NLP engines), among others.

[0951] Based on the analysis results, the suggestion unit operates to suggest products and services according to the condition of the pet and the user. These suggestions include items to reduce stress for pets and relaxation items for owners.

[0952] The message generation means then generates a message in natural language based on the analysis results. For example, "If your pet is feeling stressed, why not try purchasing some relaxation products?" This generated message, along with the sound and image of the pet, is then sent back to the user's smartphone via the transmission means. The user receives this as a notification and can check it on the application.

[0953] For example, the following prompts might be fed into a generative AI model:

[0954] Output analysis results based on the pet's image and the user's emotion. For example, if the pet is stressed and the user is surprised, return information including the emotion label. Also, suggest products (such as relaxation goods) appropriate for the pet's condition.

[0955] This system allows users to quickly understand the condition of their pets and easily select and purchase appropriate products and services based on that information, ultimately improving the happiness and health of both pets and their owners.

[0956] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0957] Step 1:

[0958] The user uses a smartphone to capture the sound, images, and videos of their pet, and then sends the data to the server through a dedicated application. The input data is the sound, images, and videos of the pet, and the output is the raw data sent to the server. This step uses the smartphone's built-in camera and microphone.

[0959] Step 2:

[0960] The server receives data sent by the user using a data receiving means. The input data is the unprocessed data sent from the smartphone, and the output is the received data. In this step, communication means such as HTTP requests and WebSockets are used.

[0961] Step 3:

[0962] The server uses data preprocessing to convert the received data into an analyzable format. The input data is the received raw data, and the output is the preprocessed data. This step involves noise removal for audio data, resizing of image data, and format conversion. Specifically, the Python libraries OpenCV and Librosa are used.

[0963] Step 4:

[0964] The server analyzes the preprocessed data using a generative AI model. The input data is the preprocessed data, and the output is the analysis results showing the pet's emotions and health status. In this step, a generative AI model built using TensorFlow and PyTorch is used. The model assigns emotion labels such as "joy" or "anxiety" to the pet's cries, and determines its state, such as "stress" or "health," from images and videos.

[0965] Step 5:

[0966] The server analyzes the user's emotions using an emotion engine. The input data is the user's facial expression data captured by the smartphone camera, and the output is an analysis result indicating the user's emotions. In this step, the Facial Recognition API and a natural language processing engine (NLP engine) are used.

[0967] Step 6:

[0968] The server runs a suggestion unit that suggests products and services based on the analysis results. The input data are the analysis results of the pet and the user, and the output is a list of suggested products and services. In this step, a recommendation engine is used to select products and services suitable for the user. If the pet is feeling stressed, for example, relaxation goods will be suggested.

[0969] Step 7:

[0970] The server uses a message generation means to generate a message in natural language. The input data is the analysis results and a list of suggestions, and the output is a message to be sent to the user. The generated message will explain the pet's condition and the suggested products and services. In this step, an NLP engine is used to generate an intuitive and easy-to-understand message.

[0971] Step 8:

[0972] The server operates a sending means that sends the generated message and the generated pet sound and image to the user's device. The input data is the generated message and the pet sound and image, and the output is a notification that arrives on the user's smartphone. In this step, a push notification service (e.g., Firebase Cloud Messaging) is used.

[0973] Step 9:

[0974] The user checks the notification through the application and views the analysis results and suggested products and services. The input data is the notification sent from the server, and the output is the information provided to the user. In this step, a dedicated application on the smartphone is used, allowing the user to easily check the information and select and purchase the suggested products and services.

[0975] By linking these processing steps, the user can quickly grasp the condition of their pet and provide appropriate care based on that information.

[0976] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0977] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0978] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0979] [Third embodiment]

[0980] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0981] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0982] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0983] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0984] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0985] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0986] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0987] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0988] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0989] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0990] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0991] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0992] This invention relates to a system that analyzes pet sounds, images, and videos and provides information to pet owners in an easy-to-understand format. This system receives data sent from a user's device, preprocesses it, analyzes it using a generative AI model, and provides the results to the pet owner.

[0993] System Configuration

[0994] 1. Data Receiving Method

[0995] The user uses a dedicated app to record and take pictures and videos of their pets, and the device sends the data to the server. This data receiving means is the part where the server receives and stores the data.

[0996] 2. Data preprocessing methods

[0997] The server converts the received data into an analyzable format. Specifically, it performs noise reduction on audio data and resizes and converts the format of image and video data. This data preprocessing method enables efficient analysis.

[0998] 3. Analysis method

[0999] The server uses a generative AI model to analyze the preprocessed data. For example, if it is audio data, it will assign an emotional label to the sound of the animal's bark (such as "joy" or "anxiety"), and if it is image or video data, it will determine the pet's health condition (such as "stress" or "health"). This analysis method allows for a detailed understanding of the pet's condition.

[1000] 4. Message Generation Methods

[1001] Based on the analysis results, the server generates a message in natural language explaining the pet's condition. Furthermore, it uses a generation AI to create virtual pet sounds and facial expressions. This message generation method makes it easier for owners to intuitively understand their pet's condition.

[1002] 5. Transmission Method

[1003] The server sends the generated message and the generated pet's bark and image to the device, which then notifies the user via the app, allowing them to quickly update the pet's status.

[1004] Specific Examples

[1005] Example 1: Dog bark analysis

[1006] Users record their dog's barking using a dedicated app.

[1007] The device sends the recorded data to the server.

[1008] The server receives the audio data and performs noise reduction.

[1009] The server uses a generative AI model to analyze the sound and determine that it is "anxious."

[1010] The server generates a message saying "The dog is anxious" and generates a virtual anxious bark.

[1011] The terminal receives the data from the server and notifies the user.

[1012] Users can check the notification in the app for more details.

[1013] Example 2: Cat image analysis

[1014] Users take photos of their cats using a dedicated app.

[1015] The terminal transmits the image data to the server.

[1016] The server receives the image data and performs any necessary pre-processing.

[1017] The server uses a generative AI model to analyze the image and determine whether it is "stressed."

[1018] The server generates a message saying "The cat is stressed" and an image of a facial expression showing stress.

[1019] The terminal receives the data from the server and notifies the user.

[1020] Users can check the notification in the app for more details.

[1021] In this way, the system of the present invention provides owners with an easy-to-understand view of their pet's condition, making it easier to meet their pet's needs and improving their pet's happiness and health.

[1022] The processing flow will be explained below.

[1023] Example 1: Dog bark analysis

[1024] Step 1:

[1025] Users use a dedicated app to record their dog's barking.

[1026] The device temporarily stores the recorded data.

[1027] Step 2:

[1028] The device sends the recording data to a server via an internet connection.

[1029] The server receives the recording data and stores it in a database.

[1030] Step 3:

[1031] The server performs pre-processing to remove noise from the received audio data.

[1032] The denoised audio data is converted into an analyzable format.

[1033] Step 4:

[1034] The server inputs the preprocessed audio data into a generative AI model.

[1035] The generative AI model analyzes the emotion of the bird's cry from the audio data and determines it to be "anxiety."

[1036] Step 5:

[1037] Based on the analysis results, the server creates a natural language message saying, "The dog is feeling anxious."

[1038] Additionally, it generates virtual cries that indicate anxiety.

[1039] Step 6:

[1040] The server sends the generated message and sound to the device.

[1041] Step 7:

[1042] The terminal receives the data from the server.

[1043] The device notifies the user of the received data via the app.

[1044] Step 8:

[1045] The user opens the app and sees the analysis results provided and the generated sounds.

[1046] Example 2: Cat image analysis

[1047] Step 1:

[1048] Users take photos of their cats using a dedicated app.

[1049] The device temporarily stores the captured data.

[1050] Step 2:

[1051] The device sends the photo data to a server via an Internet connection.

[1052] The server receives the photo data and stores it in a database.

[1053] Step 3:

[1054] The server preprocesses the received image data into an analyzable format.

[1055] Performs format conversion such as adjusting resolution and color.

[1056] Step 4:

[1057] The server inputs the preprocessed image data into the generative AI model.

[1058] The generative AI model analyzes the cat's facial expression from image data and determines its "stress."

[1059] Step 5:

[1060] Based on the analysis results, the server creates a natural language message saying, "The cat is feeling stressed."

[1061] Furthermore, a virtual facial expression image showing stress is generated.

[1062] Step 6:

[1063] The server sends the generated message and image to the terminal.

[1064] Step 7:

[1065] The terminal receives the data from the server.

[1066] The device notifies the user of the received data via the app.

[1067] Step 8:

[1068] The user opens the app and checks the analysis results provided and the generated facial expression image.

[1069] In this way, the specific processing of the system is carried out to analyze the pet's condition in detail through each step and provide the results to the owner.

[1070] Example 1

[1071] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1072] In recent years, there has been an increasing demand for pet health management and emotional understanding, but it is often difficult for pet owners to accurately grasp their pet's condition. With current technology, analyzing a pet's cries and facial expressions requires a high level of specialized knowledge, making it difficult for the average pet owner to access. To solve this problem, a system that allows owners to easily and intuitively understand their pet's condition is needed.

[1073] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1074] In this invention, the server includes means for transmitting data recorded or photographed by a user's device through a dedicated app, means for the server to receive the data and perform preprocessing such as noise removal and format conversion, means for determining emotion labels and health status using a generative AI model based on the preprocessed data, and means for generating virtual pet cries and facial expression images based on the analysis results, thereby enabling owners to easily understand their pet's emotions and health status through the dedicated app.

[1075] The "data receiving means" is a means for transmitting sounds, images, and videos of pets from the user's terminal to the server.

[1076] The "data preprocessing means" is a means for converting the data received by the server into an analyzable format, and includes processes such as noise removal, format conversion, and resizing.

[1077] "Analysis means" means a means for determining the emotions and health status of a pet using a generative AI model based on pre-processed data.

[1078] The "message generating means" is a means for generating a message that explains the state of the pet in natural language based on the analysis results.

[1079] The "transmission means" is a means for transmitting the generated message and the sound and image of the virtual pet to the user's terminal.

[1080] A "dedicated app" is an application program that users use to record and take pictures and videos of their pets and send them to a server.

[1081] "Noise reduction" is the process of removing unnecessary background sounds from audio data.

[1082] "Format conversion" refers to the process of converting data into a format that is easy to analyze.

[1083] A "generative AI model" is an artificial intelligence model that determines a pet's emotional label and health condition based on preprocessed data and outputs the results.

[1084] An "emotion label" is an emotion category (such as "joy" or "anxiety") that is inferred from a pet's sounds and behavior.

[1085] "Health status" refers to the health status (such as "stress" or "health") estimated from image and video data of a pet.

[1086] "Natural language" refers to words and sentences that humans use on a daily basis, and is used in this system to convey analysis results to users in an intuitively easy-to-understand format.

[1087] "Virtual pet cries" refer to the artificially generated cries of pets based on the analysis results.

[1088] A "virtual facial expression image" is an image of a pet's facial expression that is artificially generated based on the analysis results.

[1089] "User's terminal" refers to an electronic device (smartphone, tablet, etc.) used by a user on which a dedicated app is installed.

[1090] This invention relates to a system that allows users to analyze their pet's cries, images, and videos to more accurately and intuitively understand their pet's emotions and health condition. This system uses the user's device, a server, and a generative AI model to collect and analyze data and provide the results to the user.

[1091] Hardware and Software

[1092] Hardware:

[1093] Server: A central device that receives, stores, and analyzes data such as bird sounds, images, and videos.

[1094] Device: An electronic device (smartphone, tablet, etc.) used by a user to record and capture sounds, images, and videos of pets and send them to a server.

[1095] software:

[1096] Dedicated app: An application installed by the user that records and photographs data, sends it to the server, and receives and displays the analysis results.

[1097] OpenSMILE: Software used for noise reduction and preprocessing of audio data.

[1098] OpenCV: Software that performs preprocessing such as resizing and format conversion of image data.

[1099] Generative AI model: An artificial intelligence model (e.g., GPT-4, ResNet) that analyzes voice data and assigns emotion labels (happiness, anxiety, etc.), or analyzes image data and determines health status (stress, health, etc.).

[1100] Server application: A program that runs on the server and manages data reception, preprocessing, analysis, and transmission of generated data.

[1101] Program flow

[1102] 1. Data collection and transmission

[1103] Users use a dedicated app to record and take pictures and videos of their pets' cries.

[1104] The device sends the recorded and photographed data to a server using Wi-Fi or mobile data networks.

[1105] 2. Data Preprocessing

[1106] The server loads the received data, performs noise reduction on the audio data, and resizes and converts the image data. OpenSMILE is used for preprocessing the audio data, and OpenCV is used for preprocessing the image data.

[1107] 3. Data Analysis

[1108] The server inputs the preprocessed data into a generative AI model, determining emotion labels for bird sounds and health status for image data. GPT-4 and ResNet are used for analysis.

[1109] 4. Message Creation

[1110] Based on the analysis results, the server generates a natural language message explaining the pet's condition. It also generates the virtual pet's cries and facial expressions. It uses GPT-4 for message generation, Tacotron for voice generation, and DALL-E for image generation.

[1111] 5. Transmission and Notification

[1112] The server transmits the generated message and virtual data to the terminal.

[1113] The device notifies the user of the received data and allows them to check the analysis results using a dedicated app.

[1114] Specific examples

[1115] Example 1: Dog bark analysis

[1116] Users record their dog's barking using a dedicated app.

[1117] The device sends the recorded data to the server.

[1118] The server receives the audio data and uses OpenSMILE to perform noise reduction.

[1119] The server analyzes the sound using a generative AI model and determines that it is "anxious" (model used: GPT-4).

[1120] The server generates a message saying "The dog is anxious" and the Tacotron generates a virtual anxious bark.

[1121] The terminal receives the data from the server and notifies the user.

[1122] Prompt Sentence Examples

[1123] Enter your dog's bark and analyze its emotions.

[1124] Input audio: Woof woof woof (audio data)

[1125] Example 2: Cat image analysis

[1126] Users take photos of their cats using a dedicated app.

[1127] The terminal transmits the image data to the server.

[1128] The server receives the image data and uses OpenCV to resize and convert the format.

[1129] The server analyzes the image using a generative AI model and determines that it is "stressed" (model used: ResNet).

[1130] The server generates a message saying "The cat is feeling stressed," and DALL-E generates an image of a facial expression showing stress.

[1131] The terminal receives the data from the server and notifies the user.

[1132] Prompt Sentence Examples

[1133] Enter a photo of your cat and rate its health.

[1134] Input image: (image data of a cat)

[1135] This system allows users to quickly and accurately understand the condition of their pet and take appropriate action.

[1136] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1137] Step 1:

[1138] Users use a dedicated app to record and take pictures and videos of their pets' cries.

[1139] Specifically, the user presses the app's "record" button to record the dog's bark.

[1140] Input: Pet sounds, images, videos

[1141] Output: Audio, image, and video files stored on the user's device

[1142] Step 2:

[1143] The device sends the recorded and photographed data to the server.

[1144] Specifically, the device uses Wi-Fi or a cellular data network to send data to the server as an HTTP POST request.

[1145] Input: User recorded and photographed data

[1146] Output: Data uploaded to the server

[1147] Step 3:

[1148] The server stores the received data.

[1149] Specifically, the server receives the request and stores it in a database. For example, it stores the audio file in cloud storage and the metadata in a database.

[1150] Input: Data sent from the terminal

[1151] Output: Data stored on the server

[1152] Step 4:

[1153] The server pre-processes the audio data.

[1154] Specifically, OpenSMILE is used to remove noise from audio data.

[1155] Input: Stored audio data

[1156] Output: Denoised audio data

[1157] Step 5:

[1158] The server pre-processes the image data.

[1159] Specifically, OpenCV is used to resize and convert the format of image data.

[1160] Input: Saved image data

[1161] Output: Resized and formatted image data

[1162] Step 6:

[1163] The server inputs the preprocessed data into the generative AI model for analysis.

[1164] Specifically, it uses pre-trained generative AI models (e.g., GPT-4 and ResNet) to determine emotion labels from audio data and health status from image data.

[1165] Input: Preprocessed audio data, image data

[1166] Output: Emotion label (e.g., happy, anxious), health status (e.g., stress, healthy)

[1167] Step 7:

[1168] The server generates a message based on the analysis results.

[1169] Specifically, it uses a generative AI model (e.g., GPT-4) to generate analysis results as natural language messages, as well as virtual pet sounds and facial expressions.

[1170] Input: emotion label, health status

[1171] Output: Natural language message (e.g., the dog is anxious), virtual barks and facial expressions

[1172] Step 8:

[1173] The server sends the generated message and data to the terminal.

[1174] As a specific operation, an HTTP response including the generated message and data is created and sent to the terminal.

[1175] Input: Natural language messages, virtual sounds and facial expressions

[1176] Output: Messages and data sent to the terminal

[1177] Step 9:

[1178] The terminal notifies the user of the received data.

[1179] Specifically, the terminal uses a notification function to notify the user that "there is a new message."

[1180] Input: Message and data sent from the server

[1181] Output: User notification

[1182] Step 10:

[1183] Users can check the notifications through the app and understand their pet's condition.

[1184] Specifically, the user opens the app and views detailed messages and virtual data.

[1185] Input: Notification from device

[1186] Output: Analysis results and virtual data displayed through the app

[1187] This series of processes allows the user to quickly and accurately grasp the condition of the pet and take appropriate action.

[1188] (Application example 1)

[1189] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1190] Pet owners need specialized knowledge to understand their pets' emotions and health conditions, and this can lead to inadequate care. Pet shops and retailers are also required to quickly and easily assess the condition of their pets and provide appropriate advice, but there is currently no efficient way to meet this need.

[1191] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1192] In this invention, the server includes a data receiving means for analyzing the pet's cries, images, and videos, a data preprocessing means for preprocessing the received data into an analyzable format, an analysis means for determining the pet's emotions and health condition using a generative AI model based on the preprocessed data, a message generating means for generating an explanatory message in natural language based on the analysis results, a transmission means for transmitting the generated message and the generated pet's cries and images to a user's terminal, and an information output means for providing pet care information within a physical retail environment accessed by the user. This makes it possible to intuitively and quickly grasp the pet's emotions and health condition and efficiently provide appropriate care and product recommendations.

[1193] The "data receiving means" is a means by which a user can send sounds, images, and videos of their pet to the server via the terminal.

[1194] The "data preprocessing means" is a means for converting received data into an analyzable format, and performs noise removal, resizing, and format conversion.

[1195] "Analysis means" refers to a means for determining a pet's emotions and health status using a generative AI model based on pre-processed data.

[1196] The "message generation means" is a means for generating an explanatory message in natural language based on the analysis results, to clearly communicate the pet's condition.

[1197] The "transmission means" is a means for transmitting the generated message and the sound and image of the pet to the user's terminal.

[1198] "Information output means" refers to means for providing pet care information within the physical retail environment accessed by the user, and includes displays, audio guidance devices, and the like.

[1199] The present invention is a system for understanding the condition of pets and providing appropriate care, and is effective in pet shops and physical retail environments. Specific embodiments for implementing the system will be described below.

[1200] 1. System Configuration

[1201] This system consists of the following elements:

[1202] Data Receiving Method

[1203] Users use their smartphones or other devices to record and take pictures and videos of their pets, and then send the data to a server, where it is collected and analyzed in a central system.

[1204] Data preprocessing measures

[1205] The server converts the received data into an analyzable format. Specific processing includes noise reduction for audio data and resizing and format conversion for image and video data. This processing allows for efficient data analysis.

[1206] Analysis means

[1207] The server uses a generative AI model to analyze the preprocessed data. For example, if it is audio data, it can determine the emotion of the animal's voice (such as "joy" or "anxiety"), and if it is image or video data, it can determine the pet's health status (such as "stress" or "health"). This analysis uses libraries such as TensorFlow and OpenCV.

[1208] Message Generation Method

[1209] Based on the analysis results, the server generates a message in natural language explaining the pet's condition, and then uses generative AI to create virtual sounds and facial expressions, using OpenAI's GPT-3 or GPT-4.

[1210] Transmission method

[1211] The server sends the generated message, along with the pet's sounds and images, to the user's device, who receives this information through a dedicated app.

[1212] Information output means

[1213] In the physical retail environment accessed by the user (e.g., a pet shop), displays and robots installed in the store display the analysis results and provide customers with information on the condition of their pets and care advice.

[1214] 2. Hardware and Software Used

[1215] Hardware: Smartphones (iOS / Android), in-store robots, cloud servers (AWS, Google Cloud, etc.)

[1216] Software: Flask (a Python web application framework), TensorFlow (a machine learning framework), OpenCV (an image analysis library), and OpenAI's GPT-3 or GPT-4

[1217] 3. Specific Examples

[1218] Example 1: Pet emotion analysis by in-store robot

[1219] A robot installed inside a pet shop will record the sounds and images of customers' pets.

[1220] The robot sends the data to a server and receives the analysis results, which provide advice such as, "This dog seems a little anxious. Why don't you try a soothing toy?"

[1221] Based on the information provided in the app, pet store staff will recommend the right products.

[1222] Example 2: Checking health status using a smartphone app

[1223] The user takes a photo of their pet using a smartphone app and sends the data to the server.

[1224] The server analyzes the image and determines that the cat is stressed. Along with the analysis result, it generates an image of the cat's facial expression that indicates stress.

[1225] Users will receive notifications in the app and can check the details to provide appropriate care.

[1226] 4. Examples of prompts

[1227] An example prompt has the following format:

[1228] Generate a message to communicate the pet's status:

[1229] Take your pet's sound, image, or any media and explain the analysis results in natural language, for example, "My dog ​​is feeling anxious."

[1230] This system makes it possible to intuitively and quickly grasp a pet's emotional and health status, and efficiently provide appropriate care and product recommendations.

[1231] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1232] Step 1:

[1233] Users can use their smartphones or in-store robots to record and capture sounds, images, and videos of their pets, and then send the data. The input includes sounds, images, and video data of the pets. This data is then sent from the device to the server.

[1234] Step 2:

[1235] The server receives the data sent from the user using the data receiving means. The input includes the pet's cry, image, and video data sent in step 1. The received data is stored on the server.

[1236] Step 3:

[1237] The server preprocesses the received data using the data preprocessing means. Specifically, it performs noise removal on audio data, resizing and format conversion on image and video data. The input is the data received in step 2, and the output is the data after noise removal and resizing processing.

[1238] Step 4:

[1239] The server uses analytical means to analyze the preprocessed data with a generative AI model. For audio data, the emotion of the animal's cry (such as "joy" or "anxiety") is determined, and for images and videos, the pet's health status (such as "stress" or "health") is determined. The input is the preprocessed data, and the output is the analysis results, such as an emotion label or a health status determination.

[1240] Step 5:

[1241] Based on the analysis results, the server uses a message generation means to generate a message explaining the pet's condition in natural language. Furthermore, it uses a generation AI to create the virtual pet's cries and facial expressions. The input is the analysis results, and the output is the generated natural language message and the virtual pet's cries and facial expressions.

[1242] Step 6:

[1243] The server transmits the generated message and the generated pet sound and image to the user's terminal using a transmission means. The input is the message and image generated in step 5, and the output is a notification to the user's terminal.

[1244] Step 7:

[1245] The user receives messages and images sent from the server through a device (smartphone or in-store robot) and checks the status of their pet. The input is the sent message and image, and the output is the user's understanding and execution of appropriate action.

[1246] Step 8:

[1247] Using information output means, pet care information is provided in the physical retail environment (pet shop) accessed by the user. The input is analysis results and messages sent from the server, and the output is information provided by in-store displays and robots. Users and staff use this information to provide pet care and product recommendations.

[1248] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1249] This invention relates to a system that analyzes pet sounds, images, and videos, and combines them with an emotion engine that recognizes the user's emotions to provide information to pet owners in an easy-to-understand format. This system receives data sent from the user's device, preprocesses it, and uses a generative AI model to analyze the pet and user's emotions, and provides the results to the owner.

[1250] System Configuration

[1251] 1. Data Receiving Method

[1252] The user uses a dedicated app to record and take pictures and videos of their pets, and the device sends the data to the server. This data receiving means is the part where the server receives and stores the data.

[1253] 2. Data preprocessing methods

[1254] The server converts the received data into an analyzable format. Specifically, it performs noise reduction on audio data and resizes and converts the format of image and video data. This data preprocessing method enables efficient analysis.

[1255] 3. Analysis method

[1256] The server uses a generative AI model to analyze the preprocessed data. For example, if it is audio data, it will assign an emotional label to the sound of the animal's bark (such as "joy" or "anxiety"), and if it is image or video data, it will determine the pet's health condition (such as "stress" or "health"). This analysis method allows for a detailed understanding of the pet's condition.

[1257] 4. Emotion Engine

[1258] The server separately receives the user's camera footage and audio, and uses an emotion engine to analyze the user's emotions. The analysis results are displayed as emotion labels such as "happiness," "sadness," and "surprise." This allows the owner's psychological state to be understood at the same time.

[1259] 5. Message Generation Methods

[1260] Based on the analysis results, the server generates a message that comprehensively explains the status of the pet and the user. Furthermore, it uses a generation AI to create virtual pet sounds and facial expressions. This message generation method makes it easier for owners to intuitively understand the status of their pet and themselves.

[1261] 6. Transmission Method

[1262] The server sends the generated message and the generated pet's sound and image to the device, which then notifies the user via the app, allowing the user to quickly update the status of their pet and themselves.

[1263] Specific Examples

[1264] Example 1: Dog barking and user emotion analysis

[1265] Users record their dog's bark using a dedicated app and capture their own facial expressions using the camera.

[1266] The terminal transmits the recording data and image data to the server.

[1267] The server receives the audio data and image data and performs noise removal and preprocessing.

[1268] The server analyzes the sound using a generative AI model and determines it as "anxiety." At the same time, it analyzes the user's emotions using an emotion engine and determines it as "surprise."

[1269] The server generates a message saying "The dog is anxious and the user is surprised" and generates a virtual anxious cry and an image of a surprised expression.

[1270] The terminal receives the data from the server and notifies the user.

[1271] The user checks the notification in the app and sees the analysis results and generated data.

[1272] Example 2: Cat images and user sentiment analysis

[1273] Users take photos of their cats using a dedicated app and then use the camera to capture their own images.

[1274] The terminal transmits the photo data and image data to the server.

[1275] The server receives the photo data and image data and performs any necessary pre-processing.

[1276] The server analyzes the image using a generative AI model and determines whether it is "stressed." At the same time, it analyzes the user's emotions using an emotion engine and determines whether it is "joy."

[1277] The server generates a message saying "The cat is stressed, and the user is happy," and generates an image of a stressed expression and an image of a happy expression.

[1278] The terminal receives the data from the server and notifies the user.

[1279] The user checks the notification in the app and sees the analysis results and generated data.

[1280] In this way, the system of the present invention provides an easy-to-understand picture of the pet and user's condition, making it easier to meet the pet's needs and improving the happiness and health of both the pet and the owner.

[1281] The processing flow will be explained below.

[1282] Example 1: Dog barking and user emotion analysis

[1283] Step 1:

[1284] Users use a dedicated app to record their dog's barking.

[1285] At the same time, the user captures their own facial expressions using the app's camera function.

[1286] Step 2:

[1287] The device temporarily stores the audio recording data and facial expression image data.

[1288] Step 3:

[1289] The device transmits the audio recording data and facial expression image data to a server via an internet connection.

[1290] The server receives the audio recording data and facial expression image data and stores them in a database.

[1291] Step 4:

[1292] The server performs noise reduction on the received audio data.

[1293] The server adjusts the layout and converts the format of the image data.

[1294] Step 5:

[1295] The server inputs the preprocessed audio data into a generative AI model.

[1296] The generative AI model determines the emotion of a dog's bark from audio data as "anxiety."

[1297] Step 6:

[1298] The server inputs the preprocessed facial expression image data into the emotion engine.

[1299] The emotion engine determines the user's emotion as "surprise" from the facial expression image.

[1300] Step 7:

[1301] Based on the analysis results, the server generates a message saying, "The dog is feeling anxious and the user is surprised."

[1302] In addition, it generates virtual cries that indicate anxiety and images of surprised facial expressions.

[1303] Step 8:

[1304] The server sends the generated message, the bird's cry, and an image of its facial expression to the device.

[1305] Step 9:

[1306] The terminal receives the data from the server.

[1307] The device notifies the user of the received data via the app.

[1308] Step 10:

[1309] The user checks the notification in the app and sees the analysis results provided and the generated sound and facial expression images.

[1310] Example 2: Cat images and user sentiment analysis

[1311] Step 1:

[1312] Users take photos of their cats using a dedicated app.

[1313] At the same time, the user captures an image of themselves using the app's camera function.

[1314] Step 2:

[1315] The device temporarily stores the photograph data and facial expression image data.

[1316] Step 3:

[1317] The device transmits the photograph data and facial expression image data to a server via an internet connection.

[1318] The server receives the photograph data and facial expression image data and stores them in a database.

[1319] Step 4:

[1320] The server adjusts the layout and corrects the color of the image data received.

[1321] Step 5:

[1322] The server inputs the preprocessed image data into the generative AI model.

[1323] The generative AI model determines from image data that the cat's facial expression is "stressed."

[1324] Step 6:

[1325] The server inputs the preprocessed facial expression image data into the emotion engine.

[1326] The emotion engine determines the user's emotion as "joy" from the facial expression image.

[1327] Step 7:

[1328] Based on the analysis results, the server generates a message saying, "The cat is stressed, and the user is happy."

[1329] Furthermore, virtual facial expression images showing stress and happy expressions are generated.

[1330] Step 8:

[1331] The server sends the generated message and image to the terminal.

[1332] Step 9:

[1333] The terminal receives the data from the server.

[1334] The device notifies the user of the received data via the app.

[1335] Step 10:

[1336] The user checks the notification in the app and sees the analysis results and generated facial expression images.

[1337] In this way, the system of the present invention provides an easy-to-understand picture of the pet and user's condition, making it easier to meet the pet's needs and improving the happiness and health of both the pet and the owner.

[1338] Example 2

[1339] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1340] Previous systems that analyzed pet sounds, images, and videos were limited to understanding the pet's emotions and health status, but were unable to consider the owner's emotional state. This made it difficult to comprehensively understand the relationship between pet and owner, and meant that the systems were unable to adequately meet the pet's needs.

[1341] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a data receiving means for analyzing the pet's cries, images, and videos, a data preprocessing means for preprocessing the received data into an analyzable format, an analysis means for determining the pet's emotions and health condition using a generative AI model based on the preprocessed data, a means for receiving the user's camera video and audio and analyzing the user's emotions using an emotion engine, a message generating means for generating an explanatory message in natural language based on the analysis results, and a transmission means for transmitting the generated message and the generated pet's cries and images to the user's terminal. This makes it possible to comprehensively analyze the emotional states of the pet and the user and provide information to the owner in a format that is easy to understand.

[1342] The "data receiving means" is a mechanism for receiving pet sounds, images, and videos from the user's device.

[1343] The "data preprocessing means" is a mechanism that performs noise removal, resizing, and format conversion to convert received data into an analyzable format.

[1344] A "generative AI model" is a pre-trained artificial intelligence algorithm used to determine a pet's emotions and health status.

[1345] The "analysis means" is a mechanism that uses a generative AI model based on pre-processed data to determine the pet's emotions and health status.

[1346] "Means for receiving camera footage and audio from a user and analyzing the user's emotions using an emotion engine" is a mechanism for receiving camera footage and audio from a user and analyzing the user's emotions using an emotion engine.

[1347] The "message generation means" is a mechanism for generating an explanatory message in natural language based on the analysis results of the pet and the user.

[1348] The "transmission means" is a mechanism for transmitting the generated message and the generated pet's cry and image to the user's terminal.

[1349] This invention relates to a system that analyzes pet sounds, images, and videos, and combines them with an emotion engine that recognizes the user's emotions to provide information to pet owners in an easy-to-understand format. This system receives data sent from the user's device, preprocesses it, and uses a generative AI model to analyze the emotions of the pet and the user, and provides the results to the owner.

[1350] System Configuration

[1351] Data Receiving Method

[1352] The user uses a dedicated app to record and take pictures and videos of their pets, and the device sends the data to the server. This data receiving means is the part where the server receives and stores the data.

[1353] Data preprocessing measures

[1354] The server converts the received data into an analyzable format. Specifically, it performs noise removal on audio data and resizes and converts the format of image and video data. As data preprocessing methods, it uses FFT-based filtering for audio processing and the OpenCV library for image processing.

[1355] Analysis means

[1356] The server analyzes the preprocessed data using a generative AI model. TensorFlow or PyTorch are used for the generative AI model. For example, if the data is audio, it can assign an emotional label to the sound of the animal's bark (such as "joy" or "anxiety"), and if the data is image or video, it can determine the pet's health status (such as "stress" or "health"). This allows for a detailed understanding of the pet's condition.

[1357] Emotion Engine

[1358] The server separately receives the user's camera footage and audio, and uses an emotion engine to analyze the user's emotions. The analysis results are displayed as emotion labels such as "happiness," "sadness," and "surprise." The emotion engine uses Microsoft Azure Cognitive Services, which allows the owner's psychological state to be grasped at the same time.

[1359] Message Generation Method

[1360] Based on the analysis results, the server generates a message that comprehensively describes the status of the pet and the user. A natural language generation engine (e.g., GPT-4) is used to generate the message. Generative AI is also used to create virtual pet sounds and facial expressions. The DALL-E model is used to generate the virtual data.

[1361] Transmission method

[1362] The server sends the generated message and the generated pet's sound and image to the device, which then notifies the user via the app, allowing the user to quickly update the status of their pet and themselves.

[1363] Specific Examples

[1364] Example 1: Dog barking and user emotion analysis

[1365] Users record their dog's bark using a dedicated app and capture their own facial expressions using the camera.

[1366] The terminal transmits the recording data and image data to the server.

[1367] The server receives the audio data and performs noise reduction, and resizes the image data.

[1368] The server analyzes the sound using a generative AI model and determines it as "anxiety." At the same time, it analyzes the user's emotions using an emotion engine and determines it as "surprise."

[1369] The server generates a message saying "The dog is anxious and the user is surprised" and generates a virtual anxious cry and an image of a surprised expression.

[1370] The terminal receives the data from the server and notifies the user.

[1371] The user can check the analysis results and generated data in the app.

[1372] Example 2: Cat images and user sentiment analysis

[1373] Users take photos of their cats using a dedicated app and then use the camera to capture their own images.

[1374] The terminal transmits the photo data and image data to the server.

[1375] The server receives the photo data and image data and performs any necessary pre-processing.

[1376] The server analyzes the image using a generative AI model and determines whether it is "stressed." At the same time, it analyzes the user's emotions using an emotion engine and determines whether it is "joy."

[1377] The server generates a message saying "The cat is stressed, and the user is happy," and generates an image of a stressed expression and an image of a happy expression.

[1378] The terminal receives the data from the server and notifies the user.

[1379] The user checks the notification in the app and sees the analysis results and generated data.

[1380] In this way, the system of the present invention provides an easy-to-understand picture of the pet and user's condition, making it easier to meet the pet's needs and improving the happiness and health of both the pet and the owner.

[1381] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1382] Step 1: Getting and sending data

[1383] The user launches a dedicated app installed on their smartphone. They tap the record button in the app to record their pet's cries, or tap the camera button to take photos or videos of their pet. When recording or taking a photo, the user's facial expressions are also captured by the camera.

[1384] Input: Pet sounds, images, videos, user face images

[1385] Output: Audio files, image files, video files, user face image data

[1386] The terminal sends this data to the server using the HTTP protocol.

[1387] Step 2: Receiving and storing data

[1388] The server receives data sent from the device via a RESTful API. The received data is stored in local storage or cloud storage (e.g., AWS S3). Once stored, metadata is recorded in a database (e.g., MySQL) for data management.

[1389] Input: Audio files, image files, video files, user face image data

[1390] Output: Saved files, metadata

[1391] Step 3: Preprocessing the data

[1392] The server picks up the stored audio data and applies noise reduction algorithms (e.g., FFT-based filtering). For image and video data, it resizes and converts the format using the OpenCV library. Specifically, images are resized to 640x480 pixels and videos are encoded in MP4 format.

[1393] Input: saved audio files, image files, video files

[1394] Output: Pre-processed audio data, resized and formatted image and video data

[1395] Step 4: Analyze the data

[1396] The server inputs the preprocessed data into a generative AI model (e.g., TensorFlow model) for analysis. An emotion analysis model is applied to the audio data to assign labels such as "joy," "anxiety," and "anger," and an object detection model is applied to the image and video data to determine the pet's health status (e.g., "stressed," "relaxed," etc.).

[1397] Input: Preprocessed audio data, image and video data

[1398] Output: Audio data with emotion labels, image and video data with health status labels

[1399] Step 5: Analyzing user sentiment

[1400] The server receives the user's camera footage and audio data and analyzes it using an emotion engine (e.g., Microsoft Azure Cognitive Services). As a result, it obtains the user's emotion label (e.g., "joy," "sadness," "surprise," etc.).

[1401] Input: User's face image data, voice data

[1402] Output: User data with emotion labels

[1403] Step 6: Message Generation

[1404] The server generates messages using a natural language generation engine (e.g., GPT-4) based on the analysis results of the pet and the user. It also uses generative AI to create virtual pet sounds and facial expressions. The DALL-E model is used to generate the virtual data.

[1405] Input: Analysis results of pet and user (emotion labels, health status labels, etc.)

[1406] Output: Generated natural language messages, virtual sounds and facial expressions

[1407] Step 7: Data submission and notification

[1408] The server sends the generated message and virtual data to the API endpoint of the dedicated app.

[1409] The device receives the data from the server, stores it locally, and displays a notification to the user through the app. When the user opens the app, they can view the analysis results and the generated virtual data.

[1410] Input: Generated natural language messages, virtual sounds, and facial expressions

[1411] Output: Notification to user terminal, display of analysis results and generated data

[1412] This system can comprehensively analyze the emotional states of both the pet and the user, and provide information in a format that is easy for the owner to understand.

[1413] (Application example 2)

[1414] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1415] Pet owners spend a lot of time and effort understanding their pets' emotions and health conditions and providing appropriate care. It is also difficult to find products and services that suit their pets' conditions. As a result, pet needs cannot be met quickly and accurately, which can lead to a decline in the happiness and health of both the pet and the owner. To solve this problem, a system is needed that analyzes the conditions of both the pet and the owner and suggests appropriate products and services based on that analysis.

[1416] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data receiving means for analyzing the pet's cries, images, and videos; a data preprocessing means for preprocessing the received data into an analyzable format; an analysis means for determining the pet's emotions and health condition using a generative AI model based on the preprocessed data; a proposal means for proposing products and services based on the pet's condition and the user's emotions determined by the analysis means; a message generation means for generating an explanatory message in natural language based on the analysis results; and a transmission means for transmitting the generated message and the generated pet's cries and images to the user's terminal. This allows the owner to quickly understand the pet's condition and easily select and purchase appropriate products and services based on that information.

[1417] The "data receiving means" is a device or system for receiving data related to the sounds, images, and videos of pets, as well as the user's emotions.

[1418] "Data preprocessing means" refers to processing devices and algorithms that convert received data into an analyzable format. This means performs preprocessing such as noise removal and format conversion.

[1419] "Analysis means" refers to a device or system that uses a generative AI model to determine the pet's emotions and health status based on pre-processed data.

[1420] The "suggestion means" is a device or system that suggests appropriate products or services based on the state of the pet and the user's emotions determined by the analysis means.

[1421] The "message generating means" is a device or system that generates a message explaining the state of the pet and the user in natural language based on the analysis results.

[1422] The "transmission means" is a device or system that transmits the generated message and the sounds and images of the pet to the user's terminal.

[1423] A "generative AI model" is an artificial intelligence model used to analyze a pet's emotions and health condition from received data.

[1424] "Natural language" refers to a language format that humans use on a daily basis, and is used to provide analysis results to users in an intuitively understandable format.

[1425] "Products and services" is a general term for items and support activities suggested depending on the pet's condition and the user's emotions.

[1426] The present invention provides a system for analyzing the state of a pet and the emotions of a user, and for proposing appropriate products and services based on the results. This system has the following configuration.

[1427] First, the user installs a dedicated application on their smartphone and captures their pet's cries, images, and videos. This activates the data receiving means, which then transmits the captured data to the server. This data receiving means includes a standard smartphone camera and microphone.

[1428] Next, the server preprocesses the received data using a data preprocessing means. This preprocessing means includes, for example, noise removal from audio data, resizing of image data, and format conversion. This process typically uses a programming language such as Python, with libraries such as OpenCV or Librosa.

[1429] Once preprocessed, the data is analyzed using a generative AI model. The analysis methods include analyzing pet sounds to determine emotions, and analyzing images and videos to determine health and stress levels. The generative AI model is built using deep learning frameworks such as TensorFlow and PyTorch.

[1430] Furthermore, the analysis method analyzes not only the pet's condition but also the user's emotions. The user captures their own facial expressions with their smartphone camera, and the data is sent to the server. The emotion engine determines the user's emotions based on this data. This analysis uses Facial Recognition API and natural language processing engines (NLP engines), among others.

[1431] Based on the analysis results, the suggestion unit operates to suggest products and services according to the condition of the pet and the user. These suggestions include items to reduce stress for pets and relaxation items for owners.

[1432] The message generation means then generates a message in natural language based on the analysis results. For example, "If your pet is feeling stressed, why not try purchasing some relaxation products?" This generated message, along with the sound and image of the pet, is then sent back to the user's smartphone via the transmission means. The user receives this as a notification and can check it on the application.

[1433] For example, the following prompts might be fed into a generative AI model:

[1434] Output analysis results based on the pet's image and the user's emotion. For example, if the pet is stressed and the user is surprised, return information including the emotion label. Also, suggest products (such as relaxation goods) appropriate for the pet's condition.

[1435] This system allows users to quickly understand the condition of their pets and easily select and purchase appropriate products and services based on that information, ultimately improving the happiness and health of both pets and their owners.

[1436] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1437] Step 1:

[1438] The user uses a smartphone to capture the sound, images, and videos of their pet, and then sends the data to the server through a dedicated application. The input data is the sound, images, and videos of the pet, and the output is the raw data sent to the server. This step uses the smartphone's built-in camera and microphone.

[1439] Step 2:

[1440] The server receives data sent by the user using a data receiving means. The input data is the unprocessed data sent from the smartphone, and the output is the received data. In this step, communication means such as HTTP requests and WebSockets are used.

[1441] Step 3:

[1442] The server uses data preprocessing to convert the received data into an analyzable format. The input data is the received raw data, and the output is the preprocessed data. This step involves noise removal for audio data, resizing of image data, and format conversion. Specifically, the Python libraries OpenCV and Librosa are used.

[1443] Step 4:

[1444] The server analyzes the preprocessed data using a generative AI model. The input data is the preprocessed data, and the output is the analysis results showing the pet's emotions and health status. In this step, a generative AI model built using TensorFlow and PyTorch is used. The model assigns emotion labels such as "joy" or "anxiety" to the pet's cries, and determines its state, such as "stress" or "health," from images and videos.

[1445] Step 5:

[1446] The server analyzes the user's emotions using an emotion engine. The input data is the user's facial expression data captured by the smartphone camera, and the output is an analysis result indicating the user's emotions. In this step, the Facial Recognition API and a natural language processing engine (NLP engine) are used.

[1447] Step 6:

[1448] The server runs a suggestion unit that suggests products and services based on the analysis results. The input data are the analysis results of the pet and the user, and the output is a list of suggested products and services. In this step, a recommendation engine is used to select products and services suitable for the user. If the pet is feeling stressed, for example, relaxation goods will be suggested.

[1449] Step 7:

[1450] The server uses a message generation means to generate a message in natural language. The input data is the analysis results and a list of suggestions, and the output is a message to be sent to the user. The generated message will explain the pet's condition and the suggested products and services. In this step, an NLP engine is used to generate an intuitive and easy-to-understand message.

[1451] Step 8:

[1452] The server operates a sending means that sends the generated message and the generated pet sound and image to the user's device. The input data is the generated message and the pet sound and image, and the output is a notification that arrives on the user's smartphone. In this step, a push notification service (e.g., Firebase Cloud Messaging) is used.

[1453] Step 9:

[1454] The user checks the notification through the application and views the analysis results and suggested products and services. The input data is the notification sent from the server, and the output is the information provided to the user. In this step, a dedicated application on the smartphone is used, allowing the user to easily check the information and select and purchase the suggested products and services.

[1455] By linking these processing steps, the user can quickly grasp the condition of their pet and provide appropriate care based on that information.

[1456] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1457] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1458] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1459] [Fourth embodiment]

[1460] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1461] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1462] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1463] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1464] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1465] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1466] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1467] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1468] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1469] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1470] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1471] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1472] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1473] This invention relates to a system that analyzes pet sounds, images, and videos and provides information to pet owners in an easy-to-understand format. This system receives data sent from a user's device, preprocesses it, analyzes it using a generative AI model, and provides the results to the pet owner.

[1474] System Configuration

[1475] 1. Data Receiving Method

[1476] The user uses a dedicated app to record and take pictures and videos of their pets, and the device sends the data to the server. This data receiving means is the part where the server receives and stores the data.

[1477] 2. Data preprocessing methods

[1478] The server converts the received data into an analyzable format. Specifically, it performs noise reduction on audio data and resizes and converts the format of image and video data. This data preprocessing method enables efficient analysis.

[1479] 3. Analysis method

[1480] The server uses a generative AI model to analyze the preprocessed data. For example, if it is audio data, it will assign an emotional label to the sound of the animal's bark (such as "joy" or "anxiety"), and if it is image or video data, it will determine the pet's health condition (such as "stress" or "health"). This analysis method allows for a detailed understanding of the pet's condition.

[1481] 4. Message Generation Methods

[1482] Based on the analysis results, the server generates a message in natural language explaining the pet's condition. Furthermore, it uses a generation AI to create virtual pet sounds and facial expressions. This message generation method makes it easier for owners to intuitively understand their pet's condition.

[1483] 5. Transmission Method

[1484] The server sends the generated message and the generated pet's bark and image to the device, which then notifies the user via the app, allowing them to quickly update the pet's status.

[1485] Specific Examples

[1486] Example 1: Dog bark analysis

[1487] Users record their dog's barking using a dedicated app.

[1488] The device sends the recorded data to the server.

[1489] The server receives the audio data and performs noise reduction.

[1490] The server uses a generative AI model to analyze the sound and determine that it is "anxious."

[1491] The server generates a message saying "The dog is anxious" and generates a virtual anxious bark.

[1492] The terminal receives the data from the server and notifies the user.

[1493] Users can check the notification in the app for more details.

[1494] Example 2: Cat image analysis

[1495] Users take photos of their cats using a dedicated app.

[1496] The terminal transmits the image data to the server.

[1497] The server receives the image data and performs any necessary pre-processing.

[1498] The server uses a generative AI model to analyze the image and determine whether it is "stressed."

[1499] The server generates a message saying "The cat is stressed" and an image of a facial expression showing stress.

[1500] The terminal receives the data from the server and notifies the user.

[1501] Users can check the notification in the app for more details.

[1502] In this way, the system of the present invention provides owners with an easy-to-understand view of their pet's condition, making it easier to meet their pet's needs and improving their pet's happiness and health.

[1503] The processing flow will be explained below.

[1504] Example 1: Dog bark analysis

[1505] Step 1:

[1506] Users use a dedicated app to record their dog's barking.

[1507] The device temporarily stores the recorded data.

[1508] Step 2:

[1509] The device sends the recording data to a server via an internet connection.

[1510] The server receives the recording data and stores it in a database.

[1511] Step 3:

[1512] The server performs pre-processing to remove noise from the received audio data.

[1513] The denoised audio data is converted into an analyzable format.

[1514] Step 4:

[1515] The server inputs the preprocessed audio data into a generative AI model.

[1516] The generative AI model analyzes the emotion of the bird's cry from the audio data and determines it to be "anxiety."

[1517] Step 5:

[1518] Based on the analysis results, the server creates a natural language message saying, "The dog is feeling anxious."

[1519] Additionally, it generates virtual cries that indicate anxiety.

[1520] Step 6:

[1521] The server sends the generated message and sound to the device.

[1522] Step 7:

[1523] The terminal receives the data from the server.

[1524] The device notifies the user of the received data via the app.

[1525] Step 8:

[1526] The user opens the app and sees the analysis results provided and the generated sounds.

[1527] Example 2: Cat image analysis

[1528] Step 1:

[1529] Users take photos of their cats using a dedicated app.

[1530] The device temporarily stores the captured data.

[1531] Step 2:

[1532] The device sends the photo data to a server via an Internet connection.

[1533] The server receives the photo data and stores it in a database.

[1534] Step 3:

[1535] The server preprocesses the received image data into an analyzable format.

[1536] Performs format conversion such as adjusting resolution and color.

[1537] Step 4:

[1538] The server inputs the preprocessed image data into the generative AI model.

[1539] The generative AI model analyzes the cat's facial expression from image data and determines its "stress."

[1540] Step 5:

[1541] Based on the analysis results, the server creates a natural language message saying, "The cat is feeling stressed."

[1542] Furthermore, a virtual facial expression image showing stress is generated.

[1543] Step 6:

[1544] The server sends the generated message and image to the terminal.

[1545] Step 7:

[1546] The terminal receives the data from the server.

[1547] The device notifies the user of the received data via the app.

[1548] Step 8:

[1549] The user opens the app and checks the analysis results provided and the generated facial expression image.

[1550] In this way, the specific processing of the system is carried out to analyze the pet's condition in detail through each step and provide the results to the owner.

[1551] Example 1

[1552] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1553] In recent years, there has been an increasing demand for pet health management and emotional understanding, but it is often difficult for pet owners to accurately grasp their pet's condition. With current technology, analyzing a pet's cries and facial expressions requires a high level of specialized knowledge, making it difficult for the average pet owner to access. To solve this problem, a system that allows owners to easily and intuitively understand their pet's condition is needed.

[1554] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1555] In this invention, the server includes means for transmitting data recorded or photographed by a user's device through a dedicated app, means for the server to receive the data and perform preprocessing such as noise removal and format conversion, means for determining emotion labels and health status using a generative AI model based on the preprocessed data, and means for generating virtual pet cries and facial expression images based on the analysis results, thereby enabling owners to easily understand their pet's emotions and health status through the dedicated app.

[1556] The "data receiving means" is a means for transmitting sounds, images, and videos of pets from the user's terminal to the server.

[1557] The "data preprocessing means" is a means for converting the data received by the server into an analyzable format, and includes processes such as noise removal, format conversion, and resizing.

[1558] "Analysis means" means a means for determining the emotions and health status of a pet using a generative AI model based on pre-processed data.

[1559] The "message generating means" is a means for generating a message that explains the state of the pet in natural language based on the analysis results.

[1560] The "transmission means" is a means for transmitting the generated message and the sound and image of the virtual pet to the user's terminal.

[1561] A "dedicated app" is an application program that users use to record and take pictures and videos of their pets and send them to a server.

[1562] "Noise reduction" is the process of removing unnecessary background sounds from audio data.

[1563] "Format conversion" refers to the process of converting data into a format that is easy to analyze.

[1564] A "generative AI model" is an artificial intelligence model that determines a pet's emotional label and health condition based on preprocessed data and outputs the results.

[1565] An "emotion label" is an emotion category (such as "joy" or "anxiety") that is inferred from a pet's sounds and behavior.

[1566] "Health status" refers to the health status (such as "stress" or "health") estimated from image and video data of a pet.

[1567] "Natural language" refers to words and sentences that humans use on a daily basis, and is used in this system to convey analysis results to users in an intuitively easy-to-understand format.

[1568] "Virtual pet cries" refer to the artificially generated cries of pets based on the analysis results.

[1569] A "virtual facial expression image" is an image of a pet's facial expression that is artificially generated based on the analysis results.

[1570] "User's terminal" refers to an electronic device (smartphone, tablet, etc.) used by a user on which a dedicated app is installed.

[1571] This invention relates to a system that allows users to analyze their pet's cries, images, and videos to more accurately and intuitively understand their pet's emotions and health condition. This system uses the user's device, a server, and a generative AI model to collect and analyze data and provide the results to the user.

[1572] Hardware and Software

[1573] Hardware:

[1574] Server: A central device that receives, stores, and analyzes data such as bird sounds, images, and videos.

[1575] Device: An electronic device (smartphone, tablet, etc.) used by a user to record and capture sounds, images, and videos of pets and send them to a server.

[1576] software:

[1577] Dedicated app: An application installed by the user that records and photographs data, sends it to the server, and receives and displays the analysis results.

[1578] OpenSMILE: Software used for noise reduction and preprocessing of audio data.

[1579] OpenCV: Software that performs preprocessing such as resizing and format conversion of image data.

[1580] Generative AI model: An artificial intelligence model (e.g., GPT-4, ResNet) that analyzes voice data and assigns emotion labels (happiness, anxiety, etc.), or analyzes image data and determines health status (stress, health, etc.).

[1581] Server application: A program that runs on the server and manages data reception, preprocessing, analysis, and transmission of generated data.

[1582] Program flow

[1583] 1. Data collection and transmission

[1584] Users use a dedicated app to record and take pictures and videos of their pets' cries.

[1585] The device sends the recorded and photographed data to a server using Wi-Fi or mobile data networks.

[1586] 2. Data Preprocessing

[1587] The server loads the received data, performs noise reduction on the audio data, and resizes and converts the image data. OpenSMILE is used for preprocessing the audio data, and OpenCV is used for preprocessing the image data.

[1588] 3. Data Analysis

[1589] The server inputs the preprocessed data into a generative AI model, determining emotion labels for bird sounds and health status for image data. GPT-4 and ResNet are used for analysis.

[1590] 4. Message Creation

[1591] Based on the analysis results, the server generates a natural language message explaining the pet's condition. It also generates the virtual pet's cries and facial expressions. It uses GPT-4 for message generation, Tacotron for voice generation, and DALL-E for image generation.

[1592] 5. Transmission and Notification

[1593] The server transmits the generated message and virtual data to the terminal.

[1594] The device notifies the user of the received data and allows them to check the analysis results using a dedicated app.

[1595] Specific examples

[1596] Example 1: Dog bark analysis

[1597] Users record their dog's barking using a dedicated app.

[1598] The device sends the recorded data to the server.

[1599] The server receives the audio data and uses OpenSMILE to perform noise reduction.

[1600] The server analyzes the sound using a generative AI model and determines that it is "anxious" (model used: GPT-4).

[1601] The server generates a message saying "The dog is anxious" and the Tacotron generates a virtual anxious bark.

[1602] The terminal receives the data from the server and notifies the user.

[1603] Prompt Sentence Examples

[1604] Enter your dog's bark and analyze its emotions.

[1605] Input audio: Woof woof woof (audio data)

[1606] Example 2: Cat image analysis

[1607] Users take photos of their cats using a dedicated app.

[1608] The terminal transmits the image data to the server.

[1609] The server receives the image data and uses OpenCV to resize and convert the format.

[1610] The server analyzes the image using a generative AI model and determines that it is "stressed" (model used: ResNet).

[1611] The server generates a message saying "The cat is feeling stressed," and DALL-E generates an image of a facial expression showing stress.

[1612] The terminal receives the data from the server and notifies the user.

[1613] Prompt Sentence Examples

[1614] Enter a photo of your cat and rate its health.

[1615] Input image: (image data of a cat)

[1616] This system allows users to quickly and accurately understand the condition of their pet and take appropriate action.

[1617] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1618] Step 1:

[1619] Users use a dedicated app to record and take pictures and videos of their pets' cries.

[1620] Specifically, the user presses the app's "record" button to record the dog's bark.

[1621] Input: Pet sounds, images, videos

[1622] Output: Audio, image, and video files stored on the user's device

[1623] Step 2:

[1624] The device sends the recorded and photographed data to the server.

[1625] Specifically, the device uses Wi-Fi or a cellular data network to send data to the server as an HTTP POST request.

[1626] Input: User recorded and photographed data

[1627] Output: Data uploaded to the server

[1628] Step 3:

[1629] The server stores the received data.

[1630] Specifically, the server receives the request and stores it in a database. For example, it stores the audio file in cloud storage and the metadata in a database.

[1631] Input: Data sent from the terminal

[1632] Output: Data stored on the server

[1633] Step 4:

[1634] The server pre-processes the audio data.

[1635] Specifically, OpenSMILE is used to remove noise from audio data.

[1636] Input: Stored audio data

[1637] Output: Denoised audio data

[1638] Step 5:

[1639] The server pre-processes the image data.

[1640] Specifically, OpenCV is used to resize and convert the format of image data.

[1641] Input: Saved image data

[1642] Output: Resized and formatted image data

[1643] Step 6:

[1644] The server inputs the preprocessed data into the generative AI model for analysis.

[1645] Specifically, it uses pre-trained generative AI models (e.g., GPT-4 and ResNet) to determine emotion labels from audio data and health status from image data.

[1646] Input: Preprocessed audio data, image data

[1647] Output: Emotion label (e.g., happy, anxious), health status (e.g., stress, healthy)

[1648] Step 7:

[1649] The server generates a message based on the analysis results.

[1650] Specifically, it uses a generative AI model (e.g., GPT-4) to generate analysis results as natural language messages, as well as virtual pet sounds and facial expressions.

[1651] Input: emotion label, health status

[1652] Output: Natural language message (e.g., the dog is anxious), virtual barks and facial expressions

[1653] Step 8:

[1654] The server sends the generated message and data to the terminal.

[1655] As a specific operation, an HTTP response including the generated message and data is created and sent to the terminal.

[1656] Input: Natural language messages, virtual sounds and facial expressions

[1657] Output: Messages and data sent to the terminal

[1658] Step 9:

[1659] The terminal notifies the user of the received data.

[1660] Specifically, the terminal uses a notification function to notify the user that "there is a new message."

[1661] Input: Message and data sent from the server

[1662] Output: User notification

[1663] Step 10:

[1664] Users can check the notifications through the app and understand their pet's condition.

[1665] Specifically, the user opens the app and views detailed messages and virtual data.

[1666] Input: Notification from device

[1667] Output: Analysis results and virtual data displayed through the app

[1668] This series of processes allows the user to quickly and accurately grasp the condition of the pet and take appropriate action.

[1669] (Application example 1)

[1670] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1671] Pet owners need specialized knowledge to understand their pets' emotions and health conditions, and this can lead to inadequate care. Pet shops and retailers are also required to quickly and easily assess the condition of their pets and provide appropriate advice, but there is currently no efficient way to meet this need.

[1672] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1673] In this invention, the server includes a data receiving means for analyzing the pet's cries, images, and videos, a data preprocessing means for preprocessing the received data into an analyzable format, an analysis means for determining the pet's emotions and health condition using a generative AI model based on the preprocessed data, a message generating means for generating an explanatory message in natural language based on the analysis results, a transmission means for transmitting the generated message and the generated pet's cries and images to a user's terminal, and an information output means for providing pet care information within a physical retail environment accessed by the user. This makes it possible to intuitively and quickly grasp the pet's emotions and health condition and efficiently provide appropriate care and product recommendations.

[1674] The "data receiving means" is a means by which a user can send sounds, images, and videos of their pet to the server via the terminal.

[1675] The "data preprocessing means" is a means for converting received data into an analyzable format, and performs noise removal, resizing, and format conversion.

[1676] "Analysis means" refers to a means for determining a pet's emotions and health status using a generative AI model based on pre-processed data.

[1677] The "message generation means" is a means for generating an explanatory message in natural language based on the analysis results, to clearly communicate the pet's condition.

[1678] The "transmission means" is a means for transmitting the generated message and the sound and image of the pet to the user's terminal.

[1679] "Information output means" refers to means for providing pet care information within the physical retail environment accessed by the user, and includes displays, audio guidance devices, and the like.

[1680] The present invention is a system for understanding the condition of pets and providing appropriate care, and is effective in pet shops and physical retail environments. Specific embodiments for implementing the system will be described below.

[1681] 1. System Configuration

[1682] This system consists of the following elements:

[1683] Data Receiving Method

[1684] Users use their smartphones or other devices to record and take pictures and videos of their pets, and then send the data to a server, where it is collected and analyzed in a central system.

[1685] Data preprocessing measures

[1686] The server converts the received data into an analyzable format. Specific processing includes noise reduction for audio data and resizing and format conversion for image and video data. This processing allows for efficient data analysis.

[1687] Analysis means

[1688] The server uses a generative AI model to analyze the preprocessed data. For example, if it is audio data, it can determine the emotion of the animal's voice (such as "joy" or "anxiety"), and if it is image or video data, it can determine the pet's health status (such as "stress" or "health"). This analysis uses libraries such as TensorFlow and OpenCV.

[1689] Message Generation Method

[1690] Based on the analysis results, the server generates a message in natural language explaining the pet's condition, and then uses generative AI to create virtual sounds and facial expressions, using OpenAI's GPT-3 or GPT-4.

[1691] Transmission method

[1692] The server sends the generated message, along with the pet's sounds and images, to the user's device, who receives this information through a dedicated app.

[1693] Information output means

[1694] In the physical retail environment accessed by the user (e.g., a pet shop), displays and robots installed in the store display the analysis results and provide customers with information on the condition of their pets and care advice.

[1695] 2. Hardware and Software Used

[1696] Hardware: Smartphones (iOS / Android), in-store robots, cloud servers (AWS, Google Cloud, etc.)

[1697] Software: Flask (a Python web application framework), TensorFlow (a machine learning framework), OpenCV (an image analysis library), and OpenAI's GPT-3 or GPT-4

[1698] 3. Specific Examples

[1699] Example 1: Pet emotion analysis by in-store robot

[1700] A robot installed inside a pet shop will record the sounds and images of customers' pets.

[1701] The robot sends the data to a server and receives the analysis results, which provide advice such as, "This dog seems a little anxious. Why don't you try a soothing toy?"

[1702] Based on the information provided in the app, pet store staff will recommend the right products.

[1703] Example 2: Checking health status using a smartphone app

[1704] The user takes a photo of their pet using a smartphone app and sends the data to the server.

[1705] The server analyzes the image and determines that the cat is stressed. Along with the analysis result, it generates an image of the cat's facial expression that indicates stress.

[1706] Users will receive notifications in the app and can check the details to provide appropriate care.

[1707] 4. Examples of prompts

[1708] An example prompt has the following format:

[1709] Generate a message to communicate the pet's status:

[1710] Take your pet's sound, image, or any media and explain the analysis results in natural language, for example, "My dog ​​is feeling anxious."

[1711] This system makes it possible to intuitively and quickly grasp a pet's emotional and health status, and efficiently provide appropriate care and product recommendations.

[1712] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1713] Step 1:

[1714] Users can use their smartphones or in-store robots to record and capture sounds, images, and videos of their pets, and then send the data. The input includes sounds, images, and video data of the pets. This data is then sent from the device to the server.

[1715] Step 2:

[1716] The server receives the data sent from the user using the data receiving means. The input includes the pet's cry, image, and video data sent in step 1. The received data is stored on the server.

[1717] Step 3:

[1718] The server preprocesses the received data using the data preprocessing means. Specifically, it performs noise removal on audio data, resizing and format conversion on image and video data. The input is the data received in step 2, and the output is the data after noise removal and resizing processing.

[1719] Step 4:

[1720] The server uses analytical means to analyze the preprocessed data with a generative AI model. For audio data, the emotion of the animal's cry (such as "joy" or "anxiety") is determined, and for images and videos, the pet's health status (such as "stress" or "health") is determined. The input is the preprocessed data, and the output is the analysis results, such as an emotion label or a health status determination.

[1721] Step 5:

[1722] Based on the analysis results, the server uses a message generation means to generate a message explaining the pet's condition in natural language. Furthermore, it uses a generation AI to create the virtual pet's cries and facial expressions. The input is the analysis results, and the output is the generated natural language message and the virtual pet's cries and facial expressions.

[1723] Step 6:

[1724] The server transmits the generated message and the generated pet sound and image to the user's terminal using a transmission means. The input is the message and image generated in step 5, and the output is a notification to the user's terminal.

[1725] Step 7:

[1726] The user receives messages and images sent from the server through a device (smartphone or in-store robot) and checks the status of their pet. The input is the sent message and image, and the output is the user's understanding and execution of appropriate action.

[1727] Step 8:

[1728] Using information output means, pet care information is provided in the physical retail environment (pet shop) accessed by the user. The input is analysis results and messages sent from the server, and the output is information provided by in-store displays and robots. Users and staff use this information to provide pet care and product recommendations.

[1729] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1730] This invention relates to a system that analyzes pet sounds, images, and videos, and combines them with an emotion engine that recognizes the user's emotions to provide information to pet owners in an easy-to-understand format. This system receives data sent from the user's device, preprocesses it, and uses a generative AI model to analyze the pet and user's emotions, and provides the results to the owner.

[1731] System Configuration

[1732] 1. Data Receiving Method

[1733] The user uses a dedicated app to record and take pictures and videos of their pets, and the device sends the data to the server. This data receiving means is the part where the server receives and stores the data.

[1734] 2. Data preprocessing methods

[1735] The server converts the received data into an analyzable format. Specifically, it performs noise reduction on audio data and resizes and converts the format of image and video data. This data preprocessing method enables efficient analysis.

[1736] 3. Analysis method

[1737] The server uses a generative AI model to analyze the preprocessed data. For example, if it is audio data, it will assign an emotional label to the sound of the animal's bark (such as "joy" or "anxiety"), and if it is image or video data, it will determine the pet's health condition (such as "stress" or "health"). This analysis method allows for a detailed understanding of the pet's condition.

[1738] 4. Emotion Engine

[1739] The server separately receives the user's camera footage and audio, and uses an emotion engine to analyze the user's emotions. The analysis results are displayed as emotion labels such as "happiness," "sadness," and "surprise." This allows the owner's psychological state to be understood at the same time.

[1740] 5. Message Generation Methods

[1741] Based on the analysis results, the server generates a message that comprehensively explains the status of the pet and the user. Furthermore, it uses a generation AI to create virtual pet sounds and facial expressions. This message generation method makes it easier for owners to intuitively understand the status of their pet and themselves.

[1742] 6. Transmission Method

[1743] The server sends the generated message and the generated pet's sound and image to the device, which then notifies the user via the app, allowing the user to quickly update the status of their pet and themselves.

[1744] Specific Examples

[1745] Example 1: Dog barking and user emotion analysis

[1746] Users record their dog's bark using a dedicated app and capture their own facial expressions using the camera.

[1747] The terminal transmits the recording data and image data to the server.

[1748] The server receives the audio data and image data and performs noise removal and preprocessing.

[1749] The server analyzes the sound using a generative AI model and determines it as "anxiety." At the same time, it analyzes the user's emotions using an emotion engine and determines it as "surprise."

[1750] The server generates a message saying "The dog is anxious and the user is surprised" and generates a virtual anxious cry and an image of a surprised expression.

[1751] The terminal receives the data from the server and notifies the user.

[1752] The user checks the notification in the app and sees the analysis results and generated data.

[1753] Example 2: Cat images and user sentiment analysis

[1754] Users take photos of their cats using a dedicated app and then use the camera to capture their own images.

[1755] The terminal transmits the photo data and image data to the server.

[1756] The server receives the photo data and image data and performs any necessary pre-processing.

[1757] The server analyzes the image using a generative AI model and determines whether it is "stressed." At the same time, it analyzes the user's emotions using an emotion engine and determines whether it is "joy."

[1758] The server generates a message saying "The cat is stressed, and the user is happy," and generates an image of a stressed expression and an image of a happy expression.

[1759] The terminal receives the data from the server and notifies the user.

[1760] The user checks the notification in the app and sees the analysis results and generated data.

[1761] In this way, the system of the present invention provides an easy-to-understand picture of the pet and user's condition, making it easier to meet the pet's needs and improving the happiness and health of both the pet and the owner.

[1762] The processing flow will be explained below.

[1763] Example 1: Dog barking and user emotion analysis

[1764] Step 1:

[1765] Users use a dedicated app to record their dog's barking.

[1766] At the same time, the user captures their own facial expressions using the app's camera function.

[1767] Step 2:

[1768] The device temporarily stores the audio recording data and facial expression image data.

[1769] Step 3:

[1770] The device transmits the audio recording data and facial expression image data to a server via an internet connection.

[1771] The server receives the audio recording data and facial expression image data and stores them in a database.

[1772] Step 4:

[1773] The server performs noise reduction on the received audio data.

[1774] The server adjusts the layout and converts the format of the image data.

[1775] Step 5:

[1776] The server inputs the preprocessed audio data into a generative AI model.

[1777] The generative AI model determines the emotion of a dog's bark from audio data as "anxiety."

[1778] Step 6:

[1779] The server inputs the preprocessed facial expression image data into the emotion engine.

[1780] The emotion engine determines the user's emotion as "surprise" from the facial expression image.

[1781] Step 7:

[1782] Based on the analysis results, the server generates a message saying, "The dog is feeling anxious and the user is surprised."

[1783] In addition, it generates virtual cries that indicate anxiety and images of surprised facial expressions.

[1784] Step 8:

[1785] The server sends the generated message, the bird's cry, and an image of its facial expression to the device.

[1786] Step 9:

[1787] The terminal receives the data from the server.

[1788] The device notifies the user of the received data via the app.

[1789] Step 10:

[1790] The user checks the notification in the app and sees the analysis results provided and the generated sound and facial expression images.

[1791] Example 2: Cat images and user sentiment analysis

[1792] Step 1:

[1793] Users take photos of their cats using a dedicated app.

[1794] At the same time, the user captures an image of themselves using the app's camera function.

[1795] Step 2:

[1796] The device temporarily stores the photograph data and facial expression image data.

[1797] Step 3:

[1798] The device transmits the photograph data and facial expression image data to a server via an internet connection.

[1799] The server receives the photograph data and facial expression image data and stores them in a database.

[1800] Step 4:

[1801] The server adjusts the layout and corrects the color of the image data received.

[1802] Step 5:

[1803] The server inputs the preprocessed image data into the generative AI model.

[1804] The generative AI model determines from image data that the cat's facial expression is "stressed."

[1805] Step 6:

[1806] The server inputs the preprocessed facial expression image data into the emotion engine.

[1807] The emotion engine determines the user's emotion as "joy" from the facial expression image.

[1808] Step 7:

[1809] Based on the analysis results, the server generates a message saying, "The cat is stressed, and the user is happy."

[1810] Furthermore, virtual facial expression images showing stress and happy expressions are generated.

[1811] Step 8:

[1812] The server sends the generated message and image to the terminal.

[1813] Step 9:

[1814] The terminal receives the data from the server.

[1815] The device notifies the user of the received data via the app.

[1816] Step 10:

[1817] The user checks the notification in the app and sees the analysis results and generated facial expression images.

[1818] In this way, the system of the present invention provides an easy-to-understand picture of the pet and user's condition, making it easier to meet the pet's needs and improving the happiness and health of both the pet and the owner.

[1819] Example 2

[1820] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1821] Previous systems that analyzed pet sounds, images, and videos were limited to understanding the pet's emotions and health status, but were unable to consider the owner's emotional state. This made it difficult to comprehensively understand the relationship between pet and owner, and meant that the systems were unable to adequately meet the pet's needs.

[1822] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a data receiving means for analyzing the pet's cries, images, and videos, a data preprocessing means for preprocessing the received data into an analyzable format, an analysis means for determining the pet's emotions and health condition using a generative AI model based on the preprocessed data, a means for receiving the user's camera video and audio and analyzing the user's emotions using an emotion engine, a message generating means for generating an explanatory message in natural language based on the analysis results, and a transmission means for transmitting the generated message and the generated pet's cries and images to the user's terminal. This makes it possible to comprehensively analyze the emotional states of the pet and the user and provide information to the owner in a format that is easy to understand.

[1823] The "data receiving means" is a mechanism for receiving pet sounds, images, and videos from the user's device.

[1824] The "data preprocessing means" is a mechanism that performs noise removal, resizing, and format conversion to convert received data into an analyzable format.

[1825] A "generative AI model" is a pre-trained artificial intelligence algorithm used to determine a pet's emotions and health status.

[1826] The "analysis means" is a mechanism that uses a generative AI model based on pre-processed data to determine the pet's emotions and health status.

[1827] "Means for receiving camera footage and audio from a user and analyzing the user's emotions using an emotion engine" is a mechanism for receiving camera footage and audio from a user and analyzing the user's emotions using an emotion engine.

[1828] The "message generation means" is a mechanism for generating an explanatory message in natural language based on the analysis results of the pet and the user.

[1829] The "transmission means" is a mechanism for transmitting the generated message and the generated pet's cry and image to the user's terminal.

[1830] This invention relates to a system that analyzes pet sounds, images, and videos, and combines them with an emotion engine that recognizes the user's emotions to provide information to pet owners in an easy-to-understand format. This system receives data sent from the user's device, preprocesses it, and uses a generative AI model to analyze the emotions of the pet and the user, and provides the results to the owner.

[1831] System Configuration

[1832] Data Receiving Method

[1833] The user uses a dedicated app to record and take pictures and videos of their pets, and the device sends the data to the server. This data receiving means is the part where the server receives and stores the data.

[1834] Data preprocessing measures

[1835] The server converts the received data into an analyzable format. Specifically, it performs noise removal on audio data and resizes and converts the format of image and video data. As data preprocessing methods, it uses FFT-based filtering for audio processing and the OpenCV library for image processing.

[1836] Analysis means

[1837] The server analyzes the preprocessed data using a generative AI model. TensorFlow or PyTorch are used for the generative AI model. For example, if the data is audio, it can assign an emotional label to the sound of the animal's bark (such as "joy" or "anxiety"), and if the data is image or video, it can determine the pet's health status (such as "stress" or "health"). This allows for a detailed understanding of the pet's condition.

[1838] Emotion Engine

[1839] The server separately receives the user's camera footage and audio, and uses an emotion engine to analyze the user's emotions. The analysis results are displayed as emotion labels such as "happiness," "sadness," and "surprise." The emotion engine uses Microsoft Azure Cognitive Services, which allows the owner's psychological state to be grasped at the same time.

[1840] Message Generation Method

[1841] Based on the analysis results, the server generates a message that comprehensively describes the status of the pet and the user. A natural language generation engine (e.g., GPT-4) is used to generate the message. Generative AI is also used to create virtual pet sounds and facial expressions. The DALL-E model is used to generate the virtual data.

[1842] Transmission method

[1843] The server sends the generated message and the generated pet's sound and image to the device, which then notifies the user via the app, allowing the user to quickly update the status of their pet and themselves.

[1844] Specific Examples

[1845] Example 1: Dog barking and user emotion analysis

[1846] Users record their dog's bark using a dedicated app and capture their own facial expressions using the camera.

[1847] The terminal transmits the recording data and image data to the server.

[1848] The server receives the audio data and performs noise reduction, and resizes the image data.

[1849] The server analyzes the sound using a generative AI model and determines it as "anxiety." At the same time, it analyzes the user's emotions using an emotion engine and determines it as "surprise."

[1850] The server generates a message saying "The dog is anxious and the user is surprised" and generates a virtual anxious cry and an image of a surprised expression.

[1851] The terminal receives the data from the server and notifies the user.

[1852] The user can check the analysis results and generated data in the app.

[1853] Example 2: Cat images and user sentiment analysis

[1854] Users take photos of their cats using a dedicated app and then use the camera to capture their own images.

[1855] The terminal transmits the photo data and image data to the server.

[1856] The server receives the photo data and image data and performs any necessary pre-processing.

[1857] The server analyzes the image using a generative AI model and determines whether it is "stressed." At the same time, it analyzes the user's emotions using an emotion engine and determines whether it is "joy."

[1858] The server generates a message saying "The cat is stressed, and the user is happy," and generates an image of a stressed expression and an image of a happy expression.

[1859] The terminal receives the data from the server and notifies the user.

[1860] The user checks the notification in the app and sees the analysis results and generated data.

[1861] In this way, the system of the present invention provides an easy-to-understand picture of the pet and user's condition, making it easier to meet the pet's needs and improving the happiness and health of both the pet and the owner.

[1862] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1863] Step 1: Getting and sending data

[1864] The user launches a dedicated app installed on their smartphone. They tap the record button in the app to record their pet's cries, or tap the camera button to take photos or videos of their pet. When recording or taking a photo, the user's facial expressions are also captured by the camera.

[1865] Input: Pet sounds, images, videos, user face images

[1866] Output: Audio files, image files, video files, user face image data

[1867] The terminal sends this data to the server using the HTTP protocol.

[1868] Step 2: Receiving and storing data

[1869] The server receives data sent from the device via a RESTful API. The received data is stored in local storage or cloud storage (e.g., AWS S3). Once stored, metadata is recorded in a database (e.g., MySQL) for data management.

[1870] Input: Audio files, image files, video files, user face image data

[1871] Output: Saved files, metadata

[1872] Step 3: Preprocessing the data

[1873] The server picks up the stored audio data and applies noise reduction algorithms (e.g., FFT-based filtering). For image and video data, it resizes and converts the format using the OpenCV library. Specifically, images are resized to 640x480 pixels and videos are encoded in MP4 format.

[1874] Input: saved audio files, image files, video files

[1875] Output: Pre-processed audio data, resized and formatted image and video data

[1876] Step 4: Analyze the data

[1877] The server inputs the preprocessed data into a generative AI model (e.g., TensorFlow model) for analysis. An emotion analysis model is applied to the audio data to assign labels such as "joy," "anxiety," and "anger," and an object detection model is applied to the image and video data to determine the pet's health status (e.g., "stressed," "relaxed," etc.).

[1878] Input: Preprocessed audio data, image and video data

[1879] Output: Audio data with emotion labels, image and video data with health status labels

[1880] Step 5: Analyzing user sentiment

[1881] The server receives the user's camera footage and audio data and analyzes it using an emotion engine (e.g., Microsoft Azure Cognitive Services). As a result, it obtains the user's emotion label (e.g., "joy," "sadness," "surprise," etc.).

[1882] Input: User's face image data, voice data

[1883] Output: User data with emotion labels

[1884] Step 6: Message Generation

[1885] The server generates messages using a natural language generation engine (e.g., GPT-4) based on the analysis results of the pet and the user. It also uses generative AI to create virtual pet sounds and facial expressions. The DALL-E model is used to generate the virtual data.

[1886] Input: Analysis results of pet and user (emotion labels, health status labels, etc.)

[1887] Output: Generated natural language messages, virtual sounds and facial expressions

[1888] Step 7: Data submission and notification

[1889] The server sends the generated message and virtual data to the API endpoint of the dedicated app.

[1890] The device receives the data from the server, stores it locally, and displays a notification to the user through the app. When the user opens the app, they can view the analysis results and the generated virtual data.

[1891] Input: Generated natural language messages, virtual sounds, and facial expressions

[1892] Output: Notification to user terminal, display of analysis results and generated data

[1893] This system can comprehensively analyze the emotional states of both the pet and the user, and provide information in a format that is easy for the owner to understand.

[1894] (Application example 2)

[1895] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1896] Pet owners spend a lot of time and effort understanding their pets' emotions and health conditions and providing appropriate care. It is also difficult to find products and services that suit their pets' conditions. As a result, pet needs cannot be met quickly and accurately, which can lead to a decline in the happiness and health of both the pet and the owner. To solve this problem, a system is needed that analyzes the conditions of both the pet and the owner and suggests appropriate products and services based on that analysis.

[1897] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data receiving means for analyzing the pet's cries, images, and videos; a data preprocessing means for preprocessing the received data into an analyzable format; an analysis means for determining the pet's emotions and health condition using a generative AI model based on the preprocessed data; a proposal means for proposing products and services based on the pet's condition and the user's emotions determined by the analysis means; a message generation means for generating an explanatory message in natural language based on the analysis results; and a transmission means for transmitting the generated message and the generated pet's cries and images to the user's terminal. This allows the owner to quickly understand the pet's condition and easily select and purchase appropriate products and services based on that information.

[1898] The "data receiving means" is a device or system for receiving data related to the sounds, images, and videos of pets, as well as the user's emotions.

[1899] "Data preprocessing means" refers to processing devices and algorithms that convert received data into an analyzable format. This means performs preprocessing such as noise removal and format conversion.

[1900] "Analysis means" refers to a device or system that uses a generative AI model to determine the pet's emotions and health status based on pre-processed data.

[1901] The "suggestion means" is a device or system that suggests appropriate products or services based on the state of the pet and the user's emotions determined by the analysis means.

[1902] The "message generating means" is a device or system that generates a message explaining the state of the pet and the user in natural language based on the analysis results.

[1903] The "transmission means" is a device or system that transmits the generated message and the sounds and images of the pet to the user's terminal.

[1904] A "generative AI model" is an artificial intelligence model used to analyze a pet's emotions and health condition from received data.

[1905] "Natural language" refers to a language format that humans use on a daily basis, and is used to provide analysis results to users in an intuitively understandable format.

[1906] "Products and services" is a general term for items and support activities suggested depending on the pet's condition and the user's emotions.

[1907] The present invention provides a system for analyzing the state of a pet and the emotions of a user, and for proposing appropriate products and services based on the results. This system has the following configuration.

[1908] First, the user installs a dedicated application on their smartphone and captures their pet's cries, images, and videos. This activates the data receiving means, which then transmits the captured data to the server. This data receiving means includes a standard smartphone camera and microphone.

[1909] Next, the server preprocesses the received data using a data preprocessing means. This preprocessing means includes, for example, noise removal from audio data, resizing of image data, and format conversion. This process typically uses a programming language such as Python, with libraries such as OpenCV or Librosa.

[1910] Once preprocessed, the data is analyzed using a generative AI model. The analysis methods include analyzing pet sounds to determine emotions, and analyzing images and videos to determine health and stress levels. The generative AI model is built using deep learning frameworks such as TensorFlow and PyTorch.

[1911] Furthermore, the analysis method analyzes not only the pet's condition but also the user's emotions. The user captures their own facial expressions with their smartphone camera, and the data is sent to the server. The emotion engine determines the user's emotions based on this data. This analysis uses Facial Recognition API and natural language processing engines (NLP engines), among others.

[1912] Based on the analysis results, the suggestion unit operates to suggest products and services according to the condition of the pet and the user. These suggestions include items to reduce stress for pets and relaxation items for owners.

[1913] The message generation means then generates a message in natural language based on the analysis results. For example, "If your pet is feeling stressed, why not try purchasing some relaxation products?" This generated message, along with the sound and image of the pet, is then sent back to the user's smartphone via the transmission means. The user receives this as a notification and can check it on the application.

[1914] For example, the following prompts might be fed into a generative AI model:

[1915] Output analysis results based on the pet's image and the user's emotion. For example, if the pet is stressed and the user is surprised, return information including the emotion label. Also, suggest products (such as relaxation goods) appropriate for the pet's condition.

[1916] This system allows users to quickly understand the condition of their pets and easily select and purchase appropriate products and services based on that information, ultimately improving the happiness and health of both pets and their owners.

[1917] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1918] Step 1:

[1919] The user uses a smartphone to capture the sound, images, and videos of their pet, and then sends the data to the server through a dedicated application. The input data is the sound, images, and videos of the pet, and the output is the raw data sent to the server. This step uses the smartphone's built-in camera and microphone.

[1920] Step 2:

[1921] The server receives data sent by the user using a data receiving means. The input data is the unprocessed data sent from the smartphone, and the output is the received data. In this step, communication means such as HTTP requests and WebSockets are used.

[1922] Step 3:

[1923] The server uses data preprocessing to convert the received data into an analyzable format. The input data is the received raw data, and the output is the preprocessed data. This step involves noise removal for audio data, resizing of image data, and format conversion. Specifically, the Python libraries OpenCV and Librosa are used.

[1924] Step 4:

[1925] The server analyzes the preprocessed data using a generative AI model. The input data is the preprocessed data, and the output is the analysis results showing the pet's emotions and health status. In this step, a generative AI model built using TensorFlow and PyTorch is used. The model assigns emotion labels such as "joy" or "anxiety" to the pet's cries, and determines its state, such as "stress" or "health," from images and videos.

[1926] Step 5:

[1927] The server analyzes the user's emotions using an emotion engine. The input data is the user's facial expression data captured by the smartphone camera, and the output is an analysis result indicating the user's emotions. In this step, the Facial Recognition API and a natural language processing engine (NLP engine) are used.

[1928] Step 6:

[1929] The server runs a suggestion unit that suggests products and services based on the analysis results. The input data are the analysis results of the pet and the user, and the output is a list of suggested products and services. In this step, a recommendation engine is used to select products and services suitable for the user. If the pet is feeling stressed, for example, relaxation goods will be suggested.

[1930] Step 7:

[1931] The server uses a message generation means to generate a message in natural language. The input data is the analysis results and a list of suggestions, and the output is a message to be sent to the user. The generated message will explain the pet's condition and the suggested products and services. In this step, an NLP engine is used to generate an intuitive and easy-to-understand message.

[1932] Step 8:

[1933] The server operates a sending means that sends the generated message and the generated pet sound and image to the user's device. The input data is the generated message and the pet sound and image, and the output is a notification that arrives on the user's smartphone. In this step, a push notification service (e.g., Firebase Cloud Messaging) is used.

[1934] Step 9:

[1935] The user checks the notification through the application and views the analysis results and suggested products and services. The input data is the notification sent from the server, and the output is the information provided to the user. In this step, a dedicated application on the smartphone is used, allowing the user to easily check the information and select and purchase the suggested products and services.

[1936] By linking these processing steps, the user can quickly grasp the condition of their pet and provide appropriate care based on that information.

[1937] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1938] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1939] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1940] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1941] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1942] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1943] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1944] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1945] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1946] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1947] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1948] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1949] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1950] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1951] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1952] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1953] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1954] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1955] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1956] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1957] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1958] The following is further disclosed regarding the above embodiment.

[1959] (Claim 1)

[1960] A data receiving means for analyzing pet sounds, images, and videos;

[1961] data preprocessing means for preprocessing the received data into an analyzable format;

[1962] An analytical method to determine the pet's emotions and health status using a generative AI model based on pre-processed data, and

[1963] a message generation means for generating an explanatory message in natural language based on the analysis result;

[1964] a transmitting means for transmitting the generated message and the generated pet's cry and image to a user's terminal;

[1965] A system including:

[1966] (Claim 2)

[1967] 2. The system according to claim 1, wherein the analyzing means includes means for analyzing the cries of the pet and determining emotions such as joy and anxiety.

[1968] (Claim 3)

[1969] 2. The system according to claim 1, wherein the analysis means includes means for analyzing images and videos of the pet and determining stress, health condition, etc.

[1970] "Example 1"

[1971] (Claim 1)

[1972] A data receiving means for analyzing pet sounds, images, and videos;

[1973] data preprocessing means for preprocessing the received data into an analyzable format;

[1974] An analytical method to determine the pet's emotions and health status using a generative AI model based on pre-processed data, and

[1975] a message generation means for generating an explanatory message in natural language based on the analysis result;

[1976] a transmitting means for transmitting the generated message and the generated pet's cry and image to a user's terminal;

[1977] A means for the user's device to transmit the recorded and photographed data through a dedicated app;

[1978] A means for the server to receive the data and perform pre-processing such as noise removal and format conversion;

[1979] A means to determine emotion labels and health status using generative AI models based on pre-processed data;

[1980] A means for generating a virtual pet's cry and facial expression image based on the analysis results;

[1981] A system including:

[1982] (Claim 2)

[1983] 2. The system according to claim 1, wherein the analyzing means includes means for analyzing the cries of the pet and determining emotions such as joy and anxiety.

[1984] (Claim 3)

[1985] 2. The system according to claim 1, wherein the analysis means includes means for analyzing images and videos of the pet and determining stress, health condition, etc.

[1986] "Application Example 1"

[1987] (Claim 1)

[1988] A data receiving means for analyzing pet sounds, images, and videos;

[1989] data preprocessing means for preprocessing the received data into an analyzable format;

[1990] An analytical method to determine the pet's emotions and health status using a generative AI model based on pre-processed data, and

[1991] a message generation means for generating an explanatory message in natural language based on the analysis result;

[1992] a transmitting means for transmitting the generated message and the generated pet's cry and image to a user's terminal;

[1993] an information output means for providing pet care information within a physical retail environment accessed by a user;

[1994] A system including:

[1995] (Claim 2)

[1996] 2. The system according to claim 1, wherein the analyzing means includes means for analyzing the cries of the pet and determining emotions such as joy and anxiety.

[1997] (Claim 3)

[1998] 2. The system according to claim 1, wherein the analysis means includes means for analyzing images and videos of the pet and determining stress, health condition, etc.

[1999] "Example 2: Combining Emotion Engines"

[2000] (Claim 1)

[2001] A data receiving means for analyzing pet sounds, images, and videos;

[2002] data preprocessing means for preprocessing the received data into an analyzable format;

[2003] An analytical method to determine the pet's emotions and health status using a generative AI model based on pre-processed data, and

[2004] A means for receiving a camera image and a voice of a user and analyzing the user's emotions using an emotion engine;

[2005] a message generation means for generating an explanatory message in natural language based on the analysis result;

[2006] a transmitting means for transmitting the generated message and the generated pet's cry and image to a user's terminal;

[2007] A system including:

[2008] (Claim 2)

[2009] 2. The system according to claim 1, wherein the analyzing means includes means for analyzing the cries of the pet and determining emotions such as joy and anxiety.

[2010] (Claim 3)

[2011] 2. The system according to claim 1, wherein the analysis means includes means for analyzing images and videos of the pet and determining stress, health condition, etc.

[2012] "Application example 2 when combining emotion engines"

[2013] (Claim 1)

[2014] A data receiving means for analyzing pet sounds, images, and videos;

[2015] data preprocessing means for preprocessing the received data into an analyzable format;

[2016] An analytical method to determine the pet's emotions and health status using a generative AI model based on pre-processed data, and

[2017] a suggestion means for suggesting products and services based on the state of the pet and the user's emotions determined by the analysis means;

[2018] a message generation means for generating an explanatory message in natural language based on the analysis result;

[2019] a transmitting means for transmitting the generated message and the generated pet's cry and image to a user's terminal;

[2020] A system including:

[2021] (Claim 2)

[2022] 2. The system according to claim 1, wherein the analyzing means includes means for analyzing the cries of the pet and determining emotions such as joy and anxiety.

[2023] (Claim 3)

[2024] 2. The system according to claim 1, wherein the analysis means includes means for analyzing images and videos of the pet and determining stress, health condition, etc. [Explanation of symbols]

[2025] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A data receiving means for analyzing pet sounds, images, and videos; data preprocessing means for preprocessing the received data into an analyzable format; An analytical method to determine the pet's emotions and health status using a generative AI model based on pre-processed data, and a message generation means for generating an explanatory message in natural language based on the analysis result; a transmitting means for transmitting the generated message and the generated pet's cry and image to a user's terminal; A system including:

2. 2. The system according to claim 1, wherein said analyzing means includes means for analyzing the sounds of the pet and determining emotions such as joy and anxiety.

3. The system according to claim 1 , wherein the analyzing means includes means for analyzing images or videos of the pet and determining stress, health condition, etc.

Images