System

Abstract

A system is provided.SOLUTION: Collecting a barking sound of a dog or a cat, collecting a behavior of the dog or the cat, preprocessing the collected barking sound, labeling the preprocessed barking sound, training a generative AI model by using the labeled barking sound, recording the barking sound or the behavior of the dog or the cat through an input apparatus used by a user, and analyzing the barking sound or the behavior of the dog or the cat acquired from the input apparatus of the user by using the generative AI model; A system, comprising: means for generating a translation result; means for sending the generated translation result to a terminal of a user; and means for displaying the translation result on the terminal of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Pets are important to many people, and owners always want to know what their pets are thinking and wanting. However, previous dog and cat translators had low translation accuracy, making it difficult to communicate accurately. Therefore, there is a demand for the development of a more accurate dog and cat translation system. The present invention aims to solve this problem by providing a system that can translate the sounds and actions of dogs and cats with high accuracy. [Means for solving the problem]

[0005] In this invention, data on the meows and behavior of dogs and cats is collected, preprocessed, and then labeled and trained using a generative AI model. The meows and behavior of dogs and cats are recorded and filmed via an input device (e.g., smart glasses) used by the user, and the data is analyzed on a server. The translation results based on the generative AI model are sent to the user's device and displayed, allowing the user to understand the intentions and emotions of their pet. Specifically, the system includes the following means:

[0006] 1. A means of collecting data on dog and cat meows.

[0007] 2. A means of collecting data on dog and cat behavior.

[0008] 3. A means of preprocessing the collected data.

[0009] 4. A means of labeling the preprocessed data.

[0010] 5. A means to train generative AI models with labeled data.

[0011] 6. A means of recording the sounds and behavior of dogs and cats through an input device used by the user.

[0012] 7. A means of analyzing data obtained from the user's input device using a generative AI model to generate translation results.

[0013] 8. A means for transmitting the generated translation results to the user's terminal.

[0014] 9. A means of displaying the translation results on the user's device.

[0015] This will allow owners to better communicate with their pets and understand their pets' emotions and intentions.

[0016] The "means for collecting data on dog and cat meows" is a mechanism for obtaining dog and cat meows from the Internet and other audio data sources and storing them in a database.

[0017] The "means for collecting data on dog and cat behavior" is a mechanism for recording the behavior of dogs and cats using image data and video data and storing that data in a database.

[0018] The "preprocessing means" is a mechanism for performing processes such as noise removal, normalization, and resizing on collected data, and converting it into a format suitable for model training.

[0019] The "labeling means" is a mechanism for automatically or manually assigning tags indicating specific meanings such as "hungry," "want to play," or "relaxed" to preprocessed data.

[0020] A "means for training a generative AI model" is a mechanism for training a neural network or other machine learning model with labeled data and storing the model in a database.

[0021] An "input device" is a device that a user uses to record or record the sounds or behavior of a dog or cat, including, for example, smart glasses.

[0022] The "analysis means" is a mechanism that uses a generative AI model to analyze data sent from the user's input device and translate the meaning of the sounds and behaviors.

[0023] The "translation result sending means" is a mechanism for sending the translation results generated by the generative AI model to the user's terminal.

[0024] The "display means" is a mechanism for displaying the translation results visually or audibly on the user's terminal. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0026] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0027] First, the terms used in the following description will be explained.

[0028] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0029] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0030] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0031] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0033] [First embodiment]

[0034] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0035] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0036] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0038] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0041] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0043] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0044] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0045] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0046] System Overview

[0047] This invention is a system that translates the meows and behaviors of dogs and cats. It uses generative AI to analyze data on dog and cat meows and behaviors, and outputs the translation results as language. This system allows users to deepen their communication with their pets and understand their emotions and intentions.

[0048] Program Details

[0049] Data collection

[0050] The server collects dog and cat meow and behavior data from online audio databases, social media, video sharing sites, etc. This allows for the collection of a sufficient amount of data to be used as training data.

[0051] 1. The server scrapes data from a specific URL.

[0052] 2. The server stores the collected data in a database.

[0053] Data Preprocessing

[0054] The server performs preprocessing on the collected data, such as noise removal, normalization, and resizing.

[0055] 1. The server performs filtering to remove noise from the data.

[0056] 2. The server converts the audio data into a spectrogram and resizes the image data to an appropriate size.

[0057] Labeling

[0058] The server automatically or manually assigns labels to the preprocessed data.

[0059] 1. The server automates labeling using a machine learning model.

[0060] 2. Human experts review and correct the labels, if necessary.

[0061] Training generative AI models

[0062] The server uses the labeled data to train a generative AI model.

[0063] 1. The server splits the dataset into training and validation sets.

[0064] 2. The server trains the model using a neural network algorithm.

[0065] 3. The server evaluates the trained model on the validation dataset to confirm its accuracy.

[0066] Translation system execution

[0067] The user uses an input device (e.g., smart glasses) to record and film the sounds and behavior of their dog or cat.

[0068] 1. The user turns on the smart glasses and records the sounds and behavior of their pet.

[0069] 2. The device sends the recording data to the server.

[0070] The server analyzes the received data and generates a translation result.

[0071] 1. The server analyzes the data using a generative AI model and generates translation results.

[0072] 2. The server sends the translation results to the device.

[0073] The user checks the translation results through the terminal.

[0074] 1. The device displays or speaks the translation results.

[0075] 2. The user checks the results on the device's display screen or audio output.

[0076] Specific examples

[0077] Example 1: Translating a dog's bark

[0078] 1. A user records a dog barking using smart glasses.

[0079] 2. The device sends the data to the server.

[0080] 3. The server analyzes based on the model that "the dog is barking because it is hungry."

[0081] 4. The server sends the translation results to the device.

[0082] 5. The user checks the translation results on the smart glasses.

[0083] Example 2: Translating cat behavior

[0084] 1. A user uses smart glasses to record the cat's behavior.

[0085] 2. The device sends the data to the server.

[0086] 3. Based on the model, the server analyzes that "the cat is jumping because it wants to play."

[0087] 4. The server sends the translation results to the device.

[0088] 5. The user checks the translation results on the smart glasses.

[0089] By using this system, users can more easily understand what their pets are thinking and want, allowing them to build better relationships with their pets.

[0090] The processing flow will be explained below.

[0091] Step 1: Data collection

[0092] The server scrapes audio data of dog and cat meows, as well as image and video data of their behavior, from the Internet.

[0093] The server stores the scraped data in a database.

[0094] The server categorizes and organizes the data, preparing it for further processing.

[0095] Step 2: Data Preprocessing

[0096] The server performs noise removal filtering on the collected audio data.

[0097] The server converts the audio data into a spectrogram.

[0098] The image and video data collected by the server is resized and converted into a format suitable for analysis.

[0099] The server stores the preprocessed data in a database.

[0100] Step 3: Data Labeling

[0101] The server applies an automatic labeling algorithm to the preprocessed data.

[0102] The server assigns labels such as "hungry" or "want to play" to the bird's cry data.

[0103] The server assigns labels such as "relaxed" or "excited" to the behavioral data.

[0104] If necessary, the server also performs manual labeling by an expert.

[0105] Step 4: Training the generative AI model

[0106] The server splits the labeled data into a training dataset and a validation dataset.

[0107] The server uses the training dataset with a neural network to train a generative AI model.

[0108] The server evaluates the trained model on the validation dataset to confirm accuracy and performance.

[0109] The server will improve or retrain the model as needed based on the evaluation results.

[0110] Step 5: Collect input data

[0111] The user uses smart glasses to record the sounds of a dog or cat meowing.

[0112] The user uses smart glasses to record the behavior of their dog or cat.

[0113] The device sends the recording data to the server.

[0114] Step 6: Data analysis and translation

[0115] The server receives the data sent from the terminal.

[0116] The server uses the generative AI model to analyze the incoming data.

[0117] The server generates a translation result based on the analysis, such as "This sound indicates that the bird is hungry."

[0118] Step 7: Send the translation

[0119] The server transmits the generated translation result to the terminal.

[0120] The device receives the translation result.

[0121] Step 8: Displaying the translation results

[0122] The terminal outputs the translation results on the display screen.

[0123] The user checks the displayed translation results and understands the pet's intentions and emotions.

[0124] Example 1

[0125] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0126] In recent years, there has been an increasing demand for understanding the emotions and intentions of pets from their vocalizations and behaviors. However, current technologies lack the means for owners to gain a deeper understanding of their pets' states. Therefore, an effective system for improving communication between pets and their owners is needed.

[0127] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0128] In this invention, the server includes means for collecting audio data, means for collecting video data, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording audio and video via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to the user's terminal, and means for displaying the translation result on the user's terminal, thereby enabling a user to understand the emotions and intentions of their pet and deepen communication with their pet.

[0129] "Audio data" means data recorded in electronic form of animal calls or other sounds.

[0130] "Video data" refers to data that records the behavior and appearance of animals in electronic format.

[0131] "Preprocessing" is the process of performing processes such as noise removal, normalization, and resizing on collected data to prepare it in a format suitable for analysis.

[0132] A "label" is a tag or annotation that is attached to data to identify the type or state of the subject of analysis.

[0133] A "generative AI model" is an artificial intelligence model that is trained based on collected, preprocessed, and labeled data to provide appropriate translation results or predictions for specific input data.

[0134] An "input device" refers to a device that a user uses to record their pet's sounds and behavior, such as smart glasses or a smartphone.

[0135] "Analysis" refers to the act of analyzing input data using a generative AI model and inferring translation results and intent based on that data.

[0136] The "translation result" is information obtained from the analyzed data that describes the animal's emotions and intentions.

[0137] "Terminal" refers to the device used by the user to check the translation results, such as smart glasses or smartphones.

[0138] The present invention relates to a system for analyzing audio data and video data to understand the emotions and intentions of animals. How the present invention can be implemented will be described in detail below.

[0139] System Overview

[0140] The system primarily consists of a server, a terminal, and an input device used by the user. The server collects audio and video data, preprocesses it, and labels it. The labeled data is then used to train a generative AI model and analyze new data obtained from the user's input device. The terminal receives the translation results and displays them to the user.

[0141] Hardware and software used

[0142] Server: Collects data, preprocesses it, labels it, trains generative AI models, and analyzes it. It uses software libraries and tools such as Python, TensorFlow, PyTorch, Librosa, BeautifulSoup, OpenCV, and MySQL.

[0143] Terminal: A device used by a user as an input device, such as smart glasses or a smartphone, that records audio and video and transmits the data to a server.

[0144] Input device: Smart glasses or smartphones are examples of input devices that record pet sounds and behavior in real time.

[0145] Processing flow

[0146] Data collection: The server collects audio and video data from online audio databases, social media sites, and video sharing sites. The data is scraped using Python's BeautifulSoup library and stored in a MySQL database.

[0147] Data preprocessing: The server performs preprocessing on the collected data, such as noise removal, normalization, and resizing. It uses the Python Librosa library to remove noise from the audio data, Matplotlib to generate spectrograms, and OpenCV to resize the images.

[0148] Labeling: The server automatically assigns labels using machine learning models, and experts review and correct the labels as needed. A classifier is trained from existing labeled data using frameworks such as TensorFlow, and new data is labeled.

[0149] Training the generative AI model: The server uses the labeled data to train the generative AI model. It uses Python's TensorFlow or PyTorch to build a neural network algorithm and utilizes a GPU for fast training.

[0150] Translation system execution: The user uses an input device such as smart glasses to record and record the sounds and behavior of their pet. The device compresses this data in real time and uploads it to the server via HTTPS. The server then uses a generative AI model to analyze the received data, generate translation results, and send them to the device. Finally, the device displays or outputs the translation results as audio, allowing the user to confirm the results.

[0151] Specific examples

[0152] Example 1: Translating a dog's bark

[0153] 1. A user uses the smart glasses to record the sound of a dog barking, for example, by pressing a button to record the sound of a dog barking.

[0154] 2. The device sends the data to the server.

[0155] 3. Based on the generated AI model, the server analyzes that "the dog is barking because it is hungry."

[0156] 4. The server sends the translation results to the device.

[0157] 5. The user checks the translation results on the smart glasses. The message "I'm hungry" appears on the display and the user reads it.

[0158] Example 2: Translating cat behavior

[0159] 1. A user uses smart glasses to record the cat's behavior, for example, pressing a button to record the cat jumping.

[0160] 2. The device sends the data to the server. The recorded data is compressed in real time and sent to the server.

[0161] 3. Based on the generated AI model, the server analyzes that "the cat is jumping because it wants to play."

[0162] 4. The server sends the translation results to the device.

[0163] 5. The user checks the translation results on the smart glasses. The message "I want to play" appears on the display and the user reads it.

[0164] In this way, by using the system of the present invention, the user can understand the emotions and intentions of their pet and deepen communication with their pet.

[0165] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0166] Step 1:

[0167] Data collection

[0168] The server scrapes data from a specific URL, for example, using Python's BeautifulSoup library to retrieve audio and video files from social media posts.

[0169] Input: A specific URL list.

[0170] Output: Collected audio and video data.

[0171] What it does: The server analyzes the web page, extracts links to audio and video files, and downloads those files.

[0172] Step 2:

[0173] Database storage

[0174] The server stores the collected data in a database. The MySQL database stores the paths and metadata of audio and video files.

[0175] Input: Collected audio and video data.

[0176] Output: Path and metadata of the data stored in the database.

[0177] What happens: The server inserts each file's path and associated metadata (date and time of acquisition, original URL, etc.) into a MySQL database.

[0178] Step 3:

[0179] Data Preprocessing

[0180] The server performs filtering to remove noise. The Librosa library in Python is used to remove noise from the audio data.

[0181] Input: Collected audio data.

[0182] Output: Clear audio data with noise removed.

[0183] What it does: The server uses Librosa to load the audio data and applies a specific noise filter to remove unwanted noise.

[0184] The server converts the audio data into a spectrogram and resizes the image data to an appropriate size. The spectrogram is generated using Python's Matplotlib and the image data is resized using OpenCV.

[0185] Input: Denoised audio data and collected video data.

[0186] Output: Spectrogram image and resized video data.

[0187] What it does: Audio data is converted to a spectrogram using Matplotlib, and video data is resized to a consistent size using OpenCV.

[0188] Step 4:

[0189] Labeling

[0190] The server automates labeling using machine learning models, training classifiers using TensorFlow and other tools to label new data.

[0191] Input: Preprocessed audio and video data (spectrogram images and resized video data).

[0192] Output: Labeled data.

[0193] What it does: The server runs a labeling algorithm to automatically assign an appropriate label (e.g., dog barking, cat jumping, etc.) to each piece of data.

[0194] The server then has human experts review and correct the labels as needed.

[0195] Input: Auto-labeled data.

[0196] Output: Verified and corrected labeled data.

[0197] Specific operation: Experts review the labels of each data through a web interface and correct them if necessary.

[0198] Step 5:

[0199] Training generative AI models

[0200] The server splits the dataset into training and validation sets using the train_test_split function from scikit-learn.

[0201] Input: A labeled dataset.

[0202] Output: Split training and validation data.

[0203] Specific operation: The server splits the labeled dataset into training and validation sets in an 80:20 ratio.

[0204] The server trains the model using a neural network algorithm. The model is built using TensorFlow or PyTorch, and trained quickly using a GPU.

[0205] Input: Training data.

[0206] Output: A trained generative AI model.

[0207] What it does: The server configures the model architecture and iteratively inputs data to tune the model parameters.

[0208] The server evaluates the trained model on a validation dataset to confirm its accuracy.

[0209] Input: A trained generative AI model, and validation data.

[0210] Output: Model performance metrics (accuracy, sensitivity, specificity, etc.).

[0211] Specific operation: The server evaluates the model's performance using the validation dataset and logs the results.

[0212] Step 6:

[0213] Translation system execution

[0214] A user uses an input device such as smart glasses to record and film the sounds and behavior of a dog or cat.

[0215] Input: Pet sounds and behavior.

[0216] Output: Recorded data.

[0217] Specific operation: The user activates the audio and video recording function of the smart glasses and records the sounds and behavior of their pet.

[0218] The device sends the audio and video data to the server, where it is compressed in real time and uploaded to the server via HTTPS.

[0219] Input: Recorded data.

[0220] Output: Data uploaded to the server.

[0221] Specific operation: The device compresses the recorded data and sends it to the server via the Internet.

[0222] The server uses the generative AI model to analyze the incoming data and generate translation results.

[0223] Input: Data uploaded to the server.

[0224] Output: The translation result.

[0225] Specific operation: The server inputs the received data into the generated AI model, analyzes the pet's emotions and intentions, and generates translation results.

[0226] The server generates the translation and sends it to the device, which returns it in JSON format and reformats it for display.

[0227] Input: Translation result.

[0228] Output: The translation result sent to your device.

[0229] Specific operation: The server packages the translation results in JSON format and sends them to the device.

[0230] The device displays or speaks the translation results. The results are displayed as text on the smart glasses display and spoken through the speakers.

[0231] Input: The translation result sent to the device.

[0232] Output: The displayed or spoken translation result.

[0233] Specific operations: The device displays the translation results to the user by displaying the text on a display or by using a speech synthesis system to output the results aloud from a speaker.

[0234] The user checks the results on the device's display screen or audio output.

[0235] Input: The displayed or spoken translation result.

[0236] Output: The translation result as recognized by the user.

[0237] Specific operation: The user checks the translation results by looking at the display screen of the smart glasses and listening to the voice output.

[0238] (Application example 1)

[0239] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0240] Conventional systems that translate the meows and behavior of dogs and cats collect data on dog and cat meows and behavior from online audio databases, social media, video sharing sites, etc., and analyze it using a generative AI model. However, current systems cannot display the pet's emotions and needs on a display device in real time, making it difficult for users to immediately check the pet's condition. This makes it difficult for, for example, pet shop employees and customers to quickly understand the pet's condition, making it difficult to provide appropriate responses and services.

[0241] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0242] In this invention, the server includes means for collecting data on dog and cat meows, means for collecting data on dog and cat behavior, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording dog and cat meows and behaviors via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to the user's terminal, means for displaying the translation result on the user's terminal, and means for displaying the pet's meows and behaviors on a display device in real time, thereby enabling the user to visually check the status of their pet in real time and provide appropriate responses and services immediately.

[0243] "Means for collecting data on dog and cat meows" refers to devices or functions that record sounds made by dogs and cats and store them for analysis.

[0244] "Means for collecting data on the behavior of dogs and cats" refers to devices or functions that record the behavior of dogs and cats and collect data to analyze their behavior.

[0245] The "means for preprocessing the collected data" refers to a device or function that performs preprocessing such as noise removal and data normalization on the collected audio and video data.

[0246] The "means for assigning labels to preprocessed data" refers to a device or function that automatically or manually assigns labels to preprocessed data according to their contents.

[0247] A "means for training a generative AI model using labeled data" is a device or function that uses labeled data to train an AI model and improve its accuracy.

[0248] "Means for recording the meows and behavior of dogs and cats through an input device used by a user" refers to a device or function that records the meows and behavior of dogs and cats through an input device used by a user, such as smart glasses.

[0249] "Means for analyzing data obtained from a user's input device using a generative AI model and generating a translation result" refers to a device or function that analyzes data sent from an input device using an AI model and translates the results as text.

[0250] The "means for transmitting the generated translation result to the user's terminal" refers to a device or function that transmits the translation result generated by the server to the user's smart glasses, smartphone, etc.

[0251] The "means for displaying the translation result on the user's terminal" refers to a device or function that visually displays the translation result on the user's terminal.

[0252] The "means for displaying the sounds and behavior of a pet on a display device in real time" refers to a device or function that captures the sounds and behavior of a pet in real time and immediately displays the results on a display device.

[0253] System Overview

[0254] This invention is a system that translates the sounds and behaviors of dogs and cats. Using a generative AI model, this data is analyzed and the pet's emotions and intentions are conveyed to the user. In particular, this invention has the function of displaying the pet's sounds and behaviors in real time on a display device, allowing pet shop employees and customers to instantly check the pet's status.

[0255] Program Details

[0256] Data collection methods

[0257] The server collects dog and cat meow and behavior data from online audio databases, social media, video sharing sites, etc. This allows for the collection of a sufficient amount of data to be used as training data.

[0258] Data preprocessing measures

[0259] The server performs preprocessing on the collected data, such as noise removal and normalization. For example, it filters the audio data to remove noise and converts it into a spectrogram. It also resizes the image data to an appropriate size.

[0260] Labeling method

[0261] The server automatically labels the preprocessed data using a machine learning model, and experts review and correct the labels if necessary.

[0262] A means of training generative AI models

[0263] The server trains a generative AI model using the labeled data, splits the dataset into training and validation sets, and trains the model using a neural network algorithm.The model is then evaluated on the validation dataset to confirm its accuracy.

[0264] Translation system implementation

[0265] Users use input devices such as smart glasses to record and record the sounds and behavior of their dogs and cats. The device then sends the recorded data to a server, which then analyzes the data using a generative AI model to generate a translation. The resulting translation is then sent back to the device, where the user can view it on their smart glasses or smartphone.

[0266] Real-time display means

[0267] The server analyzes the pet's sounds and behavior in real time and immediately displays the results on a display device, allowing pet shop staff and customers to understand the pet's condition on the spot.

[0268] Specific examples

[0269] Example 1: Translating a dog's bark

[0270] The user records the sound of a dog barking using smart glasses. The device sends the data to the server, which then uses the model to analyze the sound and concludes, "The dog is barking because it is hungry." The server then sends the translation results to the device, where the user can check them on the smart glasses.

[0271] Example 2: Translating cat behavior

[0272] The user records the cat's behavior using smart glasses. The device sends the data to the server, which then analyzes it based on the model and concludes that "the cat is jumping because it wants to play." The server then sends the translation results to the device, where the user can check them on the smart glasses.

[0273] Prompt Sentence Examples

[0274] "Send the data to the Pet Communication Guide app and output the translation result as 'This dog is hungry.'"

[0275] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0276] Step 1:

[0277] The user wears smart glasses and records the sounds and behavior of their dog or cat. As input, the user operates the audio / video recording function of the smart glasses to capture the voice and video of their pet. As output, the audio / video data is stored in the smart glasses' internal memory.

[0278] Step 2:

[0279] The device sends audio and video data from the user's smart glasses to the server. The audio and video data stored on the smart glasses is used as input. The data is sent to the server via the Internet. The data is received by the server as output.

[0280] Step 3:

[0281] The server performs preprocessing on the received data. It uses the transmitted audio and video data as input. The server applies noise reduction filters and data normalization algorithms, converts audio data into spectrograms, and resizes video data. As output, it produces preprocessed audio and video data.

[0282] Step 4:

[0283] The server assigns labels to the preprocessed data. As input, it uses preprocessed audio and video data and automatically labels them using a generative AI model. As output, a labeled dataset is generated.

[0284] Step 5:

[0285] The server trains a generative AI model using the labeled data. It uses the labeled dataset as input and applies a neural network algorithm. It splits the dataset into training and validation parts, and trains and evaluates the model. The output is a trained generative AI model.

[0286] Step 6:

[0287] The server uses a generative AI model to analyze data obtained from the user's input device and generate translation results. Raw audio and video data sent by the user is used as input. Analysis is performed using the generative AI model, and translation results of the pet's cries and behavior are output. The output is a translated text.

[0288] Step 7:

[0289] The server sends the translation results to the user's device. The translation result text generated by the server is used as input. The data is sent via the Internet to the user's smart glasses or smartphone. The translation results arrive at the device as output.

[0290] Step 8:

[0291] The user's device displays the translation result. It uses the translation result text received from the server as input. It shows the translation result to the user through the device's display or audio output. As an output, it generates a state that allows the user to confirm the translation result visually or audibly.

[0292] Step 9:

[0293] The server displays the pet's cries and behavior in real time on a display device. Recorded and video data acquired in real time is used as input. A generative AI model instantly analyzes the data and generates translation results. The real-time translation results are displayed on a display device as output.

[0294] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0295] System Overview

[0296] This invention combines a system that translates the meows and behaviors of dogs and cats with an emotion engine that recognizes the user's emotions, further deepening two-way communication between pets and users. It uses generative AI to analyze dog and cat meows and behavior data and output the translation results as language, and also provides a function to recognize the user's emotions in real time and generate appropriate responses.

[0297] Program Details

[0298] Data collection

[0299] The server scrapes the internet for audio data of cats and dogs' meows, as well as image and video data of their behavior, collecting a sufficient amount of data to be used as training data.

[0300] 1. The server scrapes data from a specific URL.

[0301] 2. The server stores the collected data in a database.

[0302] Data Preprocessing

[0303] The server performs preprocessing such as noise removal, normalization, and resizing on the collected data.

[0304] 1. The server performs filtering to remove noise from the data.

[0305] 2. The server converts the audio data into a spectrogram and resizes the image data to an appropriate size.

[0306] Labeling

[0307] The server automatically or manually assigns labels to the preprocessed data.

[0308] 1. The server automates labeling using a machine learning model.

[0309] 2. Human experts review and correct the labels, if necessary.

[0310] Training generative AI models

[0311] The server uses the labeled data to train a generative AI model.

[0312] 1. The server splits the dataset into training and validation sets.

[0313] 2. The server trains the model using a neural network algorithm.

[0314] 3. The server evaluates the trained model on the validation dataset to confirm its accuracy.

[0315] Collecting user input data

[0316] The user uses an input device (e.g., smart glasses) to record and film the sounds of a dog or cat meowing.

[0317] 1. The user turns on the smart glasses and records the sounds and behavior of their pet.

[0318] 2. The device sends the recording data to the server.

[0319] Data analysis and translation

[0320] The server analyzes the received data and generates translation results using a generative AI model.

[0321] 1. The server analyzes the data using a generative AI model and generates translation results.

[0322] 2. The server uses the analysis results to translate the dog or cat's emotions and intentions.

[0323] Additional processing for the emotion engine

[0324] The server analyzes the user's input data using an emotion engine that recognizes the user's emotions.

[0325] 1. The device captures the user's voice and facial expressions in real time using a camera and microphone.

[0326] 2. The server analyzes the user's facial expressions and voice data to recognize their emotional state, such as "happy," "surprised," or "sad."

[0327] 3. The server generates an appropriate response based on the user's emotional state.

[0328] Sending translation results and responses

[0329] The server sends the generated translation result and a response based on the user's emotions to the terminal.

[0330] 1. The server sends the translation results and emotional responses to the device in real time.

[0331] 2. The device displays the translation results and response.

[0332] Displaying translation results and responses

[0333] The user can check the translation results and emotion-based responses through the device.

[0334] 1. The device outputs the translation results on the display screen.

[0335] 2. The device displays an appropriate response to the user in voice or text.

[0336] 3. The user can check the results on the device's display screen or audio output, making it easier to understand the pet's intentions and emotions.

[0337] Specific examples

[0338] Example 1: Translating dog barks and responding to user emotions

[0339] 1. A user records a dog barking using smart glasses.

[0340] 2. The device sends the data to the server.

[0341] 3. Based on the model, the server analyzes that "the dog is barking because it is hungry."

[0342] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "happy."

[0343] 5. The server generates a response saying "The dog says it's hungry" along with the translation result, "You look happy too."

[0344] 6. The server sends the translation results and response to the device.

[0345] 7. The user checks the translation results and responses on the smart glasses.

[0346] Example 2: Translating cat behavior and responding to user emotions

[0347] 1. A user uses smart glasses to record the cat's behavior.

[0348] 2. The device sends the data to the server.

[0349] 3. Based on the model, the server analyzes that "the cat is jumping because it wants to play."

[0350] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "surprised."

[0351] 5. The server generates a response with the translation result "The cat wants to play" and "You look surprised too."

[0352] 6. The server sends the translation results and response to the device.

[0353] 7. The user checks the translation results and responses on the smart glasses.

[0354] By using this system, users can more easily understand what their pets are thinking and wanting, deepening two-way communication with them. Furthermore, by providing responses based on the user's emotions, it can help to improve the relationship between owners and their pets.

[0355] The processing flow will be explained below.

[0356] Step 1: Data collection

[0357] The server scrapes audio data of dog and cat meows, as well as image and video data of their behavior, from the Internet.

[0358] The server stores the scraped data in a database.

[0359] The server categorizes and organizes the data, preparing it for further processing.

[0360] Step 2: Data Preprocessing

[0361] The server performs noise removal filtering on the collected audio data.

[0362] The server converts the audio data into a spectrogram.

[0363] The image and video data collected by the server is resized and converted into a format suitable for analysis.

[0364] The server stores the preprocessed data in a database.

[0365] Step 3: Data Labeling

[0366] The server applies an automatic labeling algorithm to the preprocessed data.

[0367] The server assigns labels such as "hungry" or "want to play" to the bird's cry data.

[0368] The server assigns labels such as "relaxed" or "excited" to the behavioral data.

[0369] If necessary, the server also performs manual labeling by an expert.

[0370] Step 4: Training the generative AI model

[0371] The server splits the labeled data into a training dataset and a validation dataset.

[0372] The server uses the training dataset with a neural network to train a generative AI model.

[0373] The server evaluates the trained model on the validation dataset to confirm accuracy and performance.

[0374] The server will improve or retrain the model as needed based on the evaluation results.

[0375] Step 5: Train your Emotion Engine

[0376] The server collects the user's voice and facial expression data and uses this as training data.

[0377] The server trains a machine learning model to recognize user emotions.

[0378] The server evaluates the trained emotion recognition model on validation data to confirm accuracy and performance.

[0379] Step 6: Collecting User Input Data

[0380] The user uses smart glasses to record the sounds of a dog or cat meowing.

[0381] The user uses smart glasses to record the behavior of their dog or cat.

[0382] The device sends the recording data to the server.

[0383] Step 7: Data analysis and translation

[0384] The server receives the data sent from the terminal.

[0385] The server uses the generative AI model to analyze the incoming data.

[0386] The server generates a translation result such as "This sound indicates that the bird is hungry" as an analysis result.

[0387] Step 8: Recognizing User Emotions

[0388] The device acquires the user's voice and facial expression data in real time.

[0389] The server analyzes the user's facial expressions and voice data to recognize their emotional state, such as "happy" or "surprised."

[0390] The server generates an appropriate response based on the user's emotional state.

[0391] Step 9: Sending translation results and responses

[0392] The server sends the generated translation result and a response based on the user's emotions to the terminal.

[0393] The terminal receives the translation result and the response.

[0394] Step 10: Viewing Translation Results and Responses

[0395] The terminal outputs the translation results on the display screen.

[0396] The terminal displays an appropriate response to the user in voice or text.

[0397] The user can check the results on the device's display screen and audio output to understand the pet's intentions and emotions.

[0398] Example 2

[0399] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0400] To achieve deeper communication between pets and their owners, interactions that take into account not only the pet's cries and actions but also the owner's emotions are necessary. However, while existing systems can translate pets' actions and cries, they lack the functionality to provide responses that take into account the owner's emotions. This leaves the problem of insufficient two-way communication between pets and their owners.

[0401] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0402] In this invention, the server includes means for collecting data on dog and cat meows, means for collecting data on dog and cat behavior, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording dog and cat meows and behaviors via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to a user's terminal, means for displaying the translation result on the user's terminal, means for recognizing the user's emotions, and means for generating a response based on the user's emotions, thereby enabling a response that takes into account not only the pet's meows and behaviors but also the owner's emotions.

[0403] "Data on dog and cat meows" is information that records the waveforms and temporal characteristics of sounds made by dogs and cats.

[0404] "Dog and cat behavior data" is information that records the movements and behavior of dogs and cats in the form of images or videos.

[0405] "Means of collection" refers to technologies for scraping data from the Internet and devices for acquiring data using cameras and microphones.

[0406] "Preprocessing" is the process of converting collected data into a format that is easier to analyze through processes such as noise removal, data normalization, and resizing.

[0407] "Labeling means" refers to technology that automatically or manually assigns tags that indicate meaning or category to data.

[0408] A "generative AI model" is an artificial intelligence algorithm that uses supervised learning based on large amounts of data to perform specific tasks (such as translation).

[0409] "Input devices used by users" refers to devices such as smart glasses and smartphones that users use to record and film the sounds and behavior of their dogs and cats.

[0410] The "means of analyzing and generating translation results" refers to a technology that uses a generative AI model to analyze the received data, translate the emotions and intentions of the dog or cat, and output the results as text.

[0411] "Means for sending to the user's device" refers to the protocols and technologies used to send the generated translation results to the device used by the user.

[0412] "Means for displaying the translation result on the user's terminal" refers to a function for visually displaying the translation result on a user interface or display.

[0413] "Means for recognizing user emotions" refers to technology that analyzes the user's facial expressions and voice and identifies their emotional state (happiness, surprise, sadness, etc.).

[0414] "Means for generating responses based on user emotions" refers to algorithms or technologies that generate appropriate reactions or messages in response to the recognized user emotions.

[0415] The present invention is a system that translates the sounds and behaviors of dogs and cats and recognizes the user's emotions, enabling two-way communication with pets. Specific methods for implementing this system are described below.

[0416] Data collection

[0417] First, the server collects dog and cat meow and behavior data from the Internet. This is done by scraping data using Python's BeautifulSoup library or Scrapy framework. This collection process is targeted at specific URLs, and the collected data is stored in a database (e.g., MySQL or MongoDB).

[0418] Data Preprocessing

[0419] The server then performs preprocessing on the collected data. This preprocessing includes denoising the audio data, normalizing the data, and resizing the image data. The SciPy and Librosa libraries are used for audio data processing, and the OpenCV library is used for image data processing. Specifically, the server converts the audio data into a spectrogram and resizes the image data to an appropriate size (e.g., 224x224 pixels).

[0420] Labeling

[0421] The server labels the preprocessed data using a pre-trained machine learning model (e.g., ResNet or YOLO). Labeling is automated, but human experts can review and correct the labels if necessary through a web interface.

[0422] Training generative AI models

[0423] The server trains a generative AI model using the labeled data. The dataset is split into training and validation sets (e.g., 80% for training and 20% for validation), and the model is trained using a neural network algorithm (e.g., TensorFlow or PyTorch). Evaluation is performed on the validation dataset to confirm its accuracy.

[0424] Collecting user input data

[0425] Users use smart glasses (e.g., Google Glass) to record and film the sounds and behavior of their dogs and cats. The device then transmits the data from the smart glasses to a server in real time. Protocols such as HTTP and WebSocket are used for transmission.

[0426] Data analysis and translation

[0427] The server analyzes the received data and generates a translation result using a generative AI model. For example, it may generate a result such as "The dog is barking because it is hungry." Based on this analysis result, the dog or cat's emotions and intentions are output as text.

[0428] Additional processing for the emotion engine

[0429] The server uses an emotion analysis engine (e.g., Microsoft Azure's Face API) to recognize the user's emotions. This engine recognizes the user's emotional state (e.g., "happy," "surprised," "sad") from their voice and facial expressions. The server then generates an appropriate response based on this emotional data.

[0430] Sending and displaying translation results and responses

[0431] The server sends the translation results and a response based on the user's emotions to the device. The device then displays these results visually through a user interface (e.g., a smart glasses display). It also outputs audio, if necessary. Based on this information, the user can understand their pet's emotions and intentions and communicate more appropriately.

[0432] Specific examples

[0433] Example 1: Translating dog barks and responding to user emotions

[0434] 1. A user records a dog barking using smart glasses.

[0435] 2. The device sends the data to the server.

[0436] 3. Based on the model, the server analyzes that "the dog is barking because it is hungry."

[0437] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "happy."

[0438] 5. The server generates a response saying "The dog says it's hungry" along with the translation result, "You look happy too."

[0439] 6. The server sends the translation results and response to the device.

[0440] 7. The user checks the translation results and responses on the smart glasses.

[0441] Example 2: Translating cat behavior and responding to user emotions

[0442] 1. A user uses smart glasses to record the cat's behavior.

[0443] 2. The device sends the data to the server.

[0444] 3. Based on the model, the server analyzes that "the cat is jumping because it wants to play."

[0445] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "surprised."

[0446] 5. The server generates a response with the translation "The cat wants to play" and "You look surprised too."

[0447] 6. The server sends the translation results and response to the device.

[0448] 7. The user checks the translation results and responses on the smart glasses.

[0449] This allows users to understand what their pets are thinking and wanting, deepening two-way communication with them. Furthermore, by providing responses based on the user's emotions, the relationship between pet and owner can be further improved.

[0450] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0451] Step 1:

[0452] Data collection

[0453] The server scrapes dog and cat meow data and behavior data (images and videos) from specific URLs on the Internet. In this step, data is collected using libraries such as BeautifulSoup and Scrapy. The input is a specific URL, and the output is the collected audio data, images, and video data. The server stores the collected data in a database (e.g., MySQL or MongoDB).

[0454] Step 2:

[0455] Data Preprocessing

[0456] The server performs preprocessing on the collected data, including noise removal, data normalization, and resizing. In this step, SciPy and Librosa libraries are used for filtering and spectrogram conversion of audio data. OpenCV is used for image and video data resizing. The input is scraped data, and the output is preprocessed data.

[0457] Step 3:

[0458] Labeling

[0459] The server assigns labels to the preprocessed data. In this step, automatic labeling is performed using a machine learning model (e.g., ResNet or YOLO). To improve the accuracy of the labeling, some of the data is checked and corrected by human experts via a web interface. The input is the preprocessed data, and the output is the labeled data.

[0460] Step 4:

[0461] Training generative AI models

[0462] The server trains a generative AI model using the labeled data. The dataset is split into training and validation sets (e.g., 80% for training and 20% for validation). Training is performed using a neural network algorithm (e.g., TensorFlow or PyTorch), with a set learning rate and number of epochs. The input is the labeled data, and the output is a trained AI model.

[0463] Step 5:

[0464] User-generated data collection

[0465] The user uses smart glasses (e.g., Google Glass) to record and record the sounds and behavior of their dog or cat in real time. The device sends the data from the smart glasses to a server. The input is the data obtained from the smart glasses, and the output is the data sent to the server.

[0466] Step 6:

[0467] Data analysis and translation

[0468] The server analyzes the received data and generates translation results using a trained generative AI model. In this step, the data is input into the generative AI model, and the analysis results output translated text that indicates the pet's emotions and intentions. The input is the data sent from the device, and the output is the translated text.

[0469] Step 7:

[0470] Additional processing for the emotion engine

[0471] The server uses an emotion engine (e.g., Microsoft Azure's Face API) to recognize the user's emotions. The device collects the user's voice and facial expressions in real time through a camera and microphone and sends them to the server. The server analyzes this data to recognize the user's emotional state and generate an appropriate response. The input is the user's facial and voice data, and the output is their emotional state and a response based on it.

[0472] Step 8:

[0473] Sending and displaying translation results and responses

[0474] The server sends the generated translation results and responses based on the user's emotions to the device. The device displays these results visually through a user interface (e.g., a smart glasses display) and, if necessary, provides audio output. The input is the translation results and responses sent from the server, and the output is the translation results and responses displayed to the user.

[0475] (Application example 2)

[0476] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0477] Understanding pet sounds and behaviors is challenging for pet owners. Therefore, there is a need for a way to understand what pets are thinking and needing. There is also a need for a system that can not only translate pet intentions and emotions, but also understand the owner's emotions and respond accordingly, thereby further deepening communication between pets and their owners.

[0478] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data on dog and cat meows, means for collecting data on dog and cat behavior, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording dog and cat meows and behaviors via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to the user's terminal, means for displaying the translation result on the user's terminal, means for analyzing input data using an emotion engine that recognizes the user's emotions, means for generating a response corresponding to the user's emotions based on the analysis result, means for transmitting and displaying the response corresponding to the user's emotions on the user's terminal, and means for providing interactive content corresponding to the dog or cat's intentions. This makes it easier to understand the intentions and emotions of pets and provides appropriate responses corresponding to the owner's emotions, thereby deepening communication between pets and their owners.

[0479] "Means for collecting dog and cat meow data" refers to a device or process for recording pet sounds and electronically recording that data.

[0480] "Means for collecting data on dog and cat behavior" refers to devices or processes for monitoring pet movements using cameras, sensors, etc., and collecting behavioral data.

[0481] "Measures for pre-processing collected data" refers to processes for removing noise from audio and video data, normalizing the data, and converting it into a format suitable for analysis.

[0482] "Means for labeling preprocessed data" refers to the process of assigning appropriate tags or labels to data after preprocessing so that it is possible to identify what each piece of data represents.

[0483] "Means for training a generative AI model with labeled data" refers to the process of using a labeled dataset to train a generative AI model to accurately analyze pet sounds and behaviors.

[0484] "Means for recording the sounds and behavior of dogs and cats through input devices used by users" refers to the process of recording the sounds and behavior of pets using input devices such as smartphones, tablets, and smart glasses.

[0485] "Means for analyzing data obtained from a user's input device using a generative AI model and generating a translation result" refers to the process of receiving data sent from a user's input device, analyzing that data using a generative AI model, and translating the pet's intentions and emotions.

[0486] "Means for sending the generated translation results to the user's device" refers to the process of sending the translation results obtained by the generative AI model to the user's smartphone or tablet device.

[0487] "Means for displaying translation results on the user's device" refers to the process of displaying the pet's intentions and emotions analyzed by the generative AI model on the user's smartphone or tablet.

[0488] "Means for analyzing input data using an emotion engine that recognizes the user's emotions" refers to the process of capturing the user's facial expressions and voice using a camera or microphone, and analyzing them to recognize the user's emotional state.

[0489] The "means for generating a response according to the user's emotions based on the analysis results" refers to a process for generating an optimal response according to the recognized emotional state of the user.

[0490] "Means for transmitting and displaying a response according to the user's emotions to the user's device" refers to the process of transmitting the generated response to the user's smartphone or tablet device and displaying it.

[0491] "Means for providing interactive content according to the intentions of dogs and cats" refers to a process for providing users with educational and entertainment content according to the intentions and state of their pets.

[0492] The present invention is a system that translates the sounds and behaviors of dogs and cats and recognizes the user's emotions, thereby deepening communication between the user and the pet. The system of the present invention is implemented through the following steps.

[0493] System Overview

[0494] The system analyzes pet sounds and behaviors and provides multiple ways to recognize the user's emotions. The system then analyzes the collected data using a generative AI model to generate translation results and responses based on the user's emotions.

[0495] Data collection

[0496] The server collects the meows and behavioral data of dogs and cats. Pet sounds are recorded using a microphone, and behavioral data is recorded using a camera. These data are stored in cloud storage and later pre-processed.

[0497] Data Preprocessing

[0498] The collected data undergoes preprocessing such as noise removal and normalization. Specifically, the server uses a noise filter to remove noise from the audio data, converts the audio data into a spectrogram, and resizes the image data to an appropriate size.

[0499] Labeling

[0500] Once preprocessing is complete, the data is labeled with the meaning and intent of each piece of data. The server automatically assigns the labels using machine learning algorithms, and experts review and correct the labels as needed.

[0501] Training generative AI models

[0502] A generative AI model is trained using the labeled data. The dataset is split into training and validation datasets, and a neural network algorithm is used. Once trained, the model's accuracy is verified on the validation dataset.

[0503] Collecting user input data

[0504] Users can record and record the sounds and behavior of their dogs and cats in real time using input devices such as smart glasses or smartphones, and the recorded data is sent from the device to a server.

[0505] Data analysis and translation

[0506] The server analyzes the received data and generates translation results using a generative AI model. Based on the analysis results, the emotions and intentions of the dog or cat are translated.

[0507] Additional processing for the emotion engine

[0508] To recognize the user's emotions, the device captures the user's facial expressions and voice in real time. The server analyzes this data and recognizes the user's emotional state, such as "happy," "surprised," or "sad." An appropriate response is generated based on the user's recognized emotional state.

[0509] Sending translation results and responses

[0510] The generated translation results and responses are sent to the user's device in real time, where they can check the results on the device's display screen or as audio output, making it easier to understand their pet's intentions and emotions.

[0511] Providing interactive content

[0512] The system provides users with interactive content tailored to their pet's state. For example, if a pet expresses a desire to play, the system suggests activities for the user to play with the pet.

[0513] Specific examples

[0514] The user uses smart glasses to record the sound of their dog barking and sends it to the server. Using a generative AI model, the server analyzes the sound and concludes that "the dog is barking because it is hungry," and recognizes the user's facial expression and voice as "happy." The server then generates a translation result of "The dog says it is hungry," along with a response of "You look happy, too." The user can then check the translation result and response on the smart glasses, instantly understanding their pet's intentions and emotions.

[0515] Prompt Sentence Examples

[0516] As a concrete example, we can use pet sound data and label pairs in the following format:

[0517] "Dog barking: hungry" "Cat behavior: wants to play"

[0518] This allows users to gain a deeper understanding of their pet's intentions and emotions, enabling richer communication.

[0519] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0520] Step 1: Data collection

[0521] The server scrapes the internet for audio data of cats and dogs' meows, as well as image and video data of their behavior. The data is retrieved from specific URLs and stored in a database. This allows for the collection of a large amount of data that can be used as training data.

[0522] Input: Specific URL

[0523] Data processing: scraping and data storage

[0524] Output: Audio, image, and video data stored in a database

[0525] Step 2: Data Preprocessing

[0526] The server performs preprocessing such as noise removal, normalization, and resizing on the data collected. Noise is removed from the audio data and it is converted into a spectrogram. Image data is resized to an appropriate size to make it easier to process.

[0527] Input: Collected audio, image, and video data

[0528] Data processing: noise removal, normalization, resizing

[0529] Output: Preprocessed data

[0530] Step 3: Labeling

[0531] The server automatically or manually labels the preprocessed data. Automatic labeling is performed using a machine learning model, and experts review and correct the labels as needed.

[0532] Input: Preprocessed data

[0533] Data processing: automatic labeling, expert review

[0534] Output: Labeled data

[0535] Step 4: Training the generative AI model

[0536] The server uses the labeled data to train a generative AI model, splits the dataset into training and validation sets, and trains the model using a neural network algorithm.The accuracy of the model is then verified on the validation dataset.

[0537] Input: Labeled data

[0538] Data processing: data division, model training, model evaluation

[0539] Output: A trained generative AI model

[0540] Step 5: Collecting User Input Data

[0541] Users use smart glasses or a smartphone to record the sounds and behavior of their dogs and cats, and the recorded data is sent from the device to a server.

[0542] Input: Dog and cat meows and behavior (real-time)

[0543] Data processing: audio and video recording, data transmission

[0544] Output: Recorded data sent to the server

[0545] Step 6: Data analysis and translation

[0546] The server analyzes the received data and generates translation results using a generative AI model. Through analysis, it determines the pet's emotions and intentions.

[0547] Input: Audio and video data

[0548] Data processing: Analysis with generative AI models

[0549] Output: Translation result (pet's emotions and intentions)

[0550] Step 7: Further processing of the emotion engine

[0551] To recognize the user's emotions, the device captures the user's facial expressions and voice in real time. The server analyzes this data and recognizes the user's emotional state, such as "happy," "surprised," or "sad."

[0552] Input: User's facial expression and voice data

[0553] Data processing: Analysis using emotion engine

[0554] Output: User's emotional state

[0555] Step 8: Generate a response based on the analysis results

[0556] The server generates an appropriate response based on the user's emotional state. For example, a translation result indicating the pet's intentions is generated, along with a response based on the user's emotions.

[0557] Input: User's emotional state, pet translation results

[0558] Data processing: response generation

[0559] Output: Translation results and responses for the user

[0560] Step 9: Sending the response and translation

[0561] The generated translation results and responses are sent in real time to the user's device, where the user can check the results on the device's display screen or via voice output.

[0562] Input: Translation result and response

[0563] Data processing: Sending results

[0564] Output: Display on user terminal

[0565] Step 10: Serve interactive content

[0566] The system provides users with interactive content that responds to the pet's condition, further enriching the communication between pets and their owners.

[0567] Input: Pet state, user emotional state

[0568] Data processing: Content generation

[0569] Output: Providing interactive content

[0570] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0571] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0572] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0573] [Second embodiment]

[0574] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0575] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0576] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0577] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0578] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0579] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0580] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0581] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0582] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0583] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0584] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0585] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0586] System Overview

[0587] This invention is a system that translates the meows and behaviors of dogs and cats. It uses generative AI to analyze data on dog and cat meows and behaviors, and outputs the translation results as language. This system allows users to deepen their communication with their pets and understand their emotions and intentions.

[0588] Program Details

[0589] Data collection

[0590] The server collects dog and cat meow and behavior data from online audio databases, social media, video sharing sites, etc. This allows for the collection of a sufficient amount of data to be used as training data.

[0591] 1. The server scrapes data from a specific URL.

[0592] 2. The server stores the collected data in a database.

[0593] Data Preprocessing

[0594] The server performs preprocessing on the collected data, such as noise removal, normalization, and resizing.

[0595] 1. The server performs filtering to remove noise from the data.

[0596] 2. The server converts the audio data into a spectrogram and resizes the image data to an appropriate size.

[0597] Labeling

[0598] The server automatically or manually assigns labels to the preprocessed data.

[0599] 1. The server automates labeling using a machine learning model.

[0600] 2. Human experts review and correct the labels, if necessary.

[0601] Training generative AI models

[0602] The server uses the labeled data to train a generative AI model.

[0603] 1. The server splits the dataset into training and validation sets.

[0604] 2. The server trains the model using a neural network algorithm.

[0605] 3. The server evaluates the trained model on the validation dataset to confirm its accuracy.

[0606] Translation system execution

[0607] The user uses an input device (e.g., smart glasses) to record and film the sounds and behavior of their dog or cat.

[0608] 1. The user turns on the smart glasses and records the sounds and behavior of their pet.

[0609] 2. The device sends the recording data to the server.

[0610] The server analyzes the received data and generates a translation result.

[0611] 1. The server analyzes the data using a generative AI model and generates translation results.

[0612] 2. The server sends the translation results to the device.

[0613] The user checks the translation results through the terminal.

[0614] 1. The device displays or speaks the translation results.

[0615] 2. The user checks the results on the device's display screen or audio output.

[0616] Specific examples

[0617] Example 1: Translating a dog's bark

[0618] 1. A user records a dog barking using smart glasses.

[0619] 2. The device sends the data to the server.

[0620] 3. The server analyzes based on the model that "the dog is barking because it is hungry."

[0621] 4. The server sends the translation results to the device.

[0622] 5. The user checks the translation results on the smart glasses.

[0623] Example 2: Translating cat behavior

[0624] 1. A user uses smart glasses to record the cat's behavior.

[0625] 2. The device sends the data to the server.

[0626] 3. Based on the model, the server analyzes that "the cat is jumping because it wants to play."

[0627] 4. The server sends the translation results to the device.

[0628] 5. The user checks the translation results on the smart glasses.

[0629] By using this system, users can more easily understand what their pets are thinking and want, allowing them to build better relationships with their pets.

[0630] The processing flow will be explained below.

[0631] Step 1: Data collection

[0632] The server scrapes audio data of dog and cat meows, as well as image and video data of their behavior, from the Internet.

[0633] The server stores the scraped data in a database.

[0634] The server categorizes and organizes the data, preparing it for further processing.

[0635] Step 2: Data Preprocessing

[0636] The server performs noise removal filtering on the collected audio data.

[0637] The server converts the audio data into a spectrogram.

[0638] The image and video data collected by the server is resized and converted into a format suitable for analysis.

[0639] The server stores the preprocessed data in a database.

[0640] Step 3: Data Labeling

[0641] The server applies an automatic labeling algorithm to the preprocessed data.

[0642] The server assigns labels such as "hungry" or "want to play" to the bird's cry data.

[0643] The server assigns labels such as "relaxed" or "excited" to the behavioral data.

[0644] If necessary, the server also performs manual labeling by an expert.

[0645] Step 4: Training the generative AI model

[0646] The server splits the labeled data into a training dataset and a validation dataset.

[0647] The server uses the training dataset with a neural network to train a generative AI model.

[0648] The server evaluates the trained model on the validation dataset to confirm accuracy and performance.

[0649] The server will improve or retrain the model as needed based on the evaluation results.

[0650] Step 5: Collect input data

[0651] The user uses smart glasses to record the sounds of a dog or cat meowing.

[0652] The user uses smart glasses to record the behavior of their dog or cat.

[0653] The device sends the recording data to the server.

[0654] Step 6: Data analysis and translation

[0655] The server receives the data sent from the terminal.

[0656] The server uses the generative AI model to analyze the incoming data.

[0657] The server generates a translation result based on the analysis, such as "This sound indicates that the bird is hungry."

[0658] Step 7: Send the translation

[0659] The server transmits the generated translation result to the terminal.

[0660] The device receives the translation result.

[0661] Step 8: Displaying the translation results

[0662] The terminal outputs the translation results on the display screen.

[0663] The user checks the displayed translation results and understands the pet's intentions and emotions.

[0664] Example 1

[0665] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0666] In recent years, there has been an increasing demand for understanding the emotions and intentions of pets from their vocalizations and behaviors. However, current technologies lack the means for owners to gain a deeper understanding of their pets' states. Therefore, an effective system for improving communication between pets and their owners is needed.

[0667] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0668] In this invention, the server includes means for collecting audio data, means for collecting video data, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording audio and video via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to the user's terminal, and means for displaying the translation result on the user's terminal, thereby enabling a user to understand the emotions and intentions of their pet and deepen communication with their pet.

[0669] "Audio data" means data recorded in electronic form of animal calls or other sounds.

[0670] "Video data" refers to data that records the behavior and appearance of animals in electronic format.

[0671] "Preprocessing" is the process of performing processes such as noise removal, normalization, and resizing on collected data to prepare it in a format suitable for analysis.

[0672] A "label" is a tag or annotation that is attached to data to identify the type or state of the subject of analysis.

[0673] A "generative AI model" is an artificial intelligence model that is trained based on collected, preprocessed, and labeled data to provide appropriate translation results or predictions for specific input data.

[0674] An "input device" refers to a device that a user uses to record their pet's sounds and behavior, such as smart glasses or a smartphone.

[0675] "Analysis" refers to the act of analyzing input data using a generative AI model and inferring translation results and intent based on that data.

[0676] The "translation result" is information obtained from the analyzed data that describes the animal's emotions and intentions.

[0677] "Terminal" refers to the device used by the user to check the translation results, such as smart glasses or smartphones.

[0678] The present invention relates to a system for analyzing audio data and video data to understand the emotions and intentions of animals. How the present invention can be implemented will be described in detail below.

[0679] System Overview

[0680] The system primarily consists of a server, a terminal, and an input device used by the user. The server collects audio and video data, preprocesses it, and labels it. The labeled data is then used to train a generative AI model and analyze new data obtained from the user's input device. The terminal receives the translation results and displays them to the user.

[0681] Hardware and software used

[0682] Server: Collects data, preprocesses it, labels it, trains generative AI models, and analyzes it. It uses software libraries and tools such as Python, TensorFlow, PyTorch, Librosa, BeautifulSoup, OpenCV, and MySQL.

[0683] Terminal: A device used by a user as an input device, such as smart glasses or a smartphone, that records audio and video and transmits the data to a server.

[0684] Input device: Smart glasses or smartphones are examples of input devices that record pet sounds and behavior in real time.

[0685] Processing flow

[0686] Data collection: The server collects audio and video data from online audio databases, social media sites, and video sharing sites. The data is scraped using Python's BeautifulSoup library and stored in a MySQL database.

[0687] Data preprocessing: The server performs preprocessing on the collected data, such as noise removal, normalization, and resizing. It uses the Python Librosa library to remove noise from the audio data, Matplotlib to generate spectrograms, and OpenCV to resize the images.

[0688] Labeling: The server automatically assigns labels using machine learning models, and experts review and correct the labels as needed. A classifier is trained from existing labeled data using frameworks such as TensorFlow, and new data is labeled.

[0689] Training the generative AI model: The server uses the labeled data to train the generative AI model. It uses Python's TensorFlow or PyTorch to build a neural network algorithm and utilizes a GPU for fast training.

[0690] Translation system execution: The user uses an input device such as smart glasses to record and record the sounds and behavior of their pet. The device compresses this data in real time and uploads it to the server via HTTPS. The server then uses a generative AI model to analyze the received data, generate translation results, and send them to the device. Finally, the device displays or outputs the translation results as audio, allowing the user to confirm the results.

[0691] Specific examples

[0692] Example 1: Translating a dog's bark

[0693] 1. A user uses the smart glasses to record the sound of a dog barking, for example, by pressing a button to record the sound of a dog barking.

[0694] 2. The device sends the data to the server.

[0695] 3. Based on the generated AI model, the server analyzes that "the dog is barking because it is hungry."

[0696] 4. The server sends the translation results to the device.

[0697] 5. The user checks the translation results on the smart glasses. The message "I'm hungry" appears on the display and the user reads it.

[0698] Example 2: Translating cat behavior

[0699] 1. A user uses smart glasses to record the cat's behavior, for example, pressing a button to record the cat jumping.

[0700] 2. The device sends the data to the server. The recorded data is compressed in real time and sent to the server.

[0701] 3. Based on the generated AI model, the server analyzes that "the cat is jumping because it wants to play."

[0702] 4. The server sends the translation results to the device.

[0703] 5. The user checks the translation results on the smart glasses. The message "I want to play" appears on the display and the user reads it.

[0704] In this way, by using the system of the present invention, the user can understand the emotions and intentions of their pet and deepen communication with their pet.

[0705] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0706] Step 1:

[0707] Data collection

[0708] The server scrapes data from a specific URL, for example, using Python's BeautifulSoup library to retrieve audio and video files from social media posts.

[0709] Input: A specific URL list.

[0710] Output: Collected audio and video data.

[0711] What it does: The server analyzes the web page, extracts links to audio and video files, and downloads those files.

[0712] Step 2:

[0713] Database storage

[0714] The server stores the collected data in a database. The MySQL database stores the paths and metadata of audio and video files.

[0715] Input: Collected audio and video data.

[0716] Output: Path and metadata of the data stored in the database.

[0717] What happens: The server inserts each file's path and associated metadata (date and time of acquisition, original URL, etc.) into a MySQL database.

[0718] Step 3:

[0719] Data Preprocessing

[0720] The server performs filtering to remove noise. The Librosa library in Python is used to remove noise from the audio data.

[0721] Input: Collected audio data.

[0722] Output: Clear audio data with noise removed.

[0723] What it does: The server uses Librosa to load the audio data and applies a specific noise filter to remove unwanted noise.

[0724] The server converts the audio data into a spectrogram and resizes the image data to an appropriate size. The spectrogram is generated using Python's Matplotlib and the image data is resized using OpenCV.

[0725] Input: Denoised audio data and collected video data.

[0726] Output: Spectrogram image and resized video data.

[0727] What it does: Audio data is converted to a spectrogram using Matplotlib, and video data is resized to a consistent size using OpenCV.

[0728] Step 4:

[0729] Labeling

[0730] The server automates labeling using machine learning models, training classifiers using TensorFlow and other tools to label new data.

[0731] Input: Preprocessed audio and video data (spectrogram images and resized video data).

[0732] Output: Labeled data.

[0733] What it does: The server runs a labeling algorithm to automatically assign an appropriate label (e.g., dog barking, cat jumping, etc.) to each piece of data.

[0734] The server then has human experts review and correct the labels as needed.

[0735] Input: Auto-labeled data.

[0736] Output: Verified and corrected labeled data.

[0737] Specific operation: Experts review the labels of each data through a web interface and correct them if necessary.

[0738] Step 5:

[0739] Training generative AI models

[0740] The server splits the dataset into training and validation sets using the train_test_split function from scikit-learn.

[0741] Input: A labeled dataset.

[0742] Output: Split training and validation data.

[0743] Specific operation: The server splits the labeled dataset into training and validation sets in an 80:20 ratio.

[0744] The server trains the model using a neural network algorithm. The model is built using TensorFlow or PyTorch, and trained quickly using a GPU.

[0745] Input: Training data.

[0746] Output: A trained generative AI model.

[0747] What it does: The server configures the model architecture and iteratively inputs data to tune the model parameters.

[0748] The server evaluates the trained model on a validation dataset to confirm its accuracy.

[0749] Input: A trained generative AI model, and validation data.

[0750] Output: Model performance metrics (accuracy, sensitivity, specificity, etc.).

[0751] Specific operation: The server evaluates the model's performance using the validation dataset and logs the results.

[0752] Step 6:

[0753] Translation system execution

[0754] A user uses an input device such as smart glasses to record and film the sounds and behavior of a dog or cat.

[0755] Input: Pet sounds and behavior.

[0756] Output: Recorded data.

[0757] Specific operation: The user activates the audio and video recording function of the smart glasses and records the sounds and behavior of their pet.

[0758] The device sends the audio and video data to the server, where it is compressed in real time and uploaded to the server via HTTPS.

[0759] Input: Recorded data.

[0760] Output: Data uploaded to the server.

[0761] Specific operation: The device compresses the recorded data and sends it to the server via the Internet.

[0762] The server uses the generative AI model to analyze the incoming data and generate translation results.

[0763] Input: Data uploaded to the server.

[0764] Output: The translation result.

[0765] Specific operation: The server inputs the received data into the generated AI model, analyzes the pet's emotions and intentions, and generates translation results.

[0766] The server generates the translation results and sends them to the device, which returns them in JSON format and reformats them for display.

[0767] Input: Translation result.

[0768] Output: The translation result sent to your device.

[0769] Specific operation: The server packages the translation results in JSON format and sends them to the device.

[0770] The device displays or speaks the translation results. The smart glasses display the results as text and output the results as speech from the speakers.

[0771] Input: The translation result sent to the device.

[0772] Output: The displayed or spoken translation result.

[0773] Specific operations: The device displays the translation results to the user by displaying the text on a display or by using a speech synthesis system to output the results aloud from a speaker.

[0774] The user checks the results on the device's display screen or audio output.

[0775] Input: The displayed or spoken translation result.

[0776] Output: The translation result as recognized by the user.

[0777] Specific operation: The user checks the translation results by looking at the display screen of the smart glasses and listening to the voice output.

[0778] (Application example 1)

[0779] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0780] Conventional systems that translate the meows and behavior of dogs and cats collect data on dog and cat meows and behavior from online audio databases, social media, video sharing sites, etc., and analyze it using a generative AI model. However, current systems cannot display the pet's emotions and needs on a display device in real time, making it difficult for users to immediately check the pet's condition. This makes it difficult for, for example, pet shop employees and customers to quickly understand the pet's condition, making it difficult to provide appropriate responses and services.

[0781] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0782] In this invention, the server includes means for collecting data on dog and cat meows, means for collecting data on dog and cat behavior, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording dog and cat meows and behaviors via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to the user's terminal, means for displaying the translation result on the user's terminal, and means for displaying the pet's meows and behaviors on a display device in real time, thereby enabling the user to visually check the status of their pet in real time and provide appropriate responses and services immediately.

[0783] "Means for collecting data on dog and cat meows" refers to devices or functions that record sounds made by dogs and cats and store them for analysis.

[0784] "Means for collecting data on the behavior of dogs and cats" refers to devices or functions that record the behavior of dogs and cats and collect data to analyze their behavior.

[0785] The "means for preprocessing the collected data" refers to a device or function that performs preprocessing such as noise removal and data normalization on the collected audio and video data.

[0786] The "means for assigning labels to preprocessed data" refers to a device or function that automatically or manually assigns labels to preprocessed data according to their contents.

[0787] A "means for training a generative AI model using labeled data" is a device or function that uses labeled data to train an AI model and improve its accuracy.

[0788] "Means for recording the meows and behavior of dogs and cats through an input device used by a user" refers to a device or function that records the meows and behavior of dogs and cats through an input device used by a user, such as smart glasses.

[0789] "Means for analyzing data obtained from a user's input device using a generative AI model and generating a translation result" refers to a device or function that analyzes data sent from an input device using an AI model and translates the results as text.

[0790] The "means for transmitting the generated translation result to the user's terminal" refers to a device or function that transmits the translation result generated by the server to the user's smart glasses, smartphone, etc.

[0791] The "means for displaying the translation result on the user's terminal" refers to a device or function that visually displays the translation result on the user's terminal.

[0792] The "means for displaying the sounds and behavior of a pet on a display device in real time" refers to a device or function that captures the sounds and behavior of a pet in real time and immediately displays the results on a display device.

[0793] System Overview

[0794] This invention is a system that translates the sounds and behaviors of dogs and cats. Using a generative AI model, this data is analyzed and the pet's emotions and intentions are conveyed to the user. In particular, this invention has the function of displaying the pet's sounds and behaviors in real time on a display device, allowing pet shop employees and customers to instantly check the pet's status.

[0795] Program Details

[0796] Data collection methods

[0797] The server collects dog and cat meow and behavior data from online audio databases, social media, video sharing sites, etc. This allows for the collection of a sufficient amount of data to be used as training data.

[0798] Data preprocessing measures

[0799] The server performs preprocessing on the collected data, such as noise removal and normalization. For example, it filters the audio data to remove noise and converts it into a spectrogram. It also resizes the image data to an appropriate size.

[0800] Labeling method

[0801] The server automatically labels the preprocessed data using a machine learning model, and experts review and correct the labels if necessary.

[0802] A means of training generative AI models

[0803] The server trains a generative AI model using the labeled data, splits the dataset into training and validation sets, and trains the model using a neural network algorithm.The model is then evaluated on the validation dataset to confirm its accuracy.

[0804] Translation system implementation

[0805] Users use input devices such as smart glasses to record and record the sounds and behavior of their dogs and cats. The device then sends the recorded data to a server, which then analyzes the data using a generative AI model to generate a translation. The resulting translation is then sent back to the device, where the user can view it on their smart glasses or smartphone.

[0806] Real-time display means

[0807] The server analyzes the pet's sounds and behavior in real time and immediately displays the results on a display device, allowing pet shop staff and customers to understand the pet's condition on the spot.

[0808] Specific examples

[0809] Example 1: Translating a dog's bark

[0810] The user records the sound of a dog barking using smart glasses. The device sends the data to the server, which then uses the model to analyze the sound and concludes, "The dog is barking because it is hungry." The server then sends the translation results to the device, where the user can check them on the smart glasses.

[0811] Example 2: Translating cat behavior

[0812] The user records the cat's behavior using smart glasses. The device sends the data to the server, which then analyzes it based on the model and concludes that "the cat is jumping because it wants to play." The server then sends the translation results to the device, where the user can check them on the smart glasses.

[0813] Prompt Sentence Examples

[0814] "Send the data to the Pet Communication Guide app and output the translation result as 'This dog is hungry.'"

[0815] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0816] Step 1:

[0817] The user wears smart glasses and records the sounds and behavior of their dog or cat. As input, the user operates the audio / video recording function of the smart glasses to capture the voice and video of their pet. As output, the audio / video data is stored in the smart glasses' internal memory.

[0818] Step 2:

[0819] The device sends audio and video data from the user's smart glasses to the server. The audio and video data stored on the smart glasses is used as input. The data is sent to the server via the Internet. The data is received by the server as output.

[0820] Step 3:

[0821] The server performs preprocessing on the received data. It uses the transmitted audio and video data as input. The server applies noise reduction filters and data normalization algorithms, converts audio data into spectrograms, and resizes video data. As output, it produces preprocessed audio and video data.

[0822] Step 4:

[0823] The server assigns labels to the preprocessed data. As input, it uses preprocessed audio and video data and automatically labels them using a generative AI model. As output, a labeled dataset is generated.

[0824] Step 5:

[0825] The server trains a generative AI model using the labeled data. It uses the labeled dataset as input and applies a neural network algorithm. It splits the dataset into training and validation parts, and trains and evaluates the model. The output is a trained generative AI model.

[0826] Step 6:

[0827] The server uses a generative AI model to analyze data obtained from the user's input device and generate translation results. Raw audio and video data sent by the user is used as input. Analysis is performed using the generative AI model, and translation results of the pet's cries and behavior are output. The output is a translated text.

[0828] Step 7:

[0829] The server sends the translation results to the user's device. The translation result text generated by the server is used as input. The data is sent via the Internet to the user's smart glasses or smartphone. The translation results arrive at the device as output.

[0830] Step 8:

[0831] The user's device displays the translation result. It uses the translation result text received from the server as input. It shows the translation result to the user through the device's display or audio output. As an output, it generates a state that allows the user to confirm the translation result visually or audibly.

[0832] Step 9:

[0833] The server displays the pet's cries and behavior in real time on a display device. Recorded and video data acquired in real time is used as input. A generative AI model instantly analyzes the data and generates translation results. The real-time translation results are displayed on a display device as output.

[0834] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0835] System Overview

[0836] This invention combines a system that translates the meows and behaviors of dogs and cats with an emotion engine that recognizes the user's emotions, further deepening two-way communication between pets and users. It uses generative AI to analyze dog and cat meows and behavior data and output the translation results as language, and also provides a function to recognize the user's emotions in real time and generate appropriate responses.

[0837] Program Details

[0838] Data collection

[0839] The server scrapes the internet for audio data of cats and dogs' meows, as well as image and video data of their behavior, collecting a sufficient amount of data to be used as training data.

[0840] 1. The server scrapes data from a specific URL.

[0841] 2. The server stores the collected data in a database.

[0842] Data Preprocessing

[0843] The server performs preprocessing such as noise removal, normalization, and resizing on the collected data.

[0844] 1. The server performs filtering to remove noise from the data.

[0845] 2. The server converts the audio data into a spectrogram and resizes the image data to an appropriate size.

[0846] Labeling

[0847] The server automatically or manually assigns labels to the preprocessed data.

[0848] 1. The server automates labeling using a machine learning model.

[0849] 2. Human experts review and correct the labels, if necessary.

[0850] Training generative AI models

[0851] The server uses the labeled data to train a generative AI model.

[0852] 1. The server splits the dataset into training and validation sets.

[0853] 2. The server trains the model using a neural network algorithm.

[0854] 3. The server evaluates the trained model on the validation dataset to confirm its accuracy.

[0855] Collecting user input data

[0856] The user uses an input device (e.g., smart glasses) to record and film the sounds of a dog or cat meowing.

[0857] 1. The user turns on the smart glasses and records the sounds and behavior of their pet.

[0858] 2. The device sends the recording data to the server.

[0859] Data analysis and translation

[0860] The server analyzes the received data and generates translation results using a generative AI model.

[0861] 1. The server analyzes the data using a generative AI model and generates translation results.

[0862] 2. The server uses the analysis results to translate the dog or cat's emotions and intentions.

[0863] Additional processing for the emotion engine

[0864] The server analyzes the user's input data using an emotion engine that recognizes the user's emotions.

[0865] 1. The device captures the user's voice and facial expressions in real time using a camera and microphone.

[0866] 2. The server analyzes the user's facial expressions and voice data to recognize their emotional state, such as "happy," "surprised," or "sad."

[0867] 3. The server generates an appropriate response based on the user's emotional state.

[0868] Sending translation results and responses

[0869] The server sends the generated translation result and a response based on the user's emotions to the terminal.

[0870] 1. The server sends the translation results and emotional responses to the device in real time.

[0871] 2. The device displays the translation results and response.

[0872] Displaying translation results and responses

[0873] The user can check the translation results and emotion-based responses through the device.

[0874] 1. The device outputs the translation results on the display screen.

[0875] 2. The device displays an appropriate response to the user in voice or text.

[0876] 3. The user can check the results on the device's display screen or audio output, making it easier to understand the pet's intentions and emotions.

[0877] Specific examples

[0878] Example 1: Translating dog barks and responding to user emotions

[0879] 1. A user records a dog barking using smart glasses.

[0880] 2. The device sends the data to the server.

[0881] 3. Based on the model, the server analyzes that "the dog is barking because it is hungry."

[0882] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "happy."

[0883] 5. The server generates a response saying "The dog says it's hungry" along with the translation result, "You look happy too."

[0884] 6. The server sends the translation results and response to the device.

[0885] 7. The user checks the translation results and responses on the smart glasses.

[0886] Example 2: Translating cat behavior and responding to user emotions

[0887] 1. A user uses smart glasses to record the cat's behavior.

[0888] 2. The device sends the data to the server.

[0889] 3. Based on the model, the server analyzes that "the cat is jumping because it wants to play."

[0890] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "surprised."

[0891] 5. The server generates a response with the translation result "The cat wants to play" and "You look surprised too."

[0892] 6. The server sends the translation results and response to the device.

[0893] 7. The user checks the translation results and responses on the smart glasses.

[0894] By using this system, users can more easily understand what their pets are thinking and wanting, deepening two-way communication with them. Furthermore, by providing responses based on the user's emotions, it can help to improve the relationship between owners and their pets.

[0895] The processing flow will be explained below.

[0896] Step 1: Data collection

[0897] The server scrapes audio data of dog and cat meows, as well as image and video data of their behavior, from the Internet.

[0898] The server stores the scraped data in a database.

[0899] The server categorizes and organizes the data, preparing it for further processing.

[0900] Step 2: Data Preprocessing

[0901] The server performs noise removal filtering on the collected audio data.

[0902] The server converts the audio data into a spectrogram.

[0903] The image and video data collected by the server is resized and converted into a format suitable for analysis.

[0904] The server stores the preprocessed data in a database.

[0905] Step 3: Data Labeling

[0906] The server applies an automatic labeling algorithm to the preprocessed data.

[0907] The server assigns labels such as "hungry" or "want to play" to the bird's cry data.

[0908] The server assigns labels such as "relaxed" or "excited" to the behavioral data.

[0909] If necessary, the server also performs manual labeling by an expert.

[0910] Step 4: Training the generative AI model

[0911] The server splits the labeled data into a training dataset and a validation dataset.

[0912] The server uses the training dataset with a neural network to train a generative AI model.

[0913] The server evaluates the trained model on the validation dataset to confirm accuracy and performance.

[0914] The server will improve or retrain the model as needed based on the evaluation results.

[0915] Step 5: Train your Emotion Engine

[0916] The server collects the user's voice and facial expression data and uses this as training data.

[0917] The server trains a machine learning model to recognize user emotions.

[0918] The server evaluates the trained emotion recognition model on validation data to confirm accuracy and performance.

[0919] Step 6: Collecting User Input Data

[0920] The user uses smart glasses to record the sounds of a dog or cat meowing.

[0921] The user uses smart glasses to record the behavior of their dog or cat.

[0922] The device sends the recording data to the server.

[0923] Step 7: Data analysis and translation

[0924] The server receives the data sent from the terminal.

[0925] The server uses the generative AI model to analyze the incoming data.

[0926] The server generates a translation result such as "This sound indicates that the bird is hungry" as an analysis result.

[0927] Step 8: Recognizing User Emotions

[0928] The device acquires the user's voice and facial expression data in real time.

[0929] The server analyzes the user's facial expressions and voice data to recognize their emotional state, such as "happy" or "surprised."

[0930] The server generates an appropriate response based on the user's emotional state.

[0931] Step 9: Sending translation results and responses

[0932] The server sends the generated translation result and a response based on the user's emotions to the terminal.

[0933] The terminal receives the translation result and the response.

[0934] Step 10: Viewing Translation Results and Responses

[0935] The terminal outputs the translation results on the display screen.

[0936] The terminal displays an appropriate response to the user in voice or text.

[0937] The user can check the results on the device's display screen and audio output to understand the pet's intentions and emotions.

[0938] Example 2

[0939] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0940] To achieve deeper communication between pets and their owners, interactions that take into account not only the pet's cries and actions but also the owner's emotions are necessary. However, while existing systems can translate pets' actions and cries, they lack the functionality to provide responses that take into account the owner's emotions. This leaves the problem of insufficient two-way communication between pets and their owners.

[0941] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0942] In this invention, the server includes means for collecting data on dog and cat meows, means for collecting data on dog and cat behavior, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording dog and cat meows and behaviors via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to a user's terminal, means for displaying the translation result on the user's terminal, means for recognizing the user's emotions, and means for generating a response based on the user's emotions, thereby enabling a response that takes into account not only the pet's meows and behaviors but also the owner's emotions.

[0943] "Data on dog and cat meows" is information that records the waveforms and temporal characteristics of sounds made by dogs and cats.

[0944] "Dog and cat behavior data" is information that records the movements and behavior of dogs and cats in the form of images or videos.

[0945] "Means of collection" refers to technologies for scraping data from the Internet and devices for acquiring data using cameras and microphones.

[0946] "Preprocessing" is the process of converting collected data into a format that is easier to analyze through processes such as noise removal, data normalization, and resizing.

[0947] "Labeling means" refers to technology that automatically or manually assigns tags that indicate meaning or category to data.

[0948] A "generative AI model" is an artificial intelligence algorithm that uses supervised learning based on large amounts of data to perform specific tasks (such as translation).

[0949] "Input devices used by users" refers to devices such as smart glasses and smartphones that users use to record and film the sounds and behavior of their dogs and cats.

[0950] The "means of analyzing and generating translation results" refers to a technology that uses a generative AI model to analyze the received data, translate the emotions and intentions of the dog or cat, and output the results as text.

[0951] "Means for sending to the user's device" refers to the protocols and technologies used to send the generated translation results to the device used by the user.

[0952] "Means for displaying the translation result on the user's terminal" refers to a function for visually displaying the translation result on a user interface or display.

[0953] "Means for recognizing user emotions" refers to technology that analyzes the user's facial expressions and voice and identifies their emotional state (happiness, surprise, sadness, etc.).

[0954] "Means for generating responses based on user emotions" refers to algorithms or technologies that generate appropriate reactions or messages in response to the recognized user emotions.

[0955] The present invention is a system that translates the sounds and behaviors of dogs and cats and recognizes the user's emotions, enabling two-way communication with pets. Specific methods for implementing this system are described below.

[0956] Data collection

[0957] First, the server collects dog and cat meow and behavior data from the Internet. This is done by scraping data using Python's BeautifulSoup library or Scrapy framework. This collection process is targeted at specific URLs, and the collected data is stored in a database (e.g., MySQL or MongoDB).

[0958] Data Preprocessing

[0959] The server then performs preprocessing on the collected data. This preprocessing includes denoising the audio data, normalizing the data, and resizing the image data. The SciPy and Librosa libraries are used for audio data processing, and the OpenCV library is used for image data processing. Specifically, the server converts the audio data into a spectrogram and resizes the image data to an appropriate size (e.g., 224x224 pixels).

[0960] Labeling

[0961] The server labels the preprocessed data using a pre-trained machine learning model (e.g., ResNet or YOLO). Labeling is automated, but human experts can review and correct the labels if necessary through a web interface.

[0962] Training generative AI models

[0963] The server trains a generative AI model using the labeled data. The dataset is split into training and validation sets (e.g., 80% for training and 20% for validation), and the model is trained using a neural network algorithm (e.g., TensorFlow or PyTorch). Evaluation is performed on the validation dataset to confirm its accuracy.

[0964] Collecting user input data

[0965] Users use smart glasses (e.g., Google Glass) to record and film the sounds and behavior of their dogs and cats. The device then transmits the data from the smart glasses to a server in real time. Protocols such as HTTP and WebSocket are used for transmission.

[0966] Data analysis and translation

[0967] The server analyzes the received data and generates a translation result using a generative AI model. For example, it may generate a result such as "The dog is barking because it is hungry." Based on this analysis result, the dog or cat's emotions and intentions are output as text.

[0968] Additional processing for the emotion engine

[0969] The server uses an emotion analysis engine (e.g., Microsoft Azure's Face API) to recognize the user's emotions. This engine recognizes the user's emotional state (e.g., "happy," "surprised," "sad") from their voice and facial expressions. The server then generates an appropriate response based on this emotional data.

[0970] Sending and displaying translation results and responses

[0971] The server sends the translation results and a response based on the user's emotions to the device. The device then displays these results visually through a user interface (e.g., a smart glasses display). It also outputs audio, if necessary. Based on this information, the user can understand their pet's emotions and intentions and communicate more appropriately.

[0972] Specific examples

[0973] Example 1: Translating dog barks and responding to user emotions

[0974] 1. A user records a dog barking using smart glasses.

[0975] 2. The device sends the data to the server.

[0976] 3. Based on the model, the server analyzes that "the dog is barking because it is hungry."

[0977] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "happy."

[0978] 5. The server generates a response saying "The dog says it's hungry" along with the translation result, "You look happy too."

[0979] 6. The server sends the translation results and response to the device.

[0980] 7. The user checks the translation results and responses on the smart glasses.

[0981] Example 2: Translating cat behavior and responding to user emotions

[0982] 1. A user uses smart glasses to record the cat's behavior.

[0983] 2. The device sends the data to the server.

[0984] 3. Based on the model, the server analyzes that "the cat is jumping because it wants to play."

[0985] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "surprised."

[0986] 5. The server generates a response with the translation "The cat wants to play" and "You look surprised too."

[0987] 6. The server sends the translation results and response to the device.

[0988] 7. The user checks the translation results and responses on the smart glasses.

[0989] This allows users to understand what their pets are thinking and wanting, deepening two-way communication with them. Furthermore, by providing responses based on the user's emotions, the relationship between pet and owner can be further improved.

[0990] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0991] Step 1:

[0992] Data collection

[0993] The server scrapes dog and cat meow data and behavior data (images and videos) from specific URLs on the Internet. In this step, data is collected using libraries such as BeautifulSoup and Scrapy. The input is a specific URL, and the output is the collected audio data, images, and video data. The server stores the collected data in a database (e.g., MySQL or MongoDB).

[0994] Step 2:

[0995] Data Preprocessing

[0996] The server performs preprocessing on the collected data, including noise removal, data normalization, and resizing. In this step, SciPy and Librosa libraries are used for filtering and spectrogram conversion of audio data. OpenCV is used for image and video data resizing. The input is scraped data, and the output is preprocessed data.

[0997] Step 3:

[0998] Labeling

[0999] The server assigns labels to the preprocessed data. In this step, automatic labeling is performed using a machine learning model (e.g., ResNet or YOLO). To improve the accuracy of the labeling, some of the data is checked and corrected by human experts via a web interface. The input is the preprocessed data, and the output is the labeled data.

[1000] Step 4:

[1001] Training generative AI models

[1002] The server trains a generative AI model using the labeled data. The dataset is split into training and validation sets (e.g., 80% for training and 20% for validation). Training is performed using a neural network algorithm (e.g., TensorFlow or PyTorch), with a set learning rate and number of epochs. The input is the labeled data, and the output is a trained AI model.

[1003] Step 5:

[1004] User-generated data collection

[1005] The user uses smart glasses (e.g., Google Glass) to record and record the sounds and behavior of their dog or cat in real time. The device sends the data from the smart glasses to a server. The input is the data obtained from the smart glasses, and the output is the data sent to the server.

[1006] Step 6:

[1007] Data analysis and translation

[1008] The server analyzes the received data and generates translation results using a trained generative AI model. In this step, the data is input into the generative AI model, and the analysis results output translated text that indicates the pet's emotions and intentions. The input is the data sent from the device, and the output is the translated text.

[1009] Step 7:

[1010] Additional processing for the emotion engine

[1011] The server uses an emotion engine (e.g., Microsoft Azure's Face API) to recognize the user's emotions. The device collects the user's voice and facial expressions in real time through a camera and microphone and sends them to the server. The server analyzes this data to recognize the user's emotional state and generate an appropriate response. The input is the user's facial and voice data, and the output is their emotional state and a response based on it.

[1012] Step 8:

[1013] Sending and displaying translation results and responses

[1014] The server sends the generated translation results and responses based on the user's emotions to the device. The device displays these results visually through a user interface (e.g., a smart glasses display) and, if necessary, provides audio output. The input is the translation results and responses sent from the server, and the output is the translation results and responses displayed to the user.

[1015] (Application example 2)

[1016] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1017] Understanding pet sounds and behaviors is challenging for pet owners. Therefore, there is a need for a way to understand what pets are thinking and needing. There is also a need for a system that can not only translate pet intentions and emotions, but also understand the owner's emotions and respond accordingly, thereby further deepening communication between pets and their owners.

[1018] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data on dog and cat meows, means for collecting data on dog and cat behavior, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording dog and cat meows and behaviors via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to the user's terminal, means for displaying the translation result on the user's terminal, means for analyzing input data using an emotion engine that recognizes the user's emotions, means for generating a response corresponding to the user's emotions based on the analysis result, means for transmitting and displaying the response corresponding to the user's emotions on the user's terminal, and means for providing interactive content corresponding to the dog or cat's intentions. This makes it easier to understand the intentions and emotions of pets and provides appropriate responses corresponding to the owner's emotions, thereby deepening communication between pets and their owners.

[1019] "Means for collecting dog and cat meow data" refers to a device or process for recording pet sounds and electronically recording that data.

[1020] "Means for collecting data on dog and cat behavior" refers to devices or processes for monitoring pet movements using cameras, sensors, etc., and collecting behavioral data.

[1021] "Measures for pre-processing collected data" refers to processes for removing noise from audio and video data, normalizing the data, and converting it into a format suitable for analysis.

[1022] "Means for labeling preprocessed data" refers to the process of assigning appropriate tags or labels to data after preprocessing so that it is possible to identify what each piece of data represents.

[1023] "Means for training a generative AI model with labeled data" refers to the process of using a labeled dataset to train a generative AI model to accurately analyze pet sounds and behaviors.

[1024] "Means for recording the sounds and behavior of dogs and cats through input devices used by users" refers to the process of recording the sounds and behavior of pets using input devices such as smartphones, tablets, and smart glasses.

[1025] "Means for analyzing data obtained from a user's input device using a generative AI model and generating a translation result" refers to the process of receiving data sent from a user's input device, analyzing that data using a generative AI model, and translating the pet's intentions and emotions.

[1026] "Means for sending the generated translation results to the user's device" refers to the process of sending the translation results obtained by the generative AI model to the user's smartphone or tablet device.

[1027] "Means for displaying translation results on the user's device" refers to the process of displaying the pet's intentions and emotions analyzed by the generative AI model on the user's smartphone or tablet.

[1028] "Means for analyzing input data using an emotion engine that recognizes the user's emotions" refers to the process of capturing the user's facial expressions and voice using a camera or microphone, and analyzing them to recognize the user's emotional state.

[1029] The "means for generating a response according to the user's emotions based on the analysis results" refers to a process for generating an optimal response according to the recognized emotional state of the user.

[1030] "Means for transmitting and displaying a response according to the user's emotions to the user's device" refers to the process of transmitting the generated response to the user's smartphone or tablet device and displaying it.

[1031] "Means for providing interactive content according to the intentions of dogs and cats" refers to a process for providing users with educational and entertainment content according to the intentions and state of their pets.

[1032] The present invention is a system that translates the sounds and behaviors of dogs and cats and recognizes the user's emotions, thereby deepening communication between the user and the pet. The system of the present invention is implemented through the following steps.

[1033] System Overview

[1034] The system analyzes pet sounds and behaviors and provides multiple ways to recognize the user's emotions. The system then analyzes the collected data using a generative AI model to generate translation results and responses based on the user's emotions.

[1035] Data collection

[1036] The server collects the meows and behavioral data of dogs and cats. Pet sounds are recorded using a microphone, and behavioral data is recorded using a camera. These data are stored in cloud storage and later pre-processed.

[1037] Data Preprocessing

[1038] The collected data undergoes preprocessing such as noise removal and normalization. Specifically, the server uses a noise filter to remove noise from the audio data, converts the audio data into a spectrogram, and resizes the image data to an appropriate size.

[1039] Labeling

[1040] Once preprocessing is complete, the data is labeled with a label that indicates the meaning or intent of each piece of data. The server automatically assigns the labels using a machine learning algorithm, and experts then review and correct the labels as needed.

[1041] Training generative AI models

[1042] The labeled data is used to train a generative AI model. The dataset is split into training and validation datasets, and a neural network algorithm is used. Once trained, the model's accuracy is checked on the validation dataset.

[1043] Collecting user input data

[1044] Users can record and record the sounds and behavior of their dogs and cats in real time using input devices such as smart glasses or smartphones, and the recorded data is sent from the device to a server.

[1045] Data analysis and translation

[1046] The server analyzes the received data and generates translation results using a generative AI model. Based on the analysis results, the emotions and intentions of the dog or cat are translated.

[1047] Additional processing for the emotion engine

[1048] To recognize the user's emotions, the device captures the user's facial expressions and voice in real time. The server analyzes this data and recognizes the user's emotional state, such as "happy," "surprised," or "sad." An appropriate response is generated based on the user's recognized emotional state.

[1049] Sending translation results and responses

[1050] The generated translation results and responses are sent to the user's device in real time, where they can check the results on the device's display screen or as audio output, making it easier to understand their pet's intentions and emotions.

[1051] Providing interactive content

[1052] The system provides users with interactive content tailored to their pet's state. For example, if a pet expresses a desire to play, the system suggests activities for the user to play with the pet.

[1053] Specific examples

[1054] The user uses smart glasses to record the sound of their dog barking and sends it to the server. Using a generative AI model, the server analyzes the sound and concludes that "the dog is barking because it is hungry," and recognizes the user's facial expression and voice as "happy." The server then generates a translation result of "The dog says it is hungry," along with a response of "You look happy, too." The user can then check the translation result and response on the smart glasses, instantly understanding their pet's intentions and emotions.

[1055] Prompt Sentence Examples

[1056] As a concrete example, we can use pet sound data and label pairs in the following format:

[1057] "Dog barking: hungry" "Cat behavior: wants to play"

[1058] This allows users to gain a deeper understanding of their pet's intentions and emotions, enabling richer communication.

[1059] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1060] Step 1: Data collection

[1061] The server scrapes the internet for audio data of cats and dogs' meows, as well as image and video data of their behavior. The data is retrieved from specific URLs and stored in a database. This allows for the collection of a large amount of data that can be used as training data.

[1062] Input: Specific URL

[1063] Data processing: scraping and data storage

[1064] Output: Audio, image, and video data stored in a database

[1065] Step 2: Data Preprocessing

[1066] The server performs preprocessing such as noise removal, normalization, and resizing on the data collected. Noise is removed from the audio data and it is converted into a spectrogram. Image data is resized to an appropriate size to make it easier to process.

[1067] Input: Collected audio, image, and video data

[1068] Data processing: noise removal, normalization, resizing

[1069] Output: Preprocessed data

[1070] Step 3: Labeling

[1071] The server automatically or manually labels the preprocessed data. Automatic labeling is performed using a machine learning model, and experts review and correct the labels as needed.

[1072] Input: Preprocessed data

[1073] Data processing: automatic labeling, expert review

[1074] Output: Labeled data

[1075] Step 4: Training the generative AI model

[1076] The server uses the labeled data to train a generative AI model, splits the dataset into training and validation sets, and trains the model using a neural network algorithm.The accuracy of the model is then verified on the validation dataset.

[1077] Input: Labeled data

[1078] Data processing: data division, model training, model evaluation

[1079] Output: A trained generative AI model

[1080] Step 5: Collecting User Input Data

[1081] Users use smart glasses or a smartphone to record the sounds and behavior of their dogs and cats, and the recorded data is sent from the device to a server.

[1082] Input: Dog and cat meows and behavior (real-time)

[1083] Data processing: audio and video recording, data transmission

[1084] Output: Recorded data sent to the server

[1085] Step 6: Data analysis and translation

[1086] The server analyzes the received data and generates translation results using a generative AI model. Through analysis, it determines the pet's emotions and intentions.

[1087] Input: Audio and video data

[1088] Data processing: Analysis with generative AI models

[1089] Output: Translation result (pet's emotions and intentions)

[1090] Step 7: Further processing of the emotion engine

[1091] To recognize the user's emotions, the device captures the user's facial expressions and voice in real time. The server analyzes this data and recognizes the user's emotional state, such as "happy," "surprised," or "sad."

[1092] Input: User's facial expression and voice data

[1093] Data processing: Analysis using emotion engine

[1094] Output: User's emotional state

[1095] Step 8: Generate a response based on the analysis results

[1096] The server generates an appropriate response based on the user's emotional state. For example, a translation result indicating the pet's intentions is generated, along with a response based on the user's emotions.

[1097] Input: User's emotional state, pet translation results

[1098] Data processing: response generation

[1099] Output: Translation results and responses for the user

[1100] Step 9: Sending the response and translation

[1101] The generated translation results and responses are sent in real time to the user's device, where the user can check the results on the device's display screen or via voice output.

[1102] Input: Translation result and response

[1103] Data processing: Sending results

[1104] Output: Display on user terminal

[1105] Step 10: Serve interactive content

[1106] The system provides users with interactive content that responds to the pet's condition, further enriching the communication between pets and their owners.

[1107] Input: Pet state, user emotional state

[1108] Data processing: Content generation

[1109] Output: Providing interactive content

[1110] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1111] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1112] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1113] [Third embodiment]

[1114] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1115] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1116] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1117] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1118] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1119] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1120] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1121] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1122] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1123] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1124] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1125] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1126] System Overview

[1127] This invention is a system that translates the meows and behaviors of dogs and cats. It uses generative AI to analyze data on dog and cat meows and behaviors, and outputs the translation results as language. This system allows users to deepen their communication with their pets and understand their emotions and intentions.

[1128] Program Details

[1129] Data collection

[1130] The server collects dog and cat meow and behavior data from online audio databases, social media, video sharing sites, etc. This allows for the collection of a sufficient amount of data to be used as training data.

[1131] 1. The server scrapes data from a specific URL.

[1132] 2. The server stores the collected data in a database.

[1133] Data Preprocessing

[1134] The server performs preprocessing on the collected data, such as noise removal, normalization, and resizing.

[1135] 1. The server performs filtering to remove noise from the data.

[1136] 2. The server converts the audio data into a spectrogram and resizes the image data to an appropriate size.

[1137] Labeling

[1138] The server automatically or manually assigns labels to the preprocessed data.

[1139] 1. The server automates labeling using a machine learning model.

[1140] 2. Human experts review and correct the labels, if necessary.

[1141] Training generative AI models

[1142] The server uses the labeled data to train a generative AI model.

[1143] 1. The server splits the dataset into training and validation sets.

[1144] 2. The server trains the model using a neural network algorithm.

[1145] 3. The server evaluates the trained model on the validation dataset to confirm its accuracy.

[1146] Translation system execution

[1147] The user uses an input device (e.g., smart glasses) to record and film the sounds and behavior of their dog or cat.

[1148] 1. The user turns on the smart glasses and records the sounds and behavior of their pet.

[1149] 2. The device sends the recording data to the server.

[1150] The server analyzes the received data and generates a translation result.

[1151] 1. The server analyzes the data using a generative AI model and generates translation results.

[1152] 2. The server sends the translation results to the device.

[1153] The user checks the translation results through the terminal.

[1154] 1. The device displays or speaks the translation results.

[1155] 2. The user checks the results on the device's display screen or audio output.

[1156] Specific examples

[1157] Example 1: Translating a dog's bark

[1158] 1. A user records a dog barking using smart glasses.

[1159] 2. The device sends the data to the server.

[1160] 3. The server analyzes based on the model that "the dog is barking because it is hungry."

[1161] 4. The server sends the translation results to the device.

[1162] 5. The user checks the translation results on the smart glasses.

[1163] Example 2: Translating cat behavior

[1164] 1. A user uses smart glasses to record the cat's behavior.

[1165] 2. The device sends the data to the server.

[1166] 3. Based on the model, the server analyzes that "the cat is jumping because it wants to play."

[1167] 4. The server sends the translation results to the device.

[1168] 5. The user checks the translation results on the smart glasses.

[1169] By using this system, users can more easily understand what their pets are thinking and want, allowing them to build better relationships with their pets.

[1170] The processing flow will be explained below.

[1171] Step 1: Data collection

[1172] The server scrapes audio data of dog and cat meows, as well as image and video data of their behavior, from the Internet.

[1173] The server stores the scraped data in a database.

[1174] The server categorizes and organizes the data, preparing it for further processing.

[1175] Step 2: Data Preprocessing

[1176] The server performs noise removal filtering on the collected audio data.

[1177] The server converts the audio data into a spectrogram.

[1178] The image and video data collected by the server is resized and converted into a format suitable for analysis.

[1179] The server stores the preprocessed data in a database.

[1180] Step 3: Data Labeling

[1181] The server applies an automatic labeling algorithm to the preprocessed data.

[1182] The server assigns labels such as "hungry" or "want to play" to the bird's cry data.

[1183] The server assigns labels such as "relaxed" or "excited" to the behavioral data.

[1184] If necessary, the server also performs manual labeling by an expert.

[1185] Step 4: Training the generative AI model

[1186] The server splits the labeled data into a training dataset and a validation dataset.

[1187] The server uses the training dataset with a neural network to train a generative AI model.

[1188] The server evaluates the trained model on the validation dataset to confirm accuracy and performance.

[1189] The server will improve or retrain the model as needed based on the evaluation results.

[1190] Step 5: Collect input data

[1191] The user uses smart glasses to record the sounds of a dog or cat meowing.

[1192] The user uses smart glasses to record the behavior of their dog or cat.

[1193] The device sends the recording data to the server.

[1194] Step 6: Data analysis and translation

[1195] The server receives the data sent from the terminal.

[1196] The server uses the generative AI model to analyze the incoming data.

[1197] The server generates a translation result based on the analysis, such as "This sound indicates that the bird is hungry."

[1198] Step 7: Send the translation

[1199] The server transmits the generated translation result to the terminal.

[1200] The device receives the translation result.

[1201] Step 8: Displaying the translation results

[1202] The terminal outputs the translation results on the display screen.

[1203] The user checks the displayed translation results and understands the pet's intentions and emotions.

[1204] Example 1

[1205] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1206] In recent years, there has been an increasing demand for understanding the emotions and intentions of pets from their vocalizations and behaviors. However, current technologies lack the means for owners to gain a deeper understanding of their pets' states. Therefore, an effective system for improving communication between pets and their owners is needed.

[1207] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1208] In this invention, the server includes means for collecting audio data, means for collecting video data, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording audio and video via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to the user's terminal, and means for displaying the translation result on the user's terminal, thereby enabling a user to understand the emotions and intentions of their pet and deepen communication with their pet.

[1209] "Audio data" means data recorded in electronic form of animal calls or other sounds.

[1210] "Video data" refers to data that records the behavior and appearance of animals in electronic format.

[1211] "Preprocessing" is the process of performing processes such as noise removal, normalization, and resizing on collected data to prepare it in a format suitable for analysis.

[1212] A "label" is a tag or annotation that is attached to data to identify the type or state of the subject of analysis.

[1213] A "generative AI model" is an artificial intelligence model that is trained based on collected, preprocessed, and labeled data to provide appropriate translation results or predictions for specific input data.

[1214] An "input device" refers to a device that a user uses to record their pet's sounds and behavior, such as smart glasses or a smartphone.

[1215] "Analysis" refers to the act of analyzing input data using a generative AI model and inferring translation results and intent based on that data.

[1216] The "translation result" is information obtained from the analyzed data that describes the animal's emotions and intentions.

[1217] "Terminal" refers to the device used by the user to check the translation results, such as smart glasses or smartphones.

[1218] The present invention relates to a system for analyzing audio data and video data to understand the emotions and intentions of animals. How the present invention can be implemented will be described in detail below.

[1219] System Overview

[1220] The system primarily consists of a server, a terminal, and an input device used by the user. The server collects audio and video data, preprocesses it, and labels it. The labeled data is then used to train a generative AI model and analyze new data obtained from the user's input device. The terminal receives the translation results and displays them to the user.

[1221] Hardware and software used

[1222] Server: Collects data, preprocesses it, labels it, trains generative AI models, and analyzes it. It uses software libraries and tools such as Python, TensorFlow, PyTorch, Librosa, BeautifulSoup, OpenCV, and MySQL.

[1223] Terminal: A device used by a user as an input device, such as smart glasses or a smartphone, that records audio and video and transmits the data to a server.

[1224] Input device: Smart glasses or smartphones are examples of input devices that record pet sounds and behavior in real time.

[1225] Processing flow

[1226] Data collection: The server collects audio and video data from online audio databases, social media sites, and video sharing sites. The data is scraped using Python's BeautifulSoup library and stored in a MySQL database.

[1227] Data preprocessing: The server performs preprocessing on the collected data, such as noise removal, normalization, and resizing. It uses the Python Librosa library to remove noise from the audio data, Matplotlib to generate spectrograms, and OpenCV to resize the images.

[1228] Labeling: The server automatically assigns labels using machine learning models, and experts review and correct the labels as needed. A classifier is trained from existing labeled data using frameworks such as TensorFlow, and new data is labeled.

[1229] Training the generative AI model: The server uses the labeled data to train the generative AI model. It uses Python's TensorFlow or PyTorch to build a neural network algorithm and utilizes a GPU for fast training.

[1230] Translation system execution: The user uses an input device such as smart glasses to record and record the sounds and behavior of their pet. The device compresses this data in real time and uploads it to the server via HTTPS. The server then uses a generative AI model to analyze the received data, generate translation results, and send them to the device. Finally, the device displays or outputs the translation results as audio, allowing the user to confirm the results.

[1231] Specific examples

[1232] Example 1: Translating a dog's bark

[1233] 1. A user uses the smart glasses to record the sound of a dog barking, for example, by pressing a button to record the sound of a dog barking.

[1234] 2. The device sends the data to the server.

[1235] 3. Based on the generated AI model, the server analyzes that "the dog is barking because it is hungry."

[1236] 4. The server sends the translation results to the device.

[1237] 5. The user checks the translation results on the smart glasses. The message "I'm hungry" appears on the display and the user reads it.

[1238] Example 2: Translating cat behavior

[1239] 1. A user uses smart glasses to record the cat's behavior, for example, pressing a button to record the cat jumping.

[1240] 2. The device sends the data to the server. The recorded data is compressed in real time and sent to the server.

[1241] 3. Based on the generated AI model, the server analyzes that "the cat is jumping because it wants to play."

[1242] 4. The server sends the translation results to the device.

[1243] 5. The user checks the translation results on the smart glasses. The message "I want to play" appears on the display and the user reads it.

[1244] In this way, by using the system of the present invention, the user can understand the emotions and intentions of their pet and deepen communication with their pet.

[1245] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1246] Step 1:

[1247] Data collection

[1248] The server scrapes data from a specific URL, for example, using Python's BeautifulSoup library to retrieve audio and video files from social media posts.

[1249] Input: A specific URL list.

[1250] Output: Collected audio and video data.

[1251] What it does: The server analyzes the web page, extracts links to audio and video files, and downloads those files.

[1252] Step 2:

[1253] Database storage

[1254] The server stores the collected data in a database. The MySQL database stores the paths and metadata of audio and video files.

[1255] Input: Collected audio and video data.

[1256] Output: Path and metadata of the data stored in the database.

[1257] What happens: The server inserts each file's path and associated metadata (date and time of acquisition, original URL, etc.) into a MySQL database.

[1258] Step 3:

[1259] Data Preprocessing

[1260] The server performs filtering to remove noise. The Librosa library in Python is used to remove noise from the audio data.

[1261] Input: Collected audio data.

[1262] Output: Clear audio data with noise removed.

[1263] What it does: The server uses Librosa to load the audio data and applies a specific noise filter to remove unwanted noise.

[1264] The server converts the audio data into a spectrogram and resizes the image data to an appropriate size. The spectrogram is generated using Python's Matplotlib and the image data is resized using OpenCV.

[1265] Input: Denoised audio data and collected video data.

[1266] Output: Spectrogram image and resized video data.

[1267] What it does: Audio data is converted to a spectrogram using Matplotlib, and video data is resized to a consistent size using OpenCV.

[1268] Step 4:

[1269] Labeling

[1270] The server automates labeling using machine learning models, training classifiers using TensorFlow and other tools to label new data.

[1271] Input: Preprocessed audio and video data (spectrogram images and resized video data).

[1272] Output: Labeled data.

[1273] What it does: The server runs a labeling algorithm to automatically assign an appropriate label (e.g., dog barking, cat jumping, etc.) to each piece of data.

[1274] The server then has human experts review and correct the labels as needed.

[1275] Input: Auto-labeled data.

[1276] Output: Verified and corrected labeled data.

[1277] Specific operation: Experts review the labels of each data through a web interface and correct them if necessary.

[1278] Step 5:

[1279] Training generative AI models

[1280] The server splits the dataset into training and validation sets using the train_test_split function from scikit-learn.

[1281] Input: A labeled dataset.

[1282] Output: Split training and validation data.

[1283] Specific operation: The server splits the labeled dataset into training and validation sets in an 80:20 ratio.

[1284] The server trains the model using a neural network algorithm. The model is built using TensorFlow or PyTorch, and trained quickly using a GPU.

[1285] Input: Training data.

[1286] Output: A trained generative AI model.

[1287] What it does: The server configures the model architecture and iteratively inputs data to tune the model parameters.

[1288] The server evaluates the trained model on a validation dataset to confirm its accuracy.

[1289] Input: A trained generative AI model, and validation data.

[1290] Output: Model performance metrics (accuracy, sensitivity, specificity, etc.).

[1291] Specific operation: The server evaluates the model's performance using the validation dataset and logs the results.

[1292] Step 6:

[1293] Translation system execution

[1294] A user uses an input device such as smart glasses to record and film the sounds and behavior of a dog or cat.

[1295] Input: Pet sounds and behavior.

[1296] Output: Recorded data.

[1297] Specific operation: The user activates the audio and video recording function of the smart glasses and records the sounds and behavior of their pet.

[1298] The device sends the audio and video data to the server, where it is compressed in real time and uploaded to the server via HTTPS.

[1299] Input: Recorded data.

[1300] Output: Data uploaded to the server.

[1301] Specific operation: The device compresses the recorded data and sends it to the server via the Internet.

[1302] The server uses the generative AI model to analyze the incoming data and generate translation results.

[1303] Input: Data uploaded to the server.

[1304] Output: The translation result.

[1305] Specific operation: The server inputs the received data into the generated AI model, analyzes the pet's emotions and intentions, and generates translation results.

[1306] The server generates the translation results and sends them to the device, which returns them in JSON format and reformats them for display.

[1307] Input: Translation result.

[1308] Output: The translation result sent to your device.

[1309] Specific operation: The server packages the translation results in JSON format and sends them to the device.

[1310] The device displays or speaks the translation results. The results are displayed as text on the smart glasses display and spoken through the speakers.

[1311] Input: The translation result sent to the device.

[1312] Output: The displayed or spoken translation result.

[1313] Specific operations: The device displays the translation results to the user by displaying the text on a display or by using a speech synthesis system to output the results aloud from a speaker.

[1314] The user checks the results on the device's display screen or audio output.

[1315] Input: The displayed or spoken translation result.

[1316] Output: The translation result as recognized by the user.

[1317] Specific operation: The user checks the translation results by looking at the display screen of the smart glasses and listening to the voice output.

[1318] (Application example 1)

[1319] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1320] Conventional systems that translate the meows and behavior of dogs and cats collect data on dog and cat meows and behavior from online audio databases, social media, video sharing sites, etc., and analyze it using a generative AI model. However, current systems cannot display the pet's emotions and needs on a display device in real time, making it difficult for users to immediately check the pet's condition. This makes it difficult for, for example, pet shop employees and customers to quickly understand the pet's condition, making it difficult to provide appropriate responses and services.

[1321] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1322] In this invention, the server includes means for collecting data on dog and cat meows, means for collecting data on dog and cat behavior, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording dog and cat meows and behaviors via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to the user's terminal, means for displaying the translation result on the user's terminal, and means for displaying the pet's meows and behaviors on a display device in real time, thereby enabling the user to visually check the status of their pet in real time and provide appropriate responses and services immediately.

[1323] "Means for collecting data on dog and cat meows" refers to devices or functions that record sounds made by dogs and cats and store them for analysis.

[1324] "Means for collecting data on the behavior of dogs and cats" refers to devices or functions that record the behavior of dogs and cats and collect data to analyze their behavior.

[1325] The "means for preprocessing the collected data" refers to a device or function that performs preprocessing such as noise removal and data normalization on the collected audio and video data.

[1326] The "means for assigning labels to preprocessed data" refers to a device or function that automatically or manually assigns labels to preprocessed data according to their contents.

[1327] A "means for training a generative AI model using labeled data" is a device or function that uses labeled data to train an AI model and improve its accuracy.

[1328] "Means for recording the meows and behavior of dogs and cats through an input device used by a user" refers to a device or function that records the meows and behavior of dogs and cats through an input device used by a user, such as smart glasses.

[1329] "Means for analyzing data obtained from a user's input device using a generative AI model and generating a translation result" refers to a device or function that analyzes data sent from an input device using an AI model and translates the results as text.

[1330] The "means for transmitting the generated translation result to the user's terminal" refers to a device or function that transmits the translation result generated by the server to the user's smart glasses, smartphone, etc.

[1331] The "means for displaying the translation result on the user's terminal" refers to a device or function that visually displays the translation result on the user's terminal.

[1332] The "means for displaying the sounds and behavior of a pet on a display device in real time" refers to a device or function that captures the sounds and behavior of a pet in real time and immediately displays the results on a display device.

[1333] System Overview

[1334] This invention is a system that translates the sounds and behaviors of dogs and cats. Using a generative AI model, this data is analyzed and the pet's emotions and intentions are conveyed to the user. In particular, this invention has the function of displaying the pet's sounds and behaviors in real time on a display device, allowing pet shop employees and customers to instantly check the pet's status.

[1335] Program Details

[1336] Data collection methods

[1337] The server collects dog and cat meow and behavior data from online audio databases, social media, video sharing sites, etc. This allows for the collection of a sufficient amount of data to be used as training data.

[1338] Data preprocessing measures

[1339] The server performs preprocessing on the collected data, such as noise removal and normalization. For example, it filters the audio data to remove noise and converts it into a spectrogram. It also resizes the image data to an appropriate size.

[1340] Labeling method

[1341] The server automatically labels the preprocessed data using a machine learning model, and experts review and correct the labels if necessary.

[1342] A means of training generative AI models

[1343] The server trains a generative AI model using the labeled data, splits the dataset into training and validation sets, and trains the model using a neural network algorithm.The model is then evaluated on the validation dataset to confirm its accuracy.

[1344] Translation system implementation

[1345] Users use input devices such as smart glasses to record and record the sounds and behavior of their dogs and cats. The device then sends the recorded data to a server, which then analyzes the data using a generative AI model to generate a translation. The resulting translation is then sent back to the device, where the user can view it on their smart glasses or smartphone.

[1346] Real-time display means

[1347] The server analyzes the pet's sounds and behavior in real time and immediately displays the results on a display device, allowing pet shop staff and customers to understand the pet's condition on the spot.

[1348] Specific examples

[1349] Example 1: Translating a dog's bark

[1350] The user records the sound of a dog barking using smart glasses. The device sends the data to the server, which then uses the model to analyze the sound and concludes, "The dog is barking because it is hungry." The server then sends the translation results to the device, where the user can check them on the smart glasses.

[1351] Example 2: Translating cat behavior

[1352] The user records the cat's behavior using smart glasses. The device sends the data to the server, which then analyzes it based on the model and concludes that "the cat is jumping because it wants to play." The server then sends the translation results to the device, where the user can check them on the smart glasses.

[1353] Prompt Sentence Examples

[1354] "Send the data to the Pet Communication Guide app and output the translation result as 'This dog is hungry.'"

[1355] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1356] Step 1:

[1357] The user wears smart glasses and records the sounds and behavior of their dog or cat. As input, the user operates the audio / video recording function of the smart glasses to capture the voice and video of their pet. As output, the audio / video data is stored in the smart glasses' internal memory.

[1358] Step 2:

[1359] The device sends audio and video data from the user's smart glasses to the server. The audio and video data stored on the smart glasses is used as input. The data is sent to the server via the Internet. The server receives the data as output.

[1360] Step 3:

[1361] The server performs preprocessing on the received data. It uses the transmitted audio and video data as input. The server applies noise reduction filters and data normalization algorithms, converts audio data into spectrograms, and resizes video data. As output, it produces preprocessed audio and video data.

[1362] Step 4:

[1363] The server assigns labels to the preprocessed data. As input, it uses preprocessed audio and video data and automatically labels them using a generative AI model. As output, a labeled dataset is generated.

[1364] Step 5:

[1365] The server trains a generative AI model using the labeled data. It uses the labeled dataset as input and applies a neural network algorithm. It splits the dataset into training and validation parts, and trains and evaluates the model. The output is a trained generative AI model.

[1366] Step 6:

[1367] The server uses a generative AI model to analyze data obtained from the user's input device and generate translation results. Raw audio and video data sent by the user is used as input. Analysis is performed using the generative AI model, and translation results of the pet's cries and behavior are output. The output is a translated text.

[1368] Step 7:

[1369] The server sends the translation results to the user's device. The translation result text generated by the server is used as input. The data is sent via the Internet to the user's smart glasses or smartphone. The translation results arrive at the device as output.

[1370] Step 8:

[1371] The user's device displays the translation result. It uses the translation result text received from the server as input. It shows the translation result to the user through the device's display or audio output. As an output, it generates a state that allows the user to confirm the translation result visually or audibly.

[1372] Step 9:

[1373] The server displays the pet's cries and behavior in real time on a display device. Recorded and video data acquired in real time is used as input. A generative AI model instantly analyzes the data and generates translation results. The real-time translation results are displayed on a display device as output.

[1374] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1375] System Overview

[1376] This invention combines a system that translates the meows and behaviors of dogs and cats with an emotion engine that recognizes the user's emotions, further deepening two-way communication between pets and users. It uses generative AI to analyze dog and cat meows and behavior data and output the translation results as language, and also provides a function to recognize the user's emotions in real time and generate appropriate responses.

[1377] Program Details

[1378] Data collection

[1379] The server scrapes the internet for audio data of cats and dogs' meows, as well as image and video data of their behavior, collecting a sufficient amount of data to be used as training data.

[1380] 1. The server scrapes data from a specific URL.

[1381] 2. The server stores the collected data in a database.

[1382] Data Preprocessing

[1383] The server performs preprocessing such as noise removal, normalization, and resizing on the collected data.

[1384] 1. The server performs filtering to remove noise from the data.

[1385] 2. The server converts the audio data into a spectrogram and resizes the image data to an appropriate size.

[1386] Labeling

[1387] The server automatically or manually assigns labels to the preprocessed data.

[1388] 1. The server automates labeling using a machine learning model.

[1389] 2. Human experts review and correct the labels, if necessary.

[1390] Training generative AI models

[1391] The server uses the labeled data to train a generative AI model.

[1392] 1. The server splits the dataset into training and validation sets.

[1393] 2. The server trains the model using a neural network algorithm.

[1394] 3. The server evaluates the trained model on the validation dataset to confirm its accuracy.

[1395] Collecting user input data

[1396] The user uses an input device (e.g., smart glasses) to record and film the sounds of a dog or cat meowing.

[1397] 1. The user turns on the smart glasses and records the sounds and behavior of their pet.

[1398] 2. The device sends the recording data to the server.

[1399] Data analysis and translation

[1400] The server analyzes the received data and generates translation results using a generative AI model.

[1401] 1. The server analyzes the data using a generative AI model and generates translation results.

[1402] 2. The server uses the analysis results to translate the dog or cat's emotions and intentions.

[1403] Additional processing for the emotion engine

[1404] The server analyzes the user's input data using an emotion engine that recognizes the user's emotions.

[1405] 1. The device captures the user's voice and facial expressions in real time using a camera and microphone.

[1406] 2. The server analyzes the user's facial expressions and voice data to recognize their emotional state, such as "happy," "surprised," or "sad."

[1407] 3. The server generates an appropriate response based on the user's emotional state.

[1408] Sending translation results and responses

[1409] The server sends the generated translation result and a response based on the user's emotions to the terminal.

[1410] 1. The server sends the translation results and emotional responses to the device in real time.

[1411] 2. The device displays the translation results and response.

[1412] Displaying translation results and responses

[1413] The user can check the translation results and emotion-based responses through the device.

[1414] 1. The device outputs the translation results on the display screen.

[1415] 2. The device displays an appropriate response to the user in voice or text.

[1416] 3. The user can check the results on the device's display screen or audio output, making it easier to understand the pet's intentions and emotions.

[1417] Specific examples

[1418] Example 1: Translating dog barks and responding to user emotions

[1419] 1. A user records a dog barking using smart glasses.

[1420] 2. The device sends the data to the server.

[1421] 3. Based on the model, the server analyzes that "the dog is barking because it is hungry."

[1422] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "happy."

[1423] 5. The server generates a response saying "The dog says it's hungry" along with the translation result, "You look happy too."

[1424] 6. The server sends the translation results and response to the device.

[1425] 7. The user checks the translation results and responses on the smart glasses.

[1426] Example 2: Translating cat behavior and responding to user emotions

[1427] 1. A user uses smart glasses to record the cat's behavior.

[1428] 2. The device sends the data to the server.

[1429] 3. Based on the model, the server analyzes that "the cat is jumping because it wants to play."

[1430] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "surprised."

[1431] 5. The server generates a response with the translation result "The cat wants to play" and "You look surprised too."

[1432] 6. The server sends the translation results and response to the device.

[1433] 7. The user checks the translation results and responses on the smart glasses.

[1434] By using this system, users can more easily understand what their pets are thinking and wanting, deepening two-way communication with them. Furthermore, by providing responses based on the user's emotions, it can help to improve the relationship between owners and their pets.

[1435] The processing flow will be explained below.

[1436] Step 1: Data collection

[1437] The server scrapes audio data of dog and cat meows, as well as image and video data of their behavior, from the Internet.

[1438] The server stores the scraped data in a database.

[1439] The server categorizes and organizes the data, preparing it for further processing.

[1440] Step 2: Data Preprocessing

[1441] The server performs noise removal filtering on the collected audio data.

[1442] The server converts the audio data into a spectrogram.

[1443] The image and video data collected by the server is resized and converted into a format suitable for analysis.

[1444] The server stores the preprocessed data in a database.

[1445] Step 3: Data Labeling

[1446] The server applies an automatic labeling algorithm to the preprocessed data.

[1447] The server assigns labels such as "hungry" or "want to play" to the bird's cry data.

[1448] The server assigns labels such as "relaxed" or "excited" to the behavioral data.

[1449] If necessary, the server also performs manual labeling by an expert.

[1450] Step 4: Training the generative AI model

[1451] The server splits the labeled data into a training dataset and a validation dataset.

[1452] The server uses the training dataset with a neural network to train a generative AI model.

[1453] The server evaluates the trained model on the validation dataset to confirm accuracy and performance.

[1454] The server will improve or retrain the model as needed based on the evaluation results.

[1455] Step 5: Train your Emotion Engine

[1456] The server collects the user's voice and facial expression data and uses this as training data.

[1457] The server trains a machine learning model to recognize user emotions.

[1458] The server evaluates the trained emotion recognition model on validation data to confirm accuracy and performance.

[1459] Step 6: Collecting User Input Data

[1460] The user uses smart glasses to record the sounds of a dog or cat meowing.

[1461] The user uses smart glasses to record the behavior of their dog or cat.

[1462] The device sends the recording data to the server.

[1463] Step 7: Data analysis and translation

[1464] The server receives the data sent from the terminal.

[1465] The server uses the generative AI model to analyze the incoming data.

[1466] The server generates a translation result such as "This sound indicates that the bird is hungry" as an analysis result.

[1467] Step 8: Recognizing User Emotions

[1468] The device acquires the user's voice and facial expression data in real time.

[1469] The server analyzes the user's facial expressions and voice data to recognize their emotional state, such as "happy" or "surprised."

[1470] The server generates an appropriate response based on the user's emotional state.

[1471] Step 9: Sending translation results and responses

[1472] The server sends the generated translation result and a response based on the user's emotions to the terminal.

[1473] The terminal receives the translation result and the response.

[1474] Step 10: Viewing Translation Results and Responses

[1475] The terminal outputs the translation results on the display screen.

[1476] The terminal displays an appropriate response to the user in voice or text.

[1477] The user can check the results on the device's display screen and audio output to understand the pet's intentions and emotions.

[1478] Example 2

[1479] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1480] To achieve deeper communication between pets and their owners, interactions that take into account not only the pet's cries and actions but also the owner's emotions are necessary. However, while existing systems can translate pets' actions and cries, they lack the functionality to provide responses that take into account the owner's emotions. This leaves the problem of insufficient two-way communication between pets and their owners.

[1481] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1482] In this invention, the server includes means for collecting data on dog and cat meows, means for collecting data on dog and cat behavior, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording dog and cat meows and behaviors via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to a user's terminal, means for displaying the translation result on the user's terminal, means for recognizing the user's emotions, and means for generating a response based on the user's emotions, thereby enabling a response that takes into account not only the pet's meows and behaviors but also the owner's emotions.

[1483] "Data on dog and cat meows" is information that records the waveforms and temporal characteristics of sounds made by dogs and cats.

[1484] "Dog and cat behavior data" is information that records the movements and behavior of dogs and cats in the form of images or videos.

[1485] "Means of collection" refers to technologies for scraping data from the Internet and devices for acquiring data using cameras and microphones.

[1486] "Preprocessing" is the process of converting collected data into a format that is easier to analyze through processes such as noise removal, data normalization, and resizing.

[1487] "Labeling means" refers to technology that automatically or manually assigns tags that indicate meaning or category to data.

[1488] A "generative AI model" is an artificial intelligence algorithm that uses supervised learning based on large amounts of data to perform specific tasks (such as translation).

[1489] "User-used input devices" refer to devices such as smart glasses and smartphones that users use to record and record the sounds and behavior of their dogs and cats.

[1490] The "means of analyzing and generating translation results" refers to a technology that uses a generative AI model to analyze the received data, translate the emotions and intentions of the dog or cat, and output the results as text.

[1491] "Means for sending to the user's device" refers to the protocols and technologies used to send the generated translation results to the device used by the user.

[1492] "Means for displaying the translation result on the user's terminal" refers to a function for visually displaying the translation result on a user interface or display.

[1493] "Means for recognizing user emotions" refers to technology that analyzes the user's facial expressions and voice and identifies their emotional state (happiness, surprise, sadness, etc.).

[1494] "Means for generating responses based on user emotions" refers to algorithms or technologies that generate appropriate reactions or messages in response to the recognized user emotions.

[1495] The present invention is a system that translates the sounds and behaviors of dogs and cats and recognizes the user's emotions, enabling two-way communication with pets. Specific methods for implementing this system are described below.

[1496] Data collection

[1497] First, the server collects dog and cat meow and behavior data from the Internet. This is done by scraping data using Python's BeautifulSoup library or Scrapy framework. This collection process is targeted at specific URLs, and the collected data is stored in a database (e.g., MySQL or MongoDB).

[1498] Data Preprocessing

[1499] The server then performs preprocessing on the collected data. This preprocessing includes denoising the audio data, normalizing the data, and resizing the image data. The SciPy and Librosa libraries are used for audio data processing, and the OpenCV library is used for image data processing. Specifically, the server converts the audio data into a spectrogram and resizes the image data to an appropriate size (e.g., 224x224 pixels).

[1500] Labeling

[1501] The server labels the preprocessed data using a pre-trained machine learning model (e.g., ResNet or YOLO). Labeling is automated, but human experts can review and correct the labels if necessary through a web interface.

[1502] Training generative AI models

[1503] The server trains a generative AI model using the labeled data. The dataset is split into training and validation sets (e.g., 80% for training and 20% for validation), and the model is trained using a neural network algorithm (e.g., TensorFlow or PyTorch). Evaluation is performed on the validation dataset to confirm its accuracy.

[1504] Collecting user input data

[1505] Users use smart glasses (e.g., Google Glass) to record and film the sounds and behavior of their dogs and cats. The device then transmits the data from the smart glasses to a server in real time. Protocols such as HTTP and WebSocket are used for transmission.

[1506] Data analysis and translation

[1507] The server analyzes the received data and generates a translation result using a generative AI model. For example, it may generate a result such as "The dog is barking because it is hungry." Based on this analysis result, the dog or cat's emotions and intentions are output as text.

[1508] Additional processing for the emotion engine

[1509] The server uses an emotion analysis engine (e.g., Microsoft Azure's Face API) to recognize the user's emotions. This engine recognizes the user's emotional state (e.g., "happy," "surprised," "sad") from their voice and facial expressions. The server then generates an appropriate response based on this emotional data.

[1510] Sending and displaying translation results and responses

[1511] The server sends the translation results and a response based on the user's emotions to the device. The device then displays these results visually through a user interface (e.g., a smart glasses display). It also outputs audio, if necessary. Based on this information, the user can understand their pet's emotions and intentions and communicate more appropriately.

[1512] Specific examples

[1513] Example 1: Translating dog barks and responding to user emotions

[1514] 1. A user records a dog barking using smart glasses.

[1515] 2. The device sends the data to the server.

[1516] 3. Based on the model, the server analyzes that "the dog is barking because it is hungry."

[1517] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "happy."

[1518] 5. The server generates a response saying "The dog says it's hungry" along with the translation result, "You look happy too."

[1519] 6. The server sends the translation results and response to the device.

[1520] 7. The user checks the translation results and responses on the smart glasses.

[1521] Example 2: Translating cat behavior and responding to user emotions

[1522] 1. A user uses smart glasses to record the cat's behavior.

[1523] 2. The device sends the data to the server.

[1524] 3. Based on the model, the server analyzes that "the cat is jumping because it wants to play."

[1525] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "surprised."

[1526] 5. The server generates a response with the translation "The cat wants to play" and "You look surprised too."

[1527] 6. The server sends the translation results and response to the device.

[1528] 7. The user checks the translation results and responses on the smart glasses.

[1529] This allows users to understand what their pets are thinking and wanting, deepening two-way communication with them. Furthermore, by providing responses based on the user's emotions, the relationship between pet and owner can be further improved.

[1530] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1531] Step 1:

[1532] Data collection

[1533] The server scrapes dog and cat meow data and behavior data (images and videos) from specific URLs on the Internet. In this step, data is collected using libraries such as BeautifulSoup and Scrapy. The input is a specific URL, and the output is the collected audio data, images, and video data. The server stores the collected data in a database (e.g., MySQL or MongoDB).

[1534] Step 2:

[1535] Data Preprocessing

[1536] The server performs preprocessing on the collected data, including noise removal, data normalization, and resizing. In this step, SciPy and Librosa libraries are used for filtering and spectrogram conversion of audio data. OpenCV is used for image and video data resizing. The input is scraped data, and the output is preprocessed data.

[1537] Step 3:

[1538] Labeling

[1539] The server assigns labels to the preprocessed data. In this step, automatic labeling is performed using a machine learning model (e.g., ResNet or YOLO). To improve the accuracy of the labeling, some of the data is checked and corrected by human experts via a web interface. The input is the preprocessed data, and the output is the labeled data.

[1540] Step 4:

[1541] Training generative AI models

[1542] The server trains a generative AI model using the labeled data. The dataset is split into training and validation sets (e.g., 80% for training and 20% for validation). Training is performed using a neural network algorithm (e.g., TensorFlow or PyTorch), with a set learning rate and number of epochs. The input is the labeled data, and the output is a trained AI model.

[1543] Step 5:

[1544] User data collection

[1545] The user uses smart glasses (e.g., Google Glass) to record and record the sounds and behavior of their dog or cat in real time. The device sends the data from the smart glasses to a server. The input is the data obtained from the smart glasses, and the output is the data sent to the server.

[1546] Step 6:

[1547] Data analysis and translation

[1548] The server analyzes the received data and generates translation results using a trained generative AI model. In this step, the data is input into the generative AI model, and the analysis results output translated text that indicates the pet's emotions and intentions. The input is the data sent from the device, and the output is the translated text.

[1549] Step 7:

[1550] Additional processing for the emotion engine

[1551] The server uses an emotion engine (e.g., Microsoft Azure's Face API) to recognize the user's emotions. The device collects the user's voice and facial expressions in real time through a camera and microphone and sends them to the server. The server analyzes this data to recognize the user's emotional state and generate an appropriate response. The input is the user's facial and voice data, and the output is their emotional state and a response based on it.

[1552] Step 8:

[1553] Sending and displaying translation results and responses

[1554] The server sends the generated translation results and responses based on the user's emotions to the device. The device displays these results visually through a user interface (e.g., a smart glasses display) and, if necessary, provides audio output. The input is the translation results and responses sent from the server, and the output is the translation results and responses displayed to the user.

[1555] (Application example 2)

[1556] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1557] Understanding pet sounds and behaviors is challenging for pet owners. Therefore, there is a need for a way to understand what pets are thinking and needing. There is also a need for a system that can not only translate pet intentions and emotions, but also understand the owner's emotions and respond accordingly, thereby further deepening communication between pets and their owners.

[1558] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data on dog and cat meows, means for collecting data on dog and cat behavior, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording dog and cat meows and behaviors via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to the user's terminal, means for displaying the translation result on the user's terminal, means for analyzing input data using an emotion engine that recognizes the user's emotions, means for generating a response corresponding to the user's emotions based on the analysis result, means for transmitting and displaying the response corresponding to the user's emotions on the user's terminal, and means for providing interactive content corresponding to the dog or cat's intentions. This makes it easier to understand the intentions and emotions of pets and provides appropriate responses corresponding to the owner's emotions, thereby deepening communication between pets and their owners.

[1559] "Means for collecting dog and cat meow data" refers to a device or process for recording pet sounds and electronically recording that data.

[1560] "Means for collecting data on dog and cat behavior" refers to devices or processes for monitoring pet movements using cameras, sensors, etc., and collecting behavioral data.

[1561] "Measures for pre-processing collected data" refers to processes for removing noise from audio and video data, normalizing the data, and converting it into a format suitable for analysis.

[1562] "Means for labeling preprocessed data" refers to the process of assigning appropriate tags or labels to data after preprocessing so that it is possible to identify what each piece of data represents.

[1563] "Means for training a generative AI model with labeled data" refers to the process of using a labeled dataset to train a generative AI model to accurately analyze pet sounds and behaviors.

[1564] "Means for recording the sounds and behavior of dogs and cats through input devices used by users" refers to the process of recording the sounds and behavior of pets using input devices such as smartphones, tablets, and smart glasses.

[1565] "Means for analyzing data obtained from a user's input device using a generative AI model and generating a translation result" refers to the process of receiving data sent from a user's input device, analyzing that data using a generative AI model, and translating the pet's intentions and emotions.

[1566] "Means for sending the generated translation results to the user's device" refers to the process of sending the translation results obtained by the generative AI model to the user's smartphone or tablet device.

[1567] "Means for displaying translation results on the user's device" refers to the process of displaying the pet's intentions and emotions analyzed by the generative AI model on the user's smartphone or tablet.

[1568] "Means for analyzing input data using an emotion engine that recognizes the user's emotions" refers to the process of capturing the user's facial expressions and voice using a camera or microphone, and analyzing them to recognize the user's emotional state.

[1569] The "means for generating a response according to the user's emotions based on the analysis results" refers to a process for generating an optimal response according to the recognized emotional state of the user.

[1570] "Means for transmitting and displaying a response according to the user's emotions to the user's device" refers to the process of transmitting the generated response to the user's smartphone or tablet device and displaying it.

[1571] "Means for providing interactive content according to the intentions of dogs and cats" refers to a process for providing users with educational and entertainment content according to the intentions and state of their pets.

[1572] The present invention is a system that translates the sounds and behaviors of dogs and cats and recognizes the user's emotions, thereby deepening communication between the user and the pet. The system of the present invention is implemented through the following steps.

[1573] System Overview

[1574] The system analyzes pet sounds and behaviors and provides multiple ways to recognize the user's emotions. The system then analyzes the collected data using a generative AI model to generate translation results and responses based on the user's emotions.

[1575] Data collection

[1576] The server collects the meows and behavioral data of dogs and cats. Pet sounds are recorded using a microphone, and behavioral data is recorded using a camera. These data are stored in cloud storage and later pre-processed.

[1577] Data Preprocessing

[1578] The collected data undergoes preprocessing such as noise removal and normalization. Specifically, the server uses a noise filter to remove noise from the audio data, converts the audio data into a spectrogram, and resizes the image data to an appropriate size.

[1579] Labeling

[1580] Once preprocessing is complete, the data is labeled with the meaning and intent of each piece of data. The server automatically assigns the labels using machine learning algorithms, and experts review and correct the labels as needed.

[1581] Training generative AI models

[1582] A generative AI model is trained using the labeled data. The dataset is split into training and validation datasets, and a neural network algorithm is used. Once trained, the model's accuracy is verified on the validation dataset.

[1583] Collecting user input data

[1584] Users can record and record the sounds and behavior of their dogs and cats in real time using input devices such as smart glasses or smartphones, and the recorded data is sent from the device to a server.

[1585] Data analysis and translation

[1586] The server analyzes the received data and generates translation results using a generative AI model. Based on the analysis results, the emotions and intentions of the dog or cat are translated.

[1587] Additional processing for the emotion engine

[1588] To recognize the user's emotions, the device captures the user's facial expressions and voice in real time. The server analyzes this data and recognizes the user's emotional state, such as "happy," "surprised," or "sad." An appropriate response is generated based on the user's recognized emotional state.

[1589] Sending translation results and responses

[1590] The generated translation results and responses are sent to the user's device in real time, where they can check the results on the device's display screen or as audio output, making it easier to understand their pet's intentions and emotions.

[1591] Providing interactive content

[1592] The system provides users with interactive content tailored to their pet's state. For example, if a pet expresses a desire to play, the system suggests activities for the user to play with the pet.

[1593] Specific examples

[1594] The user uses smart glasses to record the sound of their dog barking and sends it to the server. Using a generative AI model, the server analyzes the sound and concludes that "the dog is barking because it is hungry," and recognizes the user's facial expression and voice as "happy." The server then generates a translation result of "The dog says it is hungry," along with a response of "You look happy, too." The user can then check the translation result and response on the smart glasses, instantly understanding their pet's intentions and emotions.

[1595] Prompt Sentence Examples

[1596] As a concrete example, we can use pet sound data and label pairs in the following format:

[1597] "Dog barking: hungry" "Cat behavior: wants to play"

[1598] This allows users to gain a deeper understanding of their pet's intentions and emotions, enabling richer communication.

[1599] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1600] Step 1: Data collection

[1601] The server scrapes the internet for audio data of cats and dogs' meows, as well as image and video data of their behavior. The data is retrieved from specific URLs and stored in a database. This allows for the collection of a large amount of data that can be used as training data.

[1602] Input: Specific URL

[1603] Data processing: scraping and data storage

[1604] Output: Audio, image, and video data stored in a database

[1605] Step 2: Data Preprocessing

[1606] The server performs preprocessing such as noise removal, normalization, and resizing on the data collected. Noise is removed from the audio data and it is converted into a spectrogram. Image data is resized to an appropriate size to make it easier to process.

[1607] Input: Collected audio, image, and video data

[1608] Data processing: noise removal, normalization, resizing

[1609] Output: Preprocessed data

[1610] Step 3: Labeling

[1611] The server automatically or manually labels the preprocessed data. Automatic labeling is performed using a machine learning model, and experts review and correct the labels as needed.

[1612] Input: Preprocessed data

[1613] Data processing: automatic labeling, expert review

[1614] Output: Labeled data

[1615] Step 4: Training the generative AI model

[1616] The server uses the labeled data to train a generative AI model, splits the dataset into training and validation sets, and trains the model using a neural network algorithm.Then, the accuracy of the model is confirmed on the validation dataset.

[1617] Input: Labeled data

[1618] Data processing: data division, model training, model evaluation

[1619] Output: A trained generative AI model

[1620] Step 5: Collecting User Input Data

[1621] Users use smart glasses or a smartphone to record and record the sounds and behavior of their dogs and cats. The recorded data is then sent from the device to a server.

[1622] Input: Dog and cat meows and behavior (real-time)

[1623] Data processing: audio and video recording, data transmission

[1624] Output: Recorded data sent to the server

[1625] Step 6: Data analysis and translation

[1626] The server analyzes the received data and generates translation results using a generative AI model. Through analysis, it determines the pet's emotions and intentions.

[1627] Input: Audio and video data

[1628] Data processing: Analysis with generative AI models

[1629] Output: Translation result (pet's emotions and intentions)

[1630] Step 7: Further processing of the emotion engine

[1631] To recognize the user's emotions, the device captures the user's facial expressions and voice in real time. The server analyzes this data and recognizes the user's emotional state, such as "happy," "surprised," or "sad."

[1632] Input: User's facial expression and voice data

[1633] Data processing: Analysis using emotion engine

[1634] Output: User's emotional state

[1635] Step 8: Generate a response based on the analysis results

[1636] The server generates an appropriate response based on the user's emotional state. For example, a translation result indicating the pet's intentions is generated, along with a response based on the user's emotions.

[1637] Input: User's emotional state, pet translation results

[1638] Data processing: response generation

[1639] Output: Translation results and responses for the user

[1640] Step 9: Sending the response and translation

[1641] The generated translation results and responses are sent in real time to the user's device, where the user can check the results on the device's display screen or via voice output.

[1642] Input: Translation result and response

[1643] Data processing: Sending results

[1644] Output: Display on user terminal

[1645] Step 10: Serve interactive content

[1646] The system provides users with interactive content that responds to the pet's condition, further enriching the communication between pets and their owners.

[1647] Input: Pet state, user emotional state

[1648] Data processing: Content generation

[1649] Output: Providing interactive content

[1650] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1651] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1652] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1653] [Fourth embodiment]

[1654] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1655] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1656] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1657] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1658] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1659] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1660] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1661] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1662] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1663] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1664] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1665] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1666] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1667] System Overview

[1668] This invention is a system that translates the meows and behaviors of dogs and cats. It uses generative AI to analyze data on dog and cat meows and behaviors, and outputs the translation results as language. This system allows users to deepen their communication with their pets and understand their emotions and intentions.

[1669] Program Details

[1670] Data collection

[1671] The server collects dog and cat meow and behavior data from online audio databases, social media, video sharing sites, etc. This allows for the collection of a sufficient amount of data to be used as training data.

[1672] 1. The server scrapes data from a specific URL.

[1673] 2. The server stores the collected data in a database.

[1674] Data Preprocessing

[1675] The server performs preprocessing on the collected data, such as noise removal, normalization, and resizing.

[1676] 1. The server performs filtering to remove noise from the data.

[1677] 2. The server converts the audio data into a spectrogram and resizes the image data to an appropriate size.

[1678] Labeling

[1679] The server automatically or manually assigns labels to the preprocessed data.

[1680] 1. The server automates labeling using a machine learning model.

[1681] 2. Human experts review and correct the labels, if necessary.

[1682] Training generative AI models

[1683] The server uses the labeled data to train a generative AI model.

[1684] 1. The server splits the dataset into training and validation sets.

[1685] 2. The server trains the model using a neural network algorithm.

[1686] 3. The server evaluates the trained model on the validation dataset to confirm its accuracy.

[1687] Translation system execution

[1688] The user uses an input device (e.g., smart glasses) to record and film the sounds and behavior of their dog or cat.

[1689] 1. The user turns on the smart glasses and records the sounds and behavior of their pet.

[1690] 2. The device sends the recording data to the server.

[1691] The server analyzes the received data and generates a translation result.

[1692] 1. The server analyzes the data using a generative AI model and generates translation results.

[1693] 2. The server sends the translation results to the device.

[1694] The user checks the translation results through the terminal.

[1695] 1. The device displays or speaks the translation results.

[1696] 2. The user checks the results on the device's display screen or audio output.

[1697] Specific examples

[1698] Example 1: Translating a dog's bark

[1699] 1. A user records a dog barking using smart glasses.

[1700] 2. The device sends the data to the server.

[1701] 3. The server analyzes based on the model that "the dog is barking because it is hungry."

[1702] 4. The server sends the translation results to the device.

[1703] 5. The user checks the translation results on the smart glasses.

[1704] Example 2: Translating cat behavior

[1705] 1. A user uses smart glasses to record the cat's behavior.

[1706] 2. The device sends the data to the server.

[1707] 3. Based on the model, the server analyzes that "the cat is jumping because it wants to play."

[1708] 4. The server sends the translation results to the device.

[1709] 5. The user checks the translation results on the smart glasses.

[1710] By using this system, users can more easily understand what their pets are thinking and want, allowing them to build better relationships with their pets.

[1711] The processing flow will be explained below.

[1712] Step 1: Data collection

[1713] The server scrapes audio data of dog and cat meows, as well as image and video data of their behavior, from the Internet.

[1714] The server stores the scraped data in a database.

[1715] The server categorizes and organizes the data, preparing it for further processing.

[1716] Step 2: Data Preprocessing

[1717] The server performs noise removal filtering on the collected audio data.

[1718] The server converts the audio data into a spectrogram.

[1719] The image and video data collected by the server is resized and converted into a format suitable for analysis.

[1720] The server stores the preprocessed data in a database.

[1721] Step 3: Data Labeling

[1722] The server applies an automatic labeling algorithm to the preprocessed data.

[1723] The server assigns labels such as "hungry" or "want to play" to the bird's cry data.

[1724] The server assigns labels such as "relaxed" or "excited" to the behavioral data.

[1725] If necessary, the server also performs manual labeling by an expert.

[1726] Step 4: Training the generative AI model

[1727] The server splits the labeled data into a training dataset and a validation dataset.

[1728] The server uses the training dataset with a neural network to train a generative AI model.

[1729] The server evaluates the trained model on the validation dataset to confirm accuracy and performance.

[1730] The server will improve or retrain the model as needed based on the evaluation results.

[1731] Step 5: Collect input data

[1732] The user uses smart glasses to record the sounds of a dog or cat meowing.

[1733] The user uses smart glasses to record the behavior of their dog or cat.

[1734] The device sends the recording data to the server.

[1735] Step 6: Data analysis and translation

[1736] The server receives the data sent from the terminal.

[1737] The server uses the generative AI model to analyze the incoming data.

[1738] The server generates a translation result based on the analysis, such as "This sound indicates that the bird is hungry."

[1739] Step 7: Send the translation

[1740] The server transmits the generated translation result to the terminal.

[1741] The device receives the translation result.

[1742] Step 8: Displaying the translation results

[1743] The terminal outputs the translation results on the display screen.

[1744] The user checks the displayed translation results and understands the pet's intentions and emotions.

[1745] Example 1

[1746] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1747] In recent years, there has been an increasing demand for understanding the emotions and intentions of pets from their vocalizations and behaviors. However, current technologies lack the means for owners to gain a deeper understanding of their pets' states. Therefore, an effective system for improving communication between pets and their owners is needed.

[1748] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1749] In this invention, the server includes means for collecting audio data, means for collecting video data, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording audio and video via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to the user's terminal, and means for displaying the translation result on the user's terminal, thereby enabling a user to understand the emotions and intentions of their pet and deepen communication with their pet.

[1750] "Audio data" means data recorded in electronic form of animal calls or other sounds.

[1751] "Video data" refers to data that records the behavior and appearance of animals in electronic format.

[1752] "Preprocessing" is the process of performing processes such as noise removal, normalization, and resizing on collected data to prepare it in a format suitable for analysis.

[1753] A "label" is a tag or annotation that is attached to data to identify the type or state of the subject of analysis.

[1754] A "generative AI model" is an artificial intelligence model that is trained based on collected, preprocessed, and labeled data to provide appropriate translation results or predictions for specific input data.

[1755] An "input device" refers to a device that a user uses to record their pet's sounds and behavior, such as smart glasses or a smartphone.

[1756] "Analysis" refers to the act of analyzing input data using a generative AI model and inferring translation results and intent based on that data.

[1757] The "translation result" is information obtained from the analyzed data that describes the animal's emotions and intentions.

[1758] "Terminal" refers to the device used by the user to check the translation results, such as smart glasses or smartphones.

[1759] The present invention relates to a system for analyzing audio data and video data to understand the emotions and intentions of animals. How the present invention can be implemented will be described in detail below.

[1760] System Overview

[1761] The system primarily consists of a server, a terminal, and an input device used by the user. The server collects audio and video data, preprocesses it, and labels it. The labeled data is then used to train a generative AI model and analyze new data obtained from the user's input device. The terminal receives the translation results and displays them to the user.

[1762] Hardware and software used

[1763] Server: Collects data, preprocesses it, labels it, trains generative AI models, and analyzes it. It uses software libraries and tools such as Python, TensorFlow, PyTorch, Librosa, BeautifulSoup, OpenCV, and MySQL.

[1764] Terminal: A device used by a user as an input device, such as smart glasses or a smartphone, that records audio and video and transmits the data to a server.

[1765] Input device: Smart glasses or smartphones are examples of input devices that record pet sounds and behavior in real time.

[1766] Processing flow

[1767] Data collection: The server collects audio and video data from online audio databases, social media sites, and video sharing sites. The data is scraped using Python's BeautifulSoup library and stored in a MySQL database.

[1768] Data preprocessing: The server performs preprocessing on the collected data, such as noise removal, normalization, and resizing. It uses the Python Librosa library to remove noise from the audio data, Matplotlib to generate spectrograms, and OpenCV to resize the images.

[1769] Labeling: The server automatically assigns labels using machine learning models, and experts review and correct the labels as needed. A classifier is trained from existing labeled data using frameworks such as TensorFlow, and new data is labeled.

[1770] Training the generative AI model: The server uses the labeled data to train the generative AI model. It uses Python's TensorFlow or PyTorch to build a neural network algorithm and utilizes a GPU for fast training.

[1771] Translation system execution: The user uses an input device such as smart glasses to record and record the sounds and behavior of their pet. The device compresses this data in real time and uploads it to the server via HTTPS. The server then uses a generative AI model to analyze the received data, generate translation results, and send them to the device. Finally, the device displays or outputs the translation results as audio, allowing the user to confirm the results.

[1772] Specific examples

[1773] Example 1: Translating a dog's bark

[1774] 1. A user uses the smart glasses to record the sound of a dog barking, for example, by pressing a button to record the sound of a dog barking.

[1775] 2. The device sends the data to the server.

[1776] 3. Based on the generated AI model, the server analyzes that "the dog is barking because it is hungry."

[1777] 4. The server sends the translation results to the device.

[1778] 5. The user checks the translation results on the smart glasses. The message "I'm hungry" appears on the display and the user reads it.

[1779] Example 2: Translating cat behavior

[1780] 1. A user uses smart glasses to record the cat's behavior, for example, pressing a button to record the cat jumping.

[1781] 2. The device sends the data to the server. The recorded data is compressed in real time and sent to the server.

[1782] 3. Based on the generated AI model, the server analyzes that "the cat is jumping because it wants to play."

[1783] 4. The server sends the translation results to the device.

[1784] 5. The user checks the translation results on the smart glasses. The message "I want to play" appears on the display and the user reads it.

[1785] In this way, by using the system of the present invention, the user can understand the emotions and intentions of their pet and deepen communication with their pet.

[1786] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1787] Step 1:

[1788] Data collection

[1789] The server scrapes data from a specific URL, for example, using Python's BeautifulSoup library to retrieve audio and video files from social media posts.

[1790] Input: A specific URL list.

[1791] Output: Collected audio and video data.

[1792] What it does: The server analyzes the web page, extracts links to audio and video files, and downloads those files.

[1793] Step 2:

[1794] Database storage

[1795] The server stores the collected data in a database. The MySQL database stores the paths and metadata of audio and video files.

[1796] Input: Collected audio and video data.

[1797] Output: Path and metadata of the data stored in the database.

[1798] What happens: The server inserts each file's path and associated metadata (date and time of acquisition, original URL, etc.) into a MySQL database.

[1799] Step 3:

[1800] Data Preprocessing

[1801] The server performs filtering to remove noise. The Librosa library in Python is used to remove noise from the audio data.

[1802] Input: Collected audio data.

[1803] Output: Clear audio data with noise removed.

[1804] What it does: The server uses Librosa to load the audio data and applies a specific noise filter to remove unwanted noise.

[1805] The server converts the audio data into a spectrogram and resizes the image data to an appropriate size. The spectrogram is generated using Python's Matplotlib and the image data is resized using OpenCV.

[1806] Input: Denoised audio data and collected video data.

[1807] Output: Spectrogram image and resized video data.

[1808] What it does: Audio data is converted to a spectrogram using Matplotlib, and video data is resized to a consistent size using OpenCV.

[1809] Step 4:

[1810] Labeling

[1811] The server automates labeling using machine learning models, training classifiers using TensorFlow and other tools to label new data.

[1812] Input: Preprocessed audio and video data (spectrogram images and resized video data).

[1813] Output: Labeled data.

[1814] What it does: The server runs a labeling algorithm to automatically assign an appropriate label (e.g., dog barking, cat jumping, etc.) to each piece of data.

[1815] The server then has human experts review and correct the labels as needed.

[1816] Input: Auto-labeled data.

[1817] Output: Verified and corrected labeled data.

[1818] Specific operation: Experts review the labels of each data through a web interface and correct them if necessary.

[1819] Step 5:

[1820] Training generative AI models

[1821] The server splits the dataset into training and validation sets using the train_test_split function from scikit-learn.

[1822] Input: A labeled dataset.

[1823] Output: Split training and validation data.

[1824] Specific operation: The server splits the labeled dataset into training and validation sets in an 80:20 ratio.

[1825] The server trains the model using a neural network algorithm. The model is built using TensorFlow or PyTorch, and trained quickly using a GPU.

[1826] Input: Training data.

[1827] Output: A trained generative AI model.

[1828] What it does: The server configures the model architecture and iteratively inputs data to tune the model parameters.

[1829] The server evaluates the trained model on a validation dataset to confirm its accuracy.

[1830] Input: A trained generative AI model, and validation data.

[1831] Output: Model performance metrics (accuracy, sensitivity, specificity, etc.).

[1832] Specific operation: The server evaluates the model's performance using the validation dataset and logs the results.

[1833] Step 6:

[1834] Translation system execution

[1835] A user uses an input device such as smart glasses to record and film the sounds and behavior of a dog or cat.

[1836] Input: Pet sounds and behavior.

[1837] Output: Recorded data.

[1838] Specific operation: The user activates the audio and video recording function of the smart glasses and records the sounds and behavior of their pet.

[1839] The device sends the audio and video data to the server, where it is compressed in real time and uploaded to the server via HTTPS.

[1840] Input: Recorded data.

[1841] Output: Data uploaded to the server.

[1842] Specific operation: The device compresses the recorded data and sends it to the server via the Internet.

[1843] The server uses the generative AI model to analyze the incoming data and generate translation results.

[1844] Input: Data uploaded to the server.

[1845] Output: The translation result.

[1846] Specific operation: The server inputs the received data into the generated AI model, analyzes the pet's emotions and intentions, and generates translation results.

[1847] The server generates the translation results and sends them to the device, which returns them in JSON format and reformats them for display.

[1848] Input: Translation result.

[1849] Output: The translation result sent to your device.

[1850] Specific operation: The server packages the translation results in JSON format and sends them to the device.

[1851] The device displays or speaks the translation results. The smart glasses display the results as text and output the results as speech from the speakers.

[1852] Input: The translation result sent to the device.

[1853] Output: The displayed or spoken translation result.

[1854] Specific operations: The device displays the translation results to the user by displaying the text on a display or by using a speech synthesis system to output the results aloud from a speaker.

[1855] The user checks the results on the device's display screen or audio output.

[1856] Input: The displayed or spoken translation result.

[1857] Output: The translation result as recognized by the user.

[1858] Specific operation: The user checks the translation results by looking at the display screen of the smart glasses and listening to the voice output.

[1859] (Application example 1)

[1860] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1861] Conventional systems that translate the meows and behavior of dogs and cats collect data on dog and cat meows and behavior from online audio databases, social media, video sharing sites, etc., and analyze it using a generative AI model. However, current systems cannot display the pet's emotions and needs on a display device in real time, making it difficult for users to immediately check the pet's condition. This makes it difficult for, for example, pet shop employees and customers to quickly understand the pet's condition, making it difficult to provide appropriate responses and services.

[1862] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1863] In this invention, the server includes means for collecting data on dog and cat meows, means for collecting data on dog and cat behavior, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording dog and cat meows and behaviors via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to the user's terminal, means for displaying the translation result on the user's terminal, and means for displaying the pet's meows and behaviors on a display device in real time, thereby enabling the user to visually check the status of their pet in real time and provide appropriate responses and services immediately.

[1864] "Means for collecting data on dog and cat meows" refers to devices or functions that record sounds made by dogs and cats and store them for analysis.

[1865] "Means for collecting data on the behavior of dogs and cats" refers to devices or functions that record the behavior of dogs and cats and collect data to analyze their behavior.

[1866] The "means for preprocessing the collected data" refers to a device or function that performs preprocessing such as noise removal and data normalization on the collected audio and video data.

[1867] The "means for assigning labels to preprocessed data" refers to a device or function that automatically or manually assigns labels to preprocessed data according to their contents.

[1868] A "means for training a generative AI model using labeled data" is a device or function that uses labeled data to train an AI model and improve its accuracy.

[1869] "Means for recording the meows and behavior of dogs and cats through an input device used by a user" refers to a device or function that records the meows and behavior of dogs and cats through an input device used by a user, such as smart glasses.

[1870] "Means for analyzing data obtained from a user's input device using a generative AI model and generating a translation result" refers to a device or function that analyzes data sent from an input device using an AI model and translates the results into text.

[1871] The "means for transmitting the generated translation result to the user's terminal" refers to a device or function that transmits the translation result generated by the server to the user's smart glasses, smartphone, etc.

[1872] The "means for displaying the translation result on the user's terminal" refers to a device or function that visually displays the translation result on the user's terminal.

[1873] The "means for displaying the sounds and behavior of a pet on a display device in real time" refers to a device or function that captures the sounds and behavior of a pet in real time and immediately displays the results on a display device.

[1874] System Overview

[1875] This invention is a system that translates the sounds and behaviors of dogs and cats. Using a generative AI model, this data is analyzed and the pet's emotions and intentions are conveyed to the user. In particular, this invention has the function of displaying the pet's sounds and behaviors in real time on a display device, allowing pet shop employees and customers to instantly check the pet's status.

[1876] Program Details

[1877] Data collection methods

[1878] The server collects dog and cat meow and behavior data from online audio databases, social media, video sharing sites, etc. This allows for the collection of a sufficient amount of data to be used as training data.

[1879] Data preprocessing measures

[1880] The server performs preprocessing on the collected data, such as noise removal and normalization. For example, it filters the audio data to remove noise and converts it into a spectrogram. It also resizes the image data to an appropriate size.

[1881] Labeling method

[1882] The server automatically labels the preprocessed data using a machine learning model, and experts review and correct the labels if necessary.

[1883] A means of training generative AI models

[1884] The server trains a generative AI model using the labeled data, splits the dataset into training and validation sets, and trains the model using a neural network algorithm.The model is then evaluated on the validation dataset to confirm its accuracy.

[1885] Translation system implementation

[1886] Users use input devices such as smart glasses to record and record the sounds and behavior of their dogs and cats. The device then sends the recorded data to a server, which then analyzes the data using a generative AI model to generate a translation. The resulting translation is then sent back to the device, where the user can view it on their smart glasses or smartphone.

[1887] Real-time display means

[1888] The server analyzes the pet's sounds and behavior in real time and immediately displays the results on a display device, allowing pet shop staff and customers to understand the pet's condition on the spot.

[1889] Specific examples

[1890] Example 1: Translating a dog's bark

[1891] The user records the sound of a dog barking using smart glasses. The device sends the data to the server, which then uses the model to analyze the sound and concludes, "The dog is barking because it is hungry." The server then sends the translation results to the device, where the user can check them on the smart glasses.

[1892] Example 2: Translating cat behavior

[1893] The user records the cat's behavior using smart glasses. The device sends the data to the server, which then analyzes it based on the model and concludes that "the cat is jumping because it wants to play." The server then sends the translation results to the device, where the user can check them on the smart glasses.

[1894] Prompt Sentence Examples

[1895] "Send the data to the Pet Communication Guide app and output the translation result as 'This dog is hungry.'"

[1896] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1897] Step 1:

[1898] The user wears smart glasses and records the sounds and behavior of their dog or cat. As input, the user operates the audio / video recording function of the smart glasses to capture the voice and video of their pet. As output, the audio / video data is stored in the smart glasses' internal memory.

[1899] Step 2:

[1900] The device sends audio and video data from the user's smart glasses to the server. The audio and video data stored on the smart glasses is used as input. The data is sent to the server via the Internet. The server receives the data as output.

[1901] Step 3:

[1902] The server performs preprocessing on the received data. It uses the transmitted audio and video data as input. The server applies noise reduction filters and data normalization algorithms, converts audio data into spectrograms, and resizes video data. As output, it produces preprocessed audio and video data.

[1903] Step 4:

[1904] The server assigns labels to the preprocessed data. As input, it uses preprocessed audio and video data and automatically labels them using a generative AI model. As output, a labeled dataset is generated.

[1905] Step 5:

[1906] The server trains a generative AI model using the labeled data. It uses the labeled dataset as input and applies a neural network algorithm. It splits the dataset into training and validation parts, and trains and evaluates the model. The output is a trained generative AI model.

[1907] Step 6:

[1908] The server uses a generative AI model to analyze data obtained from the user's input device and generate translation results. Raw audio and video data sent by the user is used as input. Analysis is performed using the generative AI model, and translation results of the pet's cries and behavior are output. The output is a translated text.

[1909] Step 7:

[1910] The server sends the translation results to the user's device. The translation result text generated by the server is used as input. The data is sent via the Internet to the user's smart glasses or smartphone. The translation results arrive at the device as output.

[1911] Step 8:

[1912] The user's device displays the translation result. It uses the translation result text received from the server as input. It shows the translation result to the user through the device's display or audio output. As an output, it generates a state that allows the user to confirm the translation result visually or audibly.

[1913] Step 9:

[1914] The server displays the pet's cries and behavior in real time on a display device. Recorded and video data acquired in real time is used as input. A generative AI model instantly analyzes the data and generates translation results. The real-time translation results are displayed on a display device as output.

[1915] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1916] System Overview

[1917] This invention combines a system that translates the meows and behaviors of dogs and cats with an emotion engine that recognizes the user's emotions, further deepening two-way communication between pets and users. It uses generative AI to analyze dog and cat meows and behavior data and output the translation results as language, and also provides a function to recognize the user's emotions in real time and generate appropriate responses.

[1918] Program Details

[1919] Data collection

[1920] The server scrapes the internet for audio data of cats and dogs' meows, as well as image and video data of their behavior, collecting a sufficient amount of data to be used as training data.

[1921] 1. The server scrapes data from a specific URL.

[1922] 2. The server stores the collected data in a database.

[1923] Data Preprocessing

[1924] The server performs preprocessing such as noise removal, normalization, and resizing on the collected data.

[1925] 1. The server performs filtering to remove noise from the data.

[1926] 2. The server converts the audio data into a spectrogram and resizes the image data to an appropriate size.

[1927] Labeling

[1928] The server automatically or manually assigns labels to the preprocessed data.

[1929] 1. The server automates labeling using a machine learning model.

[1930] 2. Human experts review and correct the labels, if necessary.

[1931] Training generative AI models

[1932] The server uses the labeled data to train a generative AI model.

[1933] 1. The server splits the dataset into training and validation sets.

[1934] 2. The server trains the model using a neural network algorithm.

[1935] 3. The server evaluates the trained model on the validation dataset to confirm its accuracy.

[1936] Collecting user input data

[1937] The user uses an input device (e.g., smart glasses) to record and film the sounds of a dog or cat meowing.

[1938] 1. The user turns on the smart glasses and records the sounds and behavior of their pet.

[1939] 2. The device sends the recording data to the server.

[1940] Data analysis and translation

[1941] The server analyzes the received data and generates translation results using a generative AI model.

[1942] 1. The server analyzes the data using a generative AI model and generates translation results.

[1943] 2. The server uses the analysis results to translate the dog or cat's emotions and intentions.

[1944] Additional processing for the emotion engine

[1945] The server analyzes the user's input data using an emotion engine that recognizes the user's emotions.

[1946] 1. The device captures the user's voice and facial expressions in real time using a camera and microphone.

[1947] 2. The server analyzes the user's facial expressions and voice data to recognize their emotional state, such as "happy," "surprised," or "sad."

[1948] 3. The server generates an appropriate response based on the user's emotional state.

[1949] Sending translation results and responses

[1950] The server sends the generated translation result and a response based on the user's emotions to the terminal.

[1951] 1. The server sends the translation results and emotional responses to the device in real time.

[1952] 2. The device displays the translation results and response.

[1953] Displaying translation results and responses

[1954] The user can check the translation results and emotion-based responses through the device.

[1955] 1. The device outputs the translation results on the display screen.

[1956] 2. The device displays an appropriate response to the user in voice or text.

[1957] 3. The user can check the results on the device's display screen or audio output, making it easier to understand the pet's intentions and emotions.

[1958] Specific examples

[1959] Example 1: Translating dog barks and responding to user emotions

[1960] 1. A user records a dog barking using smart glasses.

[1961] 2. The device sends the data to the server.

[1962] 3. Based on the model, the server analyzes that "the dog is barking because it is hungry."

[1963] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "happy."

[1964] 5. The server generates a response saying "The dog says it's hungry" along with the translation result, "You look happy too."

[1965] 6. The server sends the translation results and response to the device.

[1966] 7. The user checks the translation results and responses on the smart glasses.

[1967] Example 2: Translating cat behavior and responding to user emotions

[1968] 1. A user uses smart glasses to record the cat's behavior.

[1969] 2. The device sends the data to the server.

[1970] 3. Based on the model, the server analyzes that "the cat is jumping because it wants to play."

[1971] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "surprised."

[1972] 5. The server generates a response with the translation result "The cat wants to play" and "You look surprised too."

[1973] 6. The server sends the translation results and response to the device.

[1974] 7. The user checks the translation results and responses on the smart glasses.

[1975] By using this system, users can more easily understand what their pets are thinking and wanting, deepening two-way communication with them. Furthermore, by providing responses based on the user's emotions, it can help to improve the relationship between owners and their pets.

[1976] The processing flow will be explained below.

[1977] Step 1: Data collection

[1978] The server scrapes audio data of dog and cat meows, as well as image and video data of their behavior, from the Internet.

[1979] The server stores the scraped data in a database.

[1980] The server categorizes and organizes the data, preparing it for further processing.

[1981] Step 2: Data Preprocessing

[1982] The server performs noise removal filtering on the collected audio data.

[1983] The server converts the audio data into a spectrogram.

[1984] The image and video data collected by the server is resized and converted into a format suitable for analysis.

[1985] The server stores the preprocessed data in a database.

[1986] Step 3: Data Labeling

[1987] The server applies an automatic labeling algorithm to the preprocessed data.

[1988] The server assigns labels such as "hungry" or "want to play" to the bird's cry data.

[1989] The server assigns labels such as "relaxed" or "excited" to the behavioral data.

[1990] If necessary, the server also performs manual labeling by an expert.

[1991] Step 4: Training the generative AI model

[1992] The server splits the labeled data into a training dataset and a validation dataset.

[1993] The server uses the training dataset with a neural network to train a generative AI model.

[1994] The server evaluates the trained model on the validation dataset to confirm accuracy and performance.

[1995] The server will improve or retrain the model as needed based on the evaluation results.

[1996] Step 5: Train your Emotion Engine

[1997] The server collects the user's voice and facial expression data and uses this as training data.

[1998] The server trains a machine learning model to recognize user emotions.

[1999] The server evaluates the trained emotion recognition model on validation data to confirm accuracy and performance.

[2000] Step 6: Collecting User Input Data

[2001] The user uses smart glasses to record the sounds of a dog or cat meowing.

[2002] The user uses smart glasses to record the behavior of their dog or cat.

[2003] The device sends the recording data to the server.

[2004] Step 7: Data analysis and translation

[2005] The server receives the data sent from the terminal.

[2006] The server uses the generative AI model to analyze the incoming data.

[2007] The server generates a translation result such as "This sound indicates that the bird is hungry" as an analysis result.

[2008] Step 8: Recognizing User Emotions

[2009] The device acquires the user's voice and facial expression data in real time.

[2010] The server analyzes the user's facial expressions and voice data to recognize their emotional state, such as "happy" or "surprised."

[2011] The server generates an appropriate response based on the user's emotional state.

[2012] Step 9: Sending translation results and responses

[2013] The server sends the generated translation result and a response based on the user's emotions to the terminal.

[2014] The terminal receives the translation result and the response.

[2015] Step 10: Viewing Translation Results and Responses

[2016] The terminal outputs the translation results on the display screen.

[2017] The terminal displays an appropriate response to the user in voice or text.

[2018] The user can check the results on the device's display screen and audio output to understand the pet's intentions and emotions.

[2019] Example 2

[2020] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2021] To achieve deeper communication between pets and their owners, interactions that take into account not only the pet's cries and actions but also the owner's emotions are necessary. However, while existing systems can translate pets' actions and cries, they lack the functionality to provide responses that take into account the owner's emotions. This leaves the problem of insufficient two-way communication between pets and their owners.

[2022] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2023] In this invention, the server includes means for collecting data on dog and cat meows, means for collecting data on dog and cat behavior, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording dog and cat meows and behaviors via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to a user's terminal, means for displaying the translation result on the user's terminal, means for recognizing the user's emotions, and means for generating a response based on the user's emotions, thereby enabling a response that takes into account not only the pet's meows and behaviors but also the owner's emotions.

[2024] "Data on dog and cat meows" is information that records the waveforms and temporal characteristics of sounds made by dogs and cats.

[2025] "Dog and cat behavior data" is information that records the movements and behavior of dogs and cats in the form of images or videos.

[2026] "Means of collection" refers to technologies for scraping data from the Internet and devices for acquiring data using cameras and microphones.

[2027] "Preprocessing" is the process of converting collected data into a format that is easier to analyze through processes such as noise removal, data normalization, and resizing.

[2028] "Labeling means" refers to technology that automatically or manually assigns tags that indicate meaning or category to data.

[2029] A "generative AI model" is an artificial intelligence algorithm that uses supervised learning based on large amounts of data to perform specific tasks (such as translation).

[2030] "Input devices used by users" refers to devices such as smart glasses and smartphones that users use to record and film the sounds and behavior of their dogs and cats.

[2031] The "means of analyzing and generating translation results" refers to a technology that uses a generative AI model to analyze the received data, translate the emotions and intentions of the dog or cat, and output the results as text.

[2032] "Means for sending to the user's device" refers to the protocols and technologies used to send the generated translation results to the device used by the user.

[2033] "Means for displaying the translation result on the user's terminal" refers to a function for visually displaying the translation result on a user interface or display.

[2034] "Means for recognizing user emotions" refers to technology that analyzes the user's facial expressions and voice and identifies their emotional state (happiness, surprise, sadness, etc.).

[2035] "Means for generating responses based on user emotions" refers to algorithms or technologies that generate appropriate reactions or messages in response to the recognized user emotions.

[2036] The present invention is a system that translates the sounds and behaviors of dogs and cats and recognizes the user's emotions, enabling two-way communication with pets. Specific methods for implementing this system are described below.

[2037] Data collection

[2038] First, the server collects dog and cat meow and behavior data from the Internet. This is done by scraping data using Python's BeautifulSoup library or Scrapy framework. This collection process is targeted at specific URLs, and the collected data is stored in a database (e.g., MySQL or MongoDB).

[2039] Data Preprocessing

[2040] The server then performs preprocessing on the collected data. This preprocessing includes denoising the audio data, normalizing the data, and resizing the image data. The SciPy and Librosa libraries are used for audio data processing, and the OpenCV library is used for image data processing. Specifically, the server converts the audio data into a spectrogram and resizes the image data to an appropriate size (e.g., 224x224 pixels).

[2041] Labeling

[2042] The server labels the preprocessed data using a pre-trained machine learning model (e.g., ResNet or YOLO). Labeling is automated, but human experts can review and correct the labels if necessary through a web interface.

[2043] Training generative AI models

[2044] The server trains a generative AI model using the labeled data. The dataset is split into training and validation sets (e.g., 80% for training and 20% for validation), and the model is trained using a neural network algorithm (e.g., TensorFlow or PyTorch). Evaluation is performed on the validation dataset to confirm its accuracy.

[2045] Collecting user input data

[2046] Users use smart glasses (e.g., Google Glass) to record and film the sounds and behavior of their dogs and cats. The device then transmits the data from the smart glasses to a server in real time. Protocols such as HTTP and WebSocket are used for transmission.

[2047] Data analysis and translation

[2048] The server analyzes the received data and generates a translation result using a generative AI model. For example, it may generate a result such as "The dog is barking because it is hungry." Based on this analysis result, the dog or cat's emotions and intentions are output as text.

[2049] Additional processing for the emotion engine

[2050] The server uses an emotion analysis engine (e.g., Microsoft Azure's Face API) to recognize the user's emotions. This engine recognizes the user's emotional state (e.g., "happy," "surprised," "sad") from their voice and facial expressions. The server then generates an appropriate response based on this emotional data.

[2051] Sending and displaying translation results and responses

[2052] The server sends the translation results and a response based on the user's emotions to the device. The device then displays these results visually through a user interface (e.g., a smart glasses display). It also outputs audio, if necessary. Based on this information, the user can understand their pet's emotions and intentions and communicate more appropriately.

[2053] Specific examples

[2054] Example 1: Translating dog barks and responding to user emotions

[2055] 1. A user records a dog barking using smart glasses.

[2056] 2. The device sends the data to the server.

[2057] 3. Based on the model, the server analyzes that "the dog is barking because it is hungry."

[2058] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "happy."

[2059] 5. The server generates a response saying "The dog says it's hungry" along with the translation result, "You look happy too."

[2060] 6. The server sends the translation results and response to the device.

[2061] 7. The user checks the translation results and responses on the smart glasses.

[2062] Example 2: Translating cat behavior and responding to user emotions

[2063] 1. A user uses smart glasses to record the cat's behavior.

[2064] 2. The device sends the data to the server.

[2065] 3. Based on the model, the server analyzes that "the cat is jumping because it wants to play."

[2066] 4. The server analyzes the user's facial expressions and voice and recognizes that the user is "surprised."

[2067] 5. The server generates a response with the translation "The cat wants to play" and "You look surprised too."

[2068] 6. The server sends the translation results and response to the device.

[2069] 7. The user checks the translation results and responses on the smart glasses.

[2070] This allows users to understand what their pets are thinking and wanting, deepening two-way communication with them. Furthermore, by providing responses based on the user's emotions, the relationship between pet and owner can be further improved.

[2071] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2072] Step 1:

[2073] Data collection

[2074] The server scrapes dog and cat meow data and behavior data (images and videos) from specific URLs on the Internet. In this step, data is collected using libraries such as BeautifulSoup and Scrapy. The input is a specific URL, and the output is the collected audio data, images, and video data. The server stores the collected data in a database (e.g., MySQL or MongoDB).

[2075] Step 2:

[2076] Data Preprocessing

[2077] The server performs preprocessing on the collected data, including noise removal, data normalization, and resizing. In this step, SciPy and Librosa libraries are used for filtering and spectrogram conversion of audio data. OpenCV is used for image and video data resizing. The input is scraped data, and the output is preprocessed data.

[2078] Step 3:

[2079] Labeling

[2080] The server assigns labels to the preprocessed data. In this step, automatic labeling is performed using a machine learning model (e.g., ResNet or YOLO). To improve the accuracy of the labeling, some of the data is checked and corrected by human experts via a web interface. The input is the preprocessed data, and the output is the labeled data.

[2081] Step 4:

[2082] Training generative AI models

[2083] The server trains a generative AI model using the labeled data. The dataset is split into training and validation sets (e.g., 80% for training and 20% for validation). Training is performed using a neural network algorithm (e.g., TensorFlow or PyTorch), with a set learning rate and number of epochs. The input is the labeled data, and the output is a trained AI model.

[2084] Step 5:

[2085] User data collection

[2086] The user uses smart glasses (e.g., Google Glass) to record and record the sounds and behavior of their dog or cat in real time. The device sends the data from the smart glasses to a server. The input is the data obtained from the smart glasses, and the output is the data sent to the server.

[2087] Step 6:

[2088] Data analysis and translation

[2089] The server analyzes the received data and generates translation results using a trained generative AI model. In this step, the data is input into the generative AI model, and the analysis results output translated text that indicates the pet's emotions and intentions. The input is the data sent from the device, and the output is the translated text.

[2090] Step 7:

[2091] Additional processing for the emotion engine

[2092] The server uses an emotion engine (e.g., Microsoft Azure's Face API) to recognize the user's emotions. The device collects the user's voice and facial expressions in real time through a camera and microphone and sends them to the server. The server analyzes this data to recognize the user's emotional state and generate an appropriate response. The input is the user's facial and voice data, and the output is their emotional state and a response based on it.

[2093] Step 8:

[2094] Sending and displaying translation results and responses

[2095] The server sends the generated translation results and responses based on the user's emotions to the device. The device displays these results visually through a user interface (e.g., a smart glasses display) and, if necessary, provides audio output. The input is the translation results and responses sent from the server, and the output is the translation results and responses displayed to the user.

[2096] (Application example 2)

[2097] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2098] Understanding pet sounds and behaviors is challenging for pet owners. Therefore, there is a need for a way to understand what pets are thinking and needing. There is also a need for a system that can not only translate pet intentions and emotions, but also understand the owner's emotions and respond accordingly, thereby further deepening communication between pets and their owners.

[2099] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data on dog and cat meows, means for collecting data on dog and cat behavior, means for preprocessing the collected data, means for labeling the preprocessed data, means for training a generative AI model using the labeled data, means for recording dog and cat meows and behaviors via an input device used by a user, means for analyzing data acquired from the user's input device with the generative AI model and generating a translation result, means for transmitting the generated translation result to the user's terminal, means for displaying the translation result on the user's terminal, means for analyzing input data using an emotion engine that recognizes the user's emotions, means for generating a response corresponding to the user's emotions based on the analysis result, means for transmitting and displaying the response corresponding to the user's emotions on the user's terminal, and means for providing interactive content corresponding to the dog or cat's intentions. This makes it easier to understand the intentions and emotions of pets and provides appropriate responses corresponding to the owner's emotions, thereby deepening communication between pets and their owners.

[2100] "Means for collecting dog and cat meow data" refers to a device or process for recording pet sounds and electronically recording that data.

[2101] "Means for collecting data on dog and cat behavior" refers to devices or processes for monitoring pet movements using cameras, sensors, etc., and collecting behavioral data.

[2102] "Measures for pre-processing collected data" refers to processes for removing noise from audio and video data, normalizing the data, and converting it into a format suitable for analysis.

[2103] "Means for labeling preprocessed data" refers to the process of assigning appropriate tags or labels to data after preprocessing so that it is possible to identify what each piece of data represents.

[2104] "Means for training a generative AI model with labeled data" refers to the process of using a labeled dataset to train a generative AI model to accurately analyze pet sounds and behaviors.

[2105] "Means for recording the sounds and behavior of dogs and cats through input devices used by users" refers to the process of recording the sounds and behavior of pets using input devices such as smartphones, tablets, and smart glasses.

[2106] "Means for analyzing data obtained from a user's input device using a generative AI model and generating a translation result" refers to the process of receiving data sent from a user's input device, analyzing that data using a generative AI model, and translating the pet's intentions and emotions.

[2107] "Means for sending the generated translation results to the user's device" refers to the process of sending the translation results obtained by the generative AI model to the user's smartphone or tablet device.

[2108] "Means for displaying translation results on the user's device" refers to the process of displaying the pet's intentions and emotions analyzed by the generative AI model on the user's smartphone or tablet.

[2109] "Means for analyzing input data using an emotion engine that recognizes the user's emotions" refers to the process of capturing the user's facial expressions and voice using a camera or microphone, and analyzing them to recognize the user's emotional state.

[2110] The "means for generating a response according to the user's emotions based on the analysis results" refers to a process for generating an optimal response according to the recognized emotional state of the user.

[2111] "Means for transmitting and displaying a response according to the user's emotions to the user's device" refers to the process of transmitting the generated response to the user's smartphone or tablet device and displaying it.

[2112] "Means for providing interactive content according to the intentions of dogs and cats" refers to a process for providing users with educational and entertainment content according to the intentions and state of their pets.

[2113] The present invention is a system that translates the sounds and behaviors of dogs and cats and recognizes the user's emotions, thereby deepening communication between the user and the pet. The system of the present invention is implemented through the following steps.

[2114] System Overview

[2115] The system analyzes pet sounds and behaviors and provides multiple ways to recognize the user's emotions. The system then analyzes the collected data using a generative AI model to generate translation results and responses based on the user's emotions.

[2116] Data collection

[2117] The server collects the meows and behavioral data of dogs and cats. Pet sounds are recorded using a microphone, and behavioral data is recorded using a camera. These data are stored in cloud storage and later pre-processed.

[2118] Data Preprocessing

[2119] The collected data undergoes preprocessing such as noise removal and normalization. Specifically, the server uses a noise filter to remove noise from the audio data, converts the audio data into a spectrogram, and resizes the image data to an appropriate size.

[2120] Labeling

[2121] Once preprocessing is complete, the data is labeled with the meaning and intent of each piece of data. The server automatically assigns the labels using machine learning algorithms, and experts review and correct the labels as needed.

[2122] Training generative AI models

[2123] A generative AI model is trained using the labeled data. The dataset is split into training and validation datasets, and a neural network algorithm is used. Once trained, the model's accuracy is verified on the validation dataset.

[2124] Collecting user input data

[2125] Users can record and record the sounds and behavior of their dogs and cats in real time using input devices such as smart glasses or smartphones, and the recorded data is sent from the device to a server.

[2126] Data analysis and translation

[2127] The server analyzes the received data and generates translation results using a generative AI model. Based on the analysis results, the emotions and intentions of the dog or cat are translated.

[2128] Additional processing for the emotion engine

[2129] To recognize the user's emotions, the device captures the user's facial expressions and voice in real time. The server analyzes this data and recognizes the user's emotional state, such as "happy," "surprised," or "sad." An appropriate response is generated based on the user's recognized emotional state.

[2130] Sending translation results and responses

[2131] The generated translation results and responses are sent to the user's device in real time, where they can check the results on the device's display screen or as audio output, making it easier to understand their pet's intentions and emotions.

[2132] Providing interactive content

[2133] The system provides users with interactive content tailored to their pet's state. For example, if a pet expresses a desire to play, the system suggests activities for the user to play with the pet.

[2134] Specific examples

[2135] The user uses smart glasses to record the sound of their dog barking and sends it to the server. Using a generative AI model, the server analyzes the sound and concludes that "the dog is barking because it is hungry," and recognizes the user's facial expression and voice as "happy." The server then generates a translation result of "The dog says it is hungry," along with a response of "You look happy, too." The user can then check the translation result and response on the smart glasses, instantly understanding their pet's intentions and emotions.

[2136] Prompt Sentence Examples

[2137] As a concrete example, we can use pet sound data and label pairs in the following format:

[2138] "Dog barking: hungry" "Cat behavior: wants to play"

[2139] This allows users to gain a deeper understanding of their pet's intentions and emotions, enabling richer communication.

[2140] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2141] Step 1: Data collection

[2142] The server scrapes the internet for audio data of cats and dogs' meows, as well as image and video data of their behavior. The data is retrieved from specific URLs and stored in a database. This allows for the collection of a large amount of data that can be used as training data.

[2143] Input: Specific URL

[2144] Data processing: scraping and data storage

[2145] Output: Audio, image, and video data stored in a database

[2146] Step 2: Data Preprocessing

[2147] The server performs preprocessing such as noise removal, normalization, and resizing on the data collected. Noise is removed from the audio data and it is converted into a spectrogram. Image data is resized to an appropriate size to make it easier to process.

[2148] Input: Collected audio, image, and video data

[2149] Data processing: noise removal, normalization, resizing

[2150] Output: Preprocessed data

[2151] Step 3: Labeling

[2152] The server automatically or manually labels the preprocessed data. Automatic labeling is performed using a machine learning model, and experts review and correct the labels as needed.

[2153] Input: Preprocessed data

[2154] Data processing: automatic labeling, expert review

[2155] Output: Labeled data

[2156] Step 4: Training the generative AI model

[2157] The server uses the labeled data to train a generative AI model, splits the dataset into training and validation sets, and trains the model using a neural network algorithm.Then, the accuracy of the model is confirmed on the validation dataset.

[2158] Input: Labeled data

[2159] Data processing: data division, model training, model evaluation

[2160] Output: A trained generative AI model

[2161] Step 5: Collecting User Input Data

[2162] Users use smart glasses or a smartphone to record and record the sounds and behavior of their dogs and cats. The recorded data is then sent from the device to a server.

[2163] Input: Dog and cat meows and behavior (real-time)

[2164] Data processing: audio and video recording, data transmission

[2165] Output: Recorded data sent to the server

[2166] Step 6: Data analysis and translation

[2167] The server analyzes the received data and generates translation results using a generative AI model. Through analysis, it determines the pet's emotions and intentions.

[2168] Input: Audio and video data

[2169] Data processing: Analysis with generative AI models

[2170] Output: Translation result (pet's emotions and intentions)

[2171] Step 7: Further processing of the emotion engine

[2172] To recognize the user's emotions, the device captures the user's facial expressions and voice in real time. The server analyzes this data and recognizes the user's emotional state, such as "happy," "surprised," or "sad."

[2173] Input: User's facial expression and voice data

[2174] Data processing: Analysis using emotion engine

[2175] Output: User's emotional state

[2176] Step 8: Generate a response based on the analysis results

[2177] The server generates an appropriate response based on the user's emotional state. For example, a translation result indicating the pet's intentions is generated, along with a response based on the user's emotions.

[2178] Input: User's emotional state, pet translation results

[2179] Data processing: response generation

[2180] Output: Translation results and responses for the user

[2181] Step 9: Sending the response and translation

[2182] The generated translation results and responses are sent in real time to the user's device, where the user can check the results on the device's display screen or via voice output.

[2183] Input: Translation result and response

[2184] Data processing: Sending results

[2185] Output: Display on user terminal

[2186] Step 10: Serve interactive content

[2187] The system provides users with interactive content that responds to the pet's condition, further enriching the communication between pets and their owners.

[2188] Input: Pet state, user emotional state

[2189] Data processing: Content generation

[2190] Output: Providing interactive content

[2191] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2192] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2193] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2194] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determi...

Claims

1. A means of collecting data on dog and cat meows, A means of collecting data on the behavior of dogs and cats, a means for pre-processing the collected data; A means for labeling the preprocessed data; a means for training a generative AI model using the labeled data; A means for recording and filming the meows and behavior of dogs and cats via an input device used by a user; A means for analyzing data acquired from a user's input device using a generative AI model to generate a translation result; means for transmitting the generated translation result to a user's terminal; means for displaying the translation result on the user's terminal; A system including:

2. 10. The system of claim 1, wherein the preprocessing means performs noise removal and data normalization.

3. The system of claim 1 further comprising means for classifying the behavioral data of the dog or cat based on different labels.

Images