system

Abstract

Provide a system. A means for acquiring video or image data of an animal; means for extracting animal facial expressions, gestures, and sounds from said data; means for analyzing the emotion of the animal based on the extracted feature amount; means for visualizing the analyzed emotion data in a format that is easy for humans to understand; means for transmitting the visualized data to a terminal; means for displaying the data in the terminal; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] To achieve smooth communication between animals and humans, it is necessary to accurately understand the emotions and feelings of animals. However, technology for analyzing animal facial expressions, gestures, cries, etc. has not yet been established, and there is a lack of methods for visualizing these in a format that is easy for humans to understand. The objective of this invention is to provide a system that accurately analyzes the emotions and feelings of animals and provides information in a format that is easy for humans to understand. [Means for solving the problem]

[0005] The present invention includes a means for acquiring video or image data of an animal, and a means for extracting the animal's facial expressions, gestures, and cries from the data. It also includes a means for analyzing the animal's emotions based on the extracted features, and a means for visualizing the analyzed emotional data in a format that is easy for humans to understand. This allows for accurate understanding of the animal's emotions and feelings, enabling smooth communication.

[0006] "Animals" refers to living creatures that have emotions and feelings but have different means of communication than humans.

[0007] "Data" refers to media containing information such as animal facial expressions, gestures, and sounds captured in video or image format.

[0008] "Preprocessing" refers to processing to remove noise from acquired data and standardize it.

[0009] "Features" refer to specific parameters extracted from data and used for analysis, such as the shape of an animal's face or the frequency spectrum of its calls.

[0010] "Machine learning algorithms" refer to mathematical models and methods for learning features from data and analyzing animal emotions.

[0011] "Analysis" refers to the process of inferring an animal's emotions based on the extracted features.

[0012] "Visualization" refers to the process of displaying analysis results in a format that is easy for humans to understand (such as icons or graphs).

[0013] "Feedback" refers to additional information or comments provided by the user regarding the analysis results.

[0014] "Retraining" refers to the process of updating a machine learning model with collected feedback data to improve its accuracy.

[0015] The term "system" refers to a device or program that comprehensively includes the above-mentioned means. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The system of the present invention collects video or image data of animals, analyzes the emotions of the animals from the data, and displays them in a format that is easy for humans to understand. Below, the processing of the system's program will be explained in natural language and in detail with concrete examples.

[0038] Retrieving data

[0039] Users take videos and photos of animals using a device (such as a smartphone or tablet), and this data is temporarily stored on the device.

[0040] Uploading data

[0041] The device uploads the captured video and image data to the server. At this time, the format and size of the data being uploaded are checked and it is converted into the appropriate format.

[0042] Data Preprocessing

[0043] The server performs preprocessing on the received data. In the case of video data, the video is broken down into frames, and the image and audio data are stored separately. Noise removal and data standardization are also performed at this stage.

[0044] Feature extraction

[0045] The server uses machine learning algorithms to extract animal features from the data. From the image data, it detects the animal's face and body parts and extracts features such as eye shape, mouth opening and closing position, and ear position. From the audio data, it extracts features such as pitch, frequency spectrum, and sound duration.

[0046] Emotion Analysis

[0047] The server inputs the features into a pre-trained machine learning model to analyze the animal's emotions. This model uses convolutional neural networks (CNNs) and recurrent neural networks (RNNs). As a result of the analysis, the animal's emotions (happiness, anger, anxiety, excitement, etc.) are estimated and tagged.

[0048] Generate visualization data

[0049] The server generates visualization data based on the analyzed emotion data. This visualization data includes icons and graphs that represent the animal's facial expressions. The generated visualization data is sent to the device in JSON format or similar.

[0050] Displaying the results

[0051] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, it displays tags that indicate the animal's emotions, as well as detailed information such as the shape of its eyes and whether its mouth is open or closed.

[0052] Gathering feedback

[0053] The user checks the displayed information and evaluates whether the animal's emotions have been correctly analyzed. Based on the evaluation, the user enters feedback, which is then sent to the server. The feedback is used to improve the accuracy of the system.

[0054] Specific examples

[0055] Dog Emotion Analysis

[0056] 1. The user takes videos of the dog with their smartphone and records the dog's behavior and sounds.

[0057] 2. The device uploads this video data to the server.

[0058] 3. The server extracts frames from the video, separates them from the audio data, and performs preprocessing.

[0059] 4. The server detects the dog's face and body from the image data and extracts features such as the shape of the eyes, whether the mouth is open or closed, and the position of the ears. It also extracts the pitch and frequency spectrum from the audio data.

[0060] 5. The server uses a machine learning model to analyze the dog's emotion as "joy."

[0061] 6. The server generates visualization data based on the analysis results and sends it to the terminal.

[0062] 7. The device will display to the user "Dog is happy" and also show icons such as a wagging tail and raised ears.

[0063] 8. The user checks this information and decides whether the dog is truly happy. If the result is appropriate, the user enters feedback and sends it to the server.

[0064] In this way, the system of the present invention analyzes the emotions of animals and provides them in a format that is easy for humans to understand, thereby realizing smooth communication between animals and humans.

[0065] The processing flow will be explained below.

[0066] Step 1: Capture video or images of your animal

[0067] Users use devices such as smartphones and tablets to take videos or images of animals, and this data includes the animals' facial expressions, movements, sounds, etc.

[0068] Step 2: Save and upload your data

[0069] The device temporarily stores the captured video or image data, then uploads the data to a server via the Internet. At this time, the data format and size are checked and, if necessary, converted to an appropriate format.

[0070] Step 3: Preprocessing the data

[0071] The server receives the uploaded data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates the audio data from the video and saves it. Next, the server performs noise reduction and standardization on the image and audio data.

[0072] Step 4: Feature extraction

[0073] The server extracts animal features from the preprocessed data. For image data, a face detection algorithm is used to detect the animal's face and body parts, and features such as eye shape, mouth opening and closing, and ear position are extracted. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[0074] Step 5: Sentiment Analysis

[0075] The server inputs the extracted features into a machine learning model to analyze the animal's emotions. This model has been pre-trained with a large amount of data and is designed to identify basic animal emotions (happiness, anger, anxiety, excitement, etc.). As a result of the analysis, the animal's emotion is tagged.

[0076] Step 6: Generate visualization data

[0077] The server generates visualization data based on the analyzed emotion data. This visualization data includes animal facial expression icons and charts that graphically represent the analysis results. The generated visualization data is sent to the device in an appropriate format, such as JSON.

[0078] Step 7: View the results

[0079] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, detailed information such as the shape of the eyes and the opening and closing state of the mouth is displayed along with icons and tags that indicate the animal's emotions.

[0080] Step 8: Gather feedback

[0081] The user checks the displayed emotion analysis results and evaluates whether the animal's emotions have been correctly analyzed. This evaluation is performed by comparing the emotion analysis results with the actual behavior and situation of the animal. The user inputs this evaluation result as feedback and sends it to the server.

[0082] Step 9: Relearn with feedback

[0083] The server analyzes the received feedback data and uses it to retrain the sentiment analysis model. This improves the system's analysis accuracy and also improves the accuracy of future sentiment analysis. The retrained model is then used for the next data analysis.

[0084] The above steps will realize a system that accurately analyzes animal emotions and provides them to users, promoting smooth communication between animals and humans.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] Conventional animal emotion analysis systems have had difficulty accurately analyzing animal emotions from video data. Furthermore, the means for displaying the analyzed emotion data in an easy-to-understand manner for users have been insufficient, making it difficult for users to intuitively understand the emotions of animals. To address these issues, there is a need for the development of a system that can analyze animal emotions with high accuracy and provide the data in a visually easy-to-understand format.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes means for decomposing video data into frames and storing the audio data and image data separately, means for removing noise from the data and standardizing it, and means for extracting animal facial expressions, gestures, and cries from the data. This makes it possible to analyze emotions from animal video data with high accuracy and provide the analysis results in a visually easy-to-understand format.

[0090] "Video or image data of animals" refers to video or photographic information of pets or wild animals, which can be used to record the movements and expressions of animals.

[0091] "Terminal" refers to an electronic device such as a smartphone, tablet, or personal computer, which is used to acquire, transmit, and receive data.

[0092] A "server" refers to a computer system that processes, stores, and analyzes data via a network, and receives and processes data from multiple terminals.

[0093] "Means for uploading data" refers to the process by which a user's device sends video or image data to a server via a network.

[0094] "Means of decomposing into frames" refers to the process of dividing video data into individual still images (frames), which allows for detailed image analysis.

[0095] "Denoising and standardisation measures" refers to the process of removing unnecessary information and outliers and converting data into a consistent format to improve data quality.

[0096] "Means for extracting animal facial expressions, gestures, and sounds" refers to the process of detecting specific features from image and audio data and extracting them as data for analysis.

[0097] "Features" refer to specific attributes and patterns extracted from image and audio data, which can be used to analyze an animal's emotions and state.

[0098] A "generative AI model" refers to a model trained using machine learning algorithms that can infer animal emotions from features.

[0099] The "means for generating visualization data" refers to the process of converting the analyzed emotion data into visual information such as icons and graphs in order to display it in a format that is easy for the user to understand.

[0100] "Means for transmitting visualized data to a terminal" refers to a process in which the server transmits the generated visualization data to a user's terminal via a network.

[0101] "Means for displaying data" refers to the process of displaying analysis results and visualized data on a user interface on a user's terminal.

[0102] The system of this invention analyzes animal emotions with high accuracy and provides the user with a visually easy-to-understand format. This system is mainly composed of three elements: the user's terminal, an internet connection, and a server.

[0103] Retrieving data

[0104] Users use devices such as smartphones and tablets to take videos and photos of animals. The device's camera app is used to do this, and the captured data is temporarily stored on the device. For example, a user may take a video of their pet dog and record its various behaviors and sounds.

[0105] Uploading data

[0106] The device uploads the captured video and image data to a server via the Internet. At this time, the data format (JPEG, MP4, etc.) and size are automatically checked, and if necessary, the data is converted to the appropriate format using a library such as FFmpeg.

[0107] Data Preprocessing

[0108] The server breaks down video data into frames and stores the audio and image data separately. Specifically, it uses the OpenCV library to break down the video into frames, and FFmpeg to extract and store the audio data. It also uses SciPy and Pandas to perform noise removal and normalization. For example, the server exports one video frame and the corresponding audio from a video of a dog.

[0109] Feature extraction

[0110] The server uses OpenCV to extract features such as the animal's face, eye shape, ear position, and mouth open / close state from image data. It also uses Librosa to extract pitch, frequency spectrum, and sound duration from audio data. For example, it can extract whether a dog's ears are erect from image data and the pitch of a bark from audio data.

[0111] Emotion Analysis

[0112] The server uses a pre-trained generative AI model (CNN or RNN built using TENSORFLOW (registered trademark) or PyTorch) to analyze the animal's emotions based on the extracted features. This model estimates the animal's emotions from the features and tags them with emotions such as joy, anger, anxiety, and excitement. For example, it can determine that "the dog is happy based on the position of its ears and the shape of its face."

[0113] Generate visualization data

[0114] The server generates visualization data using visualization libraries such as D3.js based on the analyzed emotion data. The generated data includes icons and graphs showing the animal's facial expressions, and is transferred in JSON format. For example, it generates an icon showing a dog's happy state and a graph showing changes in emotion.

[0115] Displaying the results

[0116] The device analyzes the JSON data received from the server and displays the results on the user interface. For example, it displays the message "The dog is happy" along with a tail-wagging icon and a smiling icon.

[0117] Gathering feedback

[0118] The user checks the displayed analysis results and evaluates whether they are appropriate. The evaluation results are sent to the server as feedback and used as learning data for the system. For example, if a user evaluates the results as "correct," that evaluation helps improve the accuracy of the system.

[0119] Specific examples

[0120] 1. The user takes videos of the dog with their smartphone and records the dog's behavior and sounds.

[0121] 2. The device uploads this video data to the server.

[0122] 3. The server extracts frames from the video, separates them from the audio data, and performs preprocessing.

[0123] 4. The server detects the dog's face and body from the image data and extracts features such as the shape of the eyes, whether the mouth is open or closed, and the position of the ears. It also extracts the pitch and frequency spectrum from the audio data.

[0124] 5. The server uses a generative AI model to analyze the dog's emotion as "joy."

[0125] 6. The server generates visualization data based on the analysis results and sends it to the terminal.

[0126] 7. The device will display to the user "Dog is happy" and also show icons of the dog wagging its tail and smiling.

[0127] 8. The user checks this information and decides whether the dog is truly happy or not, and sends the evaluation result to the server as feedback.

[0128] By utilizing generative AI models, it is possible to analyze the complex emotions of animals with high accuracy and present them to users in a format that is very easy to understand.

[0129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0130] Processing Steps

[0131] Step 1: Get the data

[0132] Users use devices such as smartphones and tablets to take videos and photos of animals. This is done using the device's camera app, and the captured data is temporarily stored on the device. The user's operations record the behavior and sounds of the animal being photographed. The input for this process is the video and image data obtained through the camera app, and the output is data temporarily stored in the device's internal storage area.

[0133] Step 2: Upload your data

[0134] The device uploads the captured video and image data to a server via the Internet. At this time, the data format (JPEG, MP4, etc.) and size are automatically checked, and if there are any problems, it is converted to the appropriate format using a library such as FFmpeg. This process uses the video and image data stored on the device as input, and the converted and checked data is sent to the server as output.

[0135] Step 3: Preprocessing the data

[0136] The server breaks down the uploaded video data into frames, and stores the audio and image data separately. Here, the OpenCV library is used to break down the video into frames, and FFmpeg is used to extract and store the audio data. SciPy and Pandas are also used to remove noise from the data and standardize it. The input for this process is the video data uploaded to the server, and the output is frame-by-frame image data and separated audio data.

[0137] Step 4: Feature extraction

[0138] The server uses OpenCV to extract features such as the animal's face, eye shape, ear position, and mouth open / close state from the image data. It also uses Librosa to extract pitch, frequency spectrum, and sound duration from the audio data. The input to this process is the image data and audio data separated and saved in the previous step, and the output is each extracted feature. Specifically, it identifies the position of the dog's ears and eye shape from the image data, and the pitch of the bark from the audio data.

[0139] Step 5: Sentiment Analysis

[0140] The server uses a pre-trained generative AI model (CNN or RNN built using TensorFlow or PyTorch) to analyze the animal's emotions based on the extracted features. This model estimates the animal's emotions from the features and tags them with emotions such as joy, anger, anxiety, and excitement. The input to this process is the extracted features, and the output is estimated emotion data. Specifically, it determines the emotion as "joy" based on the position of the dog's ears and the shape of its face.

[0141] Step 6: Generate visualization data

[0142] The server generates visualized data based on the analyzed emotion data using a visualization library such as D3.js. The generated data includes icons and graphs showing the animal's facial expressions, and is transferred to the terminal in JSON format. The input to this process is the analyzed emotion data, and the output is the generated visualized data. Specifically, it generates an icon showing the dog's happy state and a graph showing changes in emotion.

[0143] Step 7: View the results

[0144] The terminal analyzes the JSON data received from the server and displays the results on the user interface. Specifically, it displays a message saying "The dog is happy" along with a tail-wagging icon, a smiling icon, etc. The input to this process is the JSON data received from the server, and the output is the visual information displayed on the user interface.

[0145] Step 8: Gather feedback

[0146] The user checks the displayed analysis results and evaluates whether they are appropriate. The evaluation results are sent as feedback to the server and used as learning data for the system. The inputs to this process are the user's evaluation and feedback, and the output is feedback data sent to the server. In concrete terms, the user evaluates the results as "correct," and the evaluation is sent to the server.

[0147] (Application example 1)

[0148] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0149] In conventional food delivery services, when delivery personnel visit homes containing animals, it is difficult for them to understand the emotions and state of the animals. As a result, the animals may feel anxious or stressed, making it difficult to provide a safe and comfortable delivery experience. The present invention aims to solve this problem by providing a system that allows delivery personnel to understand the emotional state of animals and take appropriate action.

[0150] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0151] In this invention, the server includes means for acquiring video or image data of the animal, means for extracting the animal's facial expressions, gestures, and cries from the data, means for analyzing the animal's emotions based on the extracted features, means for visualizing the analyzed emotional data in a format that is easy for humans to understand, means for transmitting the visualized data to a terminal, means for displaying the data on the terminal, and means for conveying the animal's emotions to a delivery person during delivery work, thereby enabling the delivery person to understand the animal's emotions and take appropriate action toward the animal.

[0152] "Animal video or image data" refers to digital data of videos or photographs of animals taken with a mobile phone or other device.

[0153] "Facial expressions" refer to changes in emotions shown by the position and shape of each part of an animal's face (e.g., eyes, mouth, ears, etc.).

[0154] A "gesture" refers to a specific action or movement that an animal makes with its body.

[0155] "Cry" refers to the sound that animals make, and is a way of expressing emotions and communicating.

[0156] "Features" refer to numerical values ​​or indicators extracted from image or audio data that are necessary for emotion analysis.

[0157] "Emotion analysis" is the process of using extracted features to estimate an animal's internal emotional state (joy, anger, anxiety, excitement, etc.).

[0158] "Visualization" refers to the process of displaying analyzed emotional data in visual formats such as graphs and icons.

[0159] "Terminal" refers to a digital device for receiving and displaying information, such as a smartphone or tablet.

[0160] "A means of communicating an animal's emotions to the delivery person during delivery work" is a system that notifies the delivery person of the animal's emotional state via a terminal during food delivery.

[0161] System Configuration

[0162] The system of the present invention collects video or image data of animals, analyzes the emotions of the animals from the data, and conveys the emotions to the delivery person. This system is composed of the following elements.

[0163] 1. Device

[0164] The devices used are mobile information terminals such as smartphones and tablets, which are used to acquire video and image data of animals and send them to a server.

[0165] 2. Server

[0166] The server receives video and image data sent from the device, preprocesses the data, extracts features, analyzes emotions, and visualizes the results. Specific technologies used include machine learning algorithms such as face detection and voice feature extraction, as well as convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

[0167] 3. User Interface

[0168] The device comes with a dedicated application that visually displays the analysis results to the delivery person, including icons and messages based on the animal's emotional state.

[0169] Processing Details

[0170] 1. Data Acquisition

[0171] The user (delivery person) takes videos and photos of their pet using their smartphone.

[0172] 2. Uploading data

[0173] The device uploads the captured video and image data to the server, where the data format and size are checked and converted into the appropriate format.

[0174] 3. Data Preprocessing

[0175] The server performs preprocessing on the received data, such as noise removal and data standardization. In the case of video data, it divides it into frames and separates the image data from the audio data.

[0176] 4. Feature Extraction

[0177] The server uses machine learning algorithms to extract features such as the shape of the animal's eyes, the degree to which its mouth is open or closed, and the position of its ears from the image data, as well as the pitch and frequency spectrum from the audio data.

[0178] 5. Emotion Analysis

[0179] The server uses pre-trained CNN or RNN models to analyze the animal's emotions from the extracted features, and the resulting emotion (joy, anger, anxiety, excitement, etc.) is tagged.

[0180] 6. Generate visualization data

[0181] The server generates visualization data based on the analyzed emotion data and sends it to the terminal in JSON format or similar.

[0182] 7. Displaying the results

[0183] The device uses the visualization data received from the server to display the emotional state of the animal on the user interface. For example, a tail-wagging icon is displayed along with the message "The dog is happy."

[0184] Specific examples

[0185] Delivery person visits your home scenario

[0186] When delivering food, the delivery person takes a photo with their smartphone of the dog waiting at home.

[0187] The terminal uploads the photo data to the server.

[0188] The server analyzes the shape of the dog's eyes and the position of its ears from the image data, and extracts sound characteristics from the audio data.

[0189] The server displays the dog's "anxiety" as the analysis result.

[0190] The device displays a warning message to the delivery person saying, "Dog is anxious."

[0191] Delivery personnel use this information to approach quietly and avoid causing stress to the dog.

[0192] Prompt Sentence Examples

[0193] "Analyze a photo of a dog, determine its emotion, and output it in JSON format."

[0194] "Analyze the dog's emotions from this audio data and display the results in a graph."

[0195] This system allows delivery personnel to properly understand the emotional state of animals and provide a safe and comfortable delivery experience.

[0196] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0197] Step 1: Get the data

[0198] Users take videos and image data of animals with their smartphones, and this data is temporarily stored on the device. The input is the taken video and image data, and the output is the temporarily stored data on the device.

[0199] Step 2: Upload your data

[0200] The device uploads the captured video and image data to the server. The input is data temporarily stored on the device, and the output is data sent to the server. At this time, the format and size of the data are checked and converted into the appropriate format.

[0201] Step 3: Preprocessing the data

[0202] The server performs preprocessing on the received data. The input is the data sent to the server, and the output is the preprocessed data. In the case of video data, it is broken down into frames and separated into image data and audio data. Noise removal and data standardization are also performed here.

[0203] Step 4: Feature extraction

[0204] The server uses machine learning algorithms to extract animal features from the data. The input is preprocessed data, and the output is the extracted features. Specifically, eye shape, mouth open / closed state, and ear position are extracted from image data, and pitch and frequency spectrum are extracted from audio data.

[0205] Step 5: Sentiment Analysis

[0206] The server inputs the features into a pre-trained generative AI model (CNN or RNN) and analyzes the animal's emotions. The input is the extracted features, and the output is the emotion analysis result. As a result of the analysis, the animal's emotion (joy, anger, anxiety, excitement, etc.) is estimated and tagged.

[0207] Step 6: Generate visualization data

[0208] The server generates visualization data based on the analyzed emotion data. The input is the emotion analysis results, and the output is visualization data in JSON format. The visualization data includes icons and graphs showing animal facial expressions.

[0209] Step 7: View the results

[0210] The terminal displays the results on the user interface based on the visualization data received from the server. The input is the visualization data sent from the server, and the output is visual information on the emotional state displayed on the terminal. For example, it might display "The dog is happy" and use an icon to show the dog wagging its tail.

[0211] Step 8: Gather feedback

[0212] The user checks the displayed information and evaluates whether the animal's emotion has been correctly analyzed. The input is the emotional state displayed on the device, and the output is the user's evaluation feedback. This feedback is sent to the server and used to improve the accuracy of the system.

[0213] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0214] This invention realizes two-way emotion analysis between animals and humans by combining a system that analyzes and visualizes animal emotions with an emotion engine that recognizes user emotions. Below, the processing of the system's program is explained in natural language and in detail with concrete examples.

[0215] Retrieving data

[0216] Users use a smartphone, tablet, or other device to take videos or images of animals, while simultaneously recording their own facial expressions and tone of voice. This data provides the basis for analyzing the emotions of both the animal and the user.

[0217] Data storage and upload

[0218] The device temporarily stores the captured video or image, along with the user's facial expressions and voice data. The data is then uploaded to a server via the Internet. At this time, the data format and size are checked and converted to the appropriate format.

[0219] Data Preprocessing

[0220] The server receives the uploaded data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates and saves audio data from the video. The user's facial expression and audio data are also preprocessed in the same way, with noise removal and standardization.

[0221] Feature extraction

[0222] The server extracts features of both the animal and the user from the preprocessed data. For image data, a face detection algorithm is used to detect the faces and body parts of the animal and the user, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[0223] Emotion Analysis

[0224] The server inputs the extracted features into a machine learning model to analyze the emotions of animals and users. This model is pre-trained with a large amount of data and is designed to identify basic emotions (happiness, anger, anxiety, excitement, etc.) of animals and users. As a result of the analysis, the emotions of animals and users are tagged.

[0225] Generate visualization data

[0226] The server generates visualization data based on the analyzed emotion data. This visualization data includes facial expression icons for animals and the user, as well as charts that graphically represent the analysis results. The generated visualization data is sent to the device in an appropriate format, such as JSON.

[0227] Displaying the results

[0228] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, detailed information such as the shape of the eyes and the degree to which the mouth is open or closed is displayed along with icons and tags that indicate the emotions of the animal and the user. This allows the results of two-way emotion analysis between the animal and the user to be confirmed.

[0229] Gathering feedback

[0230] The user checks the displayed emotion analysis results and evaluates whether the animal's and the user's emotions have been analyzed correctly. This evaluation is performed by comparing the emotion analysis results with the animal's actual behavior and situation, and the user's own emotions. The user inputs this evaluation result as feedback and sends it to the server.

[0231] Re-learning with feedback

[0232] The server analyzes the received feedback data and uses it to retrain the emotion analysis model for animals and users. This improves the system's analysis accuracy and also improves the accuracy of future emotion analysis. The retrained model is then used for the next data analysis.

[0233] Specific examples

[0234] Emotional analysis of dogs and users

[0235] 1. The user takes a video of the dog with their smartphone and records their own facial expressions and voice.

[0236] 2. The device uploads this video and audio data to the server.

[0237] 3. The server preprocesses the video and audio data and extracts features.

[0238] 4. The server analyzes the animal and user's emotions using a machine learning model and estimates that the dog is "happy" and the user is "pleased."

[0239] 5. The server generates visualization data based on the analysis results and sends it to the terminal.

[0240] 6. The device will display messages to the user such as "The dog is happy" and "You are happy," and also show icons such as a wagging tail and a smiling face.

[0241] 7. The user checks this information and evaluates whether the analysis results are correct, inputs the evaluation results as feedback, and sends them to the server.

[0242] In this way, the system of the present invention realizes smooth communication between animals and humans by analyzing the emotions of both animals and users in both directions and providing the results to the user.

[0243] The processing flow will be explained below.

[0244] Step 1: Get the data

[0245] Users use devices such as smartphones or tablets to take videos or images of animals, while simultaneously recording their own facial expressions and tone of voice, thereby collecting basic data for analyzing the emotions of both the animals and the user.

[0246] Step 2: Save and upload your data

[0247] The device temporarily stores the captured video or image data, along with the user's facial expressions and voice data. The data is then uploaded to a server via the Internet. At this time, the data format and size are checked and converted to the appropriate format.

[0248] Step 3: Preprocessing the data

[0249] The server receives the uploaded data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates and saves audio data from the video. The user's facial expression and audio data are also preprocessed in the same way, with noise removal and standardization.

[0250] Step 4: Feature extraction

[0251] The server extracts features of both the animal and the user from the preprocessed data. For image data, a face detection algorithm is used to detect the faces and body parts of the animal and the user, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[0252] Step 5: Sentiment Analysis

[0253] The server inputs the extracted features into a machine learning model to analyze the emotions of animals and users. This model is pre-trained with a large amount of data and is designed to identify basic emotions (happiness, anger, anxiety, excitement, etc.) of animals and users. As a result of the analysis, the emotions of animals and users are tagged.

[0254] Step 6: Generate visualization data

[0255] The server generates visualization data based on the analyzed emotion data. This visualization data includes facial expression icons for animals and the user, as well as charts that graphically represent the analysis results. The generated visualization data is sent to the device in an appropriate format, such as JSON.

[0256] Step 7: View the results

[0257] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, detailed information such as the shape of the eyes and the degree to which the mouth is open or closed is displayed along with icons and tags that indicate the emotions of the animal and the user. This allows the results of two-way emotion analysis between the animal and the user to be confirmed.

[0258] Step 8: Gather feedback

[0259] The user checks the displayed emotion analysis results and evaluates whether the animal's and the user's emotions have been analyzed correctly. This evaluation is performed by comparing the emotion analysis results with the animal's actual behavior and situation, and the user's own emotions. The user inputs this evaluation result as feedback and sends it to the server.

[0260] Step 9: Relearn with feedback

[0261] The server analyzes the received feedback data and uses it to retrain the emotion analysis model for animals and users. This improves the system's analysis accuracy and also improves the accuracy of future emotion analysis. The retrained model is then used for the next data analysis.

[0262] Through the above steps, the system of the present invention analyzes the emotions of animals and users in both directions and provides the results to the user, thereby realizing smooth communication between animals and humans.

[0263] Example 2

[0264] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0265] Conventional animal emotion analysis systems only analyze animal emotions, and are unable to analyze emotions in both directions with humans. Furthermore, there is no system that can simultaneously analyze the emotions of both humans and animals, visualize them, and promote two-way communication, making it difficult to smoothly communicate with animals.

[0266] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring video or image data of an animal, means for extracting the animal's facial expressions, gestures, and sounds from the data, means for simultaneously acquiring the user's facial expressions and voice data, means for transmitting user and animal data to the server, means for checking the format and size of the data and performing preprocessing, means for analyzing the emotions of the animal and the user based on the extracted features, means for visualizing the analyzed emotional data in a format that is easy for humans to understand, means for transmitting the visualized data to a terminal, means for displaying the data on the terminal, means for receiving feedback from the user and transmitting it to the server, and means for relearning a machine learning model using the feedback. This enables two-way emotion analysis between the animal and the user and smooth communication based on the analysis.

[0267] "Animal video or image data" refers to video or still image data that records the movements and expressions of animals captured by a user.

[0268] "Facial expressions, gestures, and sounds" refer to the facial expressions, body movements, and sounds that animals make, and are used to infer emotions.

[0269] "User facial expression and voice data" refers to data such as facial expressions, tone of voice, and timbre that are recorded when a user uses the system.

[0270] A "server" is a central computer system that receives data sent from terminals via the Internet, analyzes and stores it, and manages feedback.

[0271] "Preprocessing" refers to data processing operations such as noise removal and standardization that are performed on collected data before analysis.

[0272] "Features" are identifiable information extracted from data, such as eye shape, mouth open / closed state, ear position, voice pitch, and frequency spectrum, which are necessary for emotion analysis.

[0273] A "machine learning model" is an algorithm or program within a system that learns from large amounts of data in advance to analyze and identify emotions.

[0274] "Visualized data" refers to data such as graphical information, icons, and charts created to display the analyzed emotion results in an easy-to-understand manner.

[0275] A "terminal" is a device (smartphone, tablet, PC, etc.) used by a user to access the system.

[0276] "Feedback" refers to information on the user's evaluation of the analysis results, and is data used to improve the system and retrain the machine learning model.

[0277] "Retraining" refers to the process of reanalyzing data and updating models to improve the accuracy of machine learning models using feedback data collected by the system.

[0278] MODE FOR CARRYING OUT THE INVENTION

[0279] The present invention realizes two-way emotion analysis between animals and humans by combining a system that analyzes and visualizes animal emotions with an emotion engine that recognizes user emotions. Specific embodiments of the system are described below.

[0280] Hardware and Software Use

[0281] Users use devices such as smartphones and tablets, which are equipped with cameras and microphones that can simultaneously capture data such as facial and body expressions and voice.

[0282] The devices include smartphones and tablets running Android (registered trademark) or iOS. A dedicated application is installed on the devices, which have the ability to capture and temporarily store videos, images, user facial expressions, and audio data.

[0283] The server is a computer system built on the cloud that stores data, preprocesses it, extracts features, analyzes emotions using machine learning models, generates visualized data, and manages feedback. It can utilize common cloud services, such as AWS (registered trademark) (Amazon Web Services) and GCP (Google (registered trademark) Cloud Platform).

[0284] Data Acquisition and Transmission

[0285] Users take videos or images of animals with their smartphone camera, and simultaneously record their own facial expressions and tone of voice. The captured data is temporarily stored in a dedicated directory on the device.

[0286] Data storage and upload

[0287] The device uploads the acquired data to a cloud server via the Internet, where the data format and size are checked and, if necessary, converted into an appropriate format before being sent.

[0288] Data Preprocessing

[0289] The server performs preprocessing on the received data before analyzing it. Specifically, it breaks down video data into still images for each frame, separates audio data and stores it separately. It also performs noise removal and standardization on image data to ensure uniform quality.

[0290] Feature extraction

[0291] The server extracts features of animals and users from the preprocessed data. For example, for image data, OpenCV is used to detect facial and body features, obtaining information such as eye shape, mouth open / closed state, and ear position. For audio data, a voice analysis library such as LibROSA is used to extract features such as pitch, frequency spectrum, and sound duration.

[0292] Emotion Analysis

[0293] The server then inputs the extracted features into a pre-trained machine learning model (using TensorFlow and Scikit-learn) to analyze the emotions of animals and users. The model is designed to identify basic emotions (such as joy, anger, anxiety, and excitement) of animals and users.

[0294] Generate visualization data

[0295] The server generates visualization data based on the analyzed emotion data. Specifically, it creates icons showing the animal and user's facial expressions, as well as charts that graphically represent the analysis results. The generated visualization data is sent to the device in JSON format or similar.

[0296] Displaying the results

[0297] The device displays the results on a user interface (UI) based on the visualization data received from the server. For example, detailed information is provided to the user along with icons and tags indicating the emotions of the animals and the user.

[0298] Gathering feedback and relearning

[0299] The user checks the displayed emotion analysis results and evaluates their accuracy. The evaluation results are then sent as feedback from the device to the server. The server then retrains the machine learning model based on the received feedback data to improve analysis accuracy.

[0300] Examples and prompts

[0301] For example, suppose a user takes a video of their dog with their smartphone and records their own facial expressions and voice. This data is uploaded to a server, where it undergoes preprocessing, feature extraction, and emotion analysis to analyze the animal's and user's emotions as "the dog is happy" and "the user is happy." The analysis results are visualized and sent to a device for the user to review.

[0302] Examples of prompts include natural language sentences such as "Analyze the dog's emotions and see if it is happy" and "Analyze the emotions in my tone of voice and tell me what you think."

[0303] As described above, the present invention is a system that realizes smooth communication between animals and humans by analyzing the emotions of animals and users in both directions and providing the results to the user.

[0304] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0305] Step 1: Get the data

[0306] The user uses the camera and microphone functions of their smartphone or tablet to capture video or images of the animal. At the same time, the user's facial expressions and tone of voice are also recorded. The specific inputs are the animal's video and audio data, and the user's video and audio data, which are temporarily stored in the device's internal storage. The output is these data files.

[0307] Step 2: Save and upload your data

[0308] The device saves the acquired video and audio data in a dedicated directory. When saving, a timestamp is added to the file name to make it unique. For example, "video_YYYYMMDD_HHMMSS.mp4". Next, the format and size of the saved data are checked, and the data is uploaded to a cloud server via the Internet. The input is the saved data file, and the output is the data transferred to the server.

[0309] Step 3: Preprocessing the data

[0310] The server preprocesses the received data before analyzing it. It breaks down the video data into frames and saves them as still images (e.g., "frame_001.jpg", "frame_002.jpg", etc.). It also separates the audio data from the video and saves it (e.g., "audio.wav"). It performs filtering to remove noise and standardize the data. The input is the uploaded raw data, and the output is preprocessed still image data and audio data.

[0311] Step 4: Feature extraction

[0312] The server extracts features of animals and users from the preprocessed data. For image data, a face detection algorithm (for example, the OpenCV library) is used to detect the faces and body parts of animals and users. Features extracted include eye shape, mouth open / closed state, and ear position. For audio data, an audio analysis library such as LibROSA is used to extract features such as pitch, frequency spectrum, and sound duration. The input is preprocessed still image data and audio data, and the output is the features of animals and users.

[0313] Step 5: Sentiment Analysis

[0314] The server inputs the features into a pre-trained machine learning model (using, for example, TensorFlow or Scikit-learn) to analyze the emotions of the animal and the user. This model identifies basic emotions (e.g., joy, anger, anxiety, excitement, etc.) of the animal and the user. As a result of the analysis, emotion tags are assigned to each animal and user. The input is the feature data, and the output is the analysis result with the emotion tag.

[0315] Step 6: Generate visualization data

[0316] The server generates visualized data based on the analyzed emotion data. Specifically, it creates facial expression icons for animals and the user, as well as charts and graphs that graphically represent the analysis results. The generated visualized data is saved in JSON format or similar and sent to the terminal via the Internet. The input is the analysis results with emotion tags, and the output is the visualized data.

[0317] Step 7: View the results

[0318] The terminal analyzes the visualization data received from the server and displays the results on a user interface (UI). Specifically, it displays icons and messages indicating the emotion tags of animals and users, icons of a dog wagging its tail, smiling faces, etc. The input is the visualization data, and the output is the analysis results displayed on the UI.

[0319] Step 8: Gather feedback

[0320] The user checks the displayed emotion analysis results and evaluates their accuracy. The evaluation results are input to the terminal as feedback and sent to the server. The input is the user's evaluation data, and the output is the feedback data sent to the server.

[0321] Step 9: Relearn with feedback

[0322] The server analyzes the received feedback data and uses it to retrain the sentiment analysis model, thereby improving the accuracy of the machine learning model. The retrained model is then used for the next data analysis. The input is the feedback data, and the output is the improved machine learning model after retraining.

[0323] (Application example 2)

[0324] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0325] While modern technology has advanced to analyze the emotions of animals and humans and facilitate communication between them, there are still no systems that provide specific suggestions that are useful in real life based on the analysis results. Furthermore, there are no adequate mechanisms for suggesting appropriate products based on pet emotions, collecting user feedback on those suggestions, and improving the accuracy of the model. This makes it difficult for users to understand their pets' emotions and provide optimal care.

[0326] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0327] In this invention, the server includes means for acquiring video or image data of an animal, means for extracting the animal's facial expressions, gestures, and cries from the data, means for analyzing the animal's emotions based on the extracted features, means for visualizing the analyzed emotional data in a format that is easy for humans to understand, means for transmitting the visualized data to a terminal, means for displaying the data on the terminal, means for suggesting related products based on the analyzed emotional data, and means for collecting user feedback on the product suggestion results and using the feedback to retrain the analytical model. This enables users to appropriately understand their pet's emotions and receive suggestions that are useful in their daily lives through specific product suggestions based on the animal's emotions and by collecting feedback.

[0328] "Video or image data" refers to visual data that records the movements and expressions of animals.

[0329] "Facial expressions, gestures, and sounds" are the facial movements, body movements, and sounds that an animal makes.

[0330] "Features" are attributes of data extracted to analyze animal emotions.

[0331] "Emotion data" is information about an animal's emotions analyzed based on its features.

[0332] "Visualization" refers to presenting analysis results in a format that is easy for humans to understand.

[0333] A "terminal" is a digital device such as a smartphone or tablet.

[0334] "Product suggestions" refers to recommending related products such as pet food and toys to users based on the analysis results.

[0335] "Feedback" means collecting user ratings and opinions.

[0336] "Retraining" is the process of improving the accuracy of an analytical model based on collected feedback.

[0337] "Server" means a central computer system for storing, analyzing, transmitting and receiving data.

[0338] "Data preprocessing" is the process of removing noise and standardizing data before analyzing it.

[0339] A "machine learning algorithm" is a computational method for extracting features from data and performing analysis.

[0340] "Face detection" is a technology that identifies the location of a face from image data.

[0341] "Audio feature extraction" is the process of extracting information such as pitch and frequency from audio data.

[0342] "Database linkage" refers to the exchange of information with a database that stores information on the products being proposed.

[0343] This invention is a system that analyzes the emotions of animals and suggests the most suitable related products based on the analysis. This system is realized using a smartphone, tablet, and server.

[0344] First, the user uses a smartphone or tablet to acquire video or image data of the animal. The user takes videos or images of their pet, simultaneously recording their own facial expressions and tone of voice. This data provides the foundation for analyzing the emotions of the animal and the user.

[0345] The device then temporarily stores the captured video or image, along with the user's facial expression and audio data. These data are then uploaded to a server via the Internet. The server checks the format and size of the data and converts it into an appropriate format if necessary.

[0346] The server preprocesses the received data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates the audio data from the video and saves it. This allows the server to process the image and audio data separately, removing noise and normalizing them.

[0347] The server then extracts features of the animal and the user from the preprocessed data. For image data, a face detection algorithm (e.g., Haar) is used to detect the faces and body parts of the animal and the user, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[0348] The server analyzes the animal and user's emotions using a pre-trained generative AI model based on the extracted features. As a result, the animal and user's emotions are tagged. Then, based on the analyzed emotion data, it suggests related products (e.g., specific pet food or toys) that match the animal's emotion.

[0349] The results of these suggestions are sent to the device in an appropriate format, such as JSON, and the device displays the results on its user interface. The user then checks the products suggested by the system and makes a purchasing decision based on them. The user then inputs feedback about the effectiveness of the suggested products and the pet's reaction, which is then sent to the server. The server then retrains the analysis model based on this feedback, improving the accuracy of the next data analysis.

[0350] For example, if a user takes a video of their dog and the app analyzes it, and determines that the dog is "happy," the app will suggest pet food or toys with a "joy" theme. The user can purchase the suggested products and provide feedback on their pet's subsequent reactions, thereby improving the accuracy of the entire system.

[0351] Example prompts to input to a generative AI model:

[0352] Write a Python program that, given a video path, analyzes a pet's emotions and suggests the best pet food and toys. The program uses OpenCV to process the video and analyzes emotions using a Keras emotion recognition model.

[0353] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0354] Step 1:

[0355] Users use a smartphone or tablet to capture video or image data of animals. The input is the captured video or image data, and the output is data temporarily stored on the device. The user takes a video of their pet and simultaneously records their own facial expressions and tone of voice.

[0356] Step 2:

[0357] The device temporarily stores the captured video or image, along with the user's facial expression and audio data. It then uploads this data to a server via the Internet. The input is the temporarily stored video, image, and audio data, and the output is the data uploaded to the server. The device also checks the format and size of the data and converts it to an appropriate format if necessary.

[0358] Step 3:

[0359] The server preprocesses the received data. In the case of video data, it breaks the video down into frames and saves them as still images. It also separates the audio data from the video and saves it. The input is the uploaded video data, and the output is the image data broken down into frames and the separated audio data. The server also performs noise removal and normalization on these data.

[0360] Step 4:

[0361] The server extracts features of animals and users from the preprocessed data. The input is preprocessed image and audio data, and the output is the extracted features. For image data, a face detection algorithm (e.g., Haar) is used to detect the faces and body parts of animals and users, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[0362] Step 5:

[0363] The server uses the extracted features to analyze the emotions of the animal and the user using a pre-trained generative AI model. The input is the extracted features, and the output is the analyzed emotion data. The generative AI model is designed to identify basic emotions (happiness, anger, anxiety, excitement, etc.). Depending on the emotion data, tagging is performed.

[0364] Step 6:

[0365] Based on the analysis results, the server suggests related products (e.g., specific pet food or toys) that match the animal's emotions. The input is the analyzed emotion data, and the output is data about the suggested products. The server also sends these suggestion data to the device in an appropriate format, such as JSON.

[0366] Step 7:

[0367] The terminal displays the results on a user interface (UI) based on the recommendation data received from the server. The input is data about the recommended products, and the output is visualized data that is displayed to the user. The user reviews the recommended products and makes a purchasing decision based on their content.

[0368] Step 8:

[0369] The user inputs the effects of the proposed product and the pet's reaction as feedback and sends it to the server. The input is feedback information, and the output is feedback data stored on the server.

[0370] Step 9:

[0371] The server retrains the sentiment analysis model based on the received feedback. The input is the feedback data, and the output is an updated sentiment analysis model. This improves the accuracy of the next data analysis.

[0372] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0373] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0374] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0375] [Second embodiment]

[0376] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0377] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0378] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0379] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0380] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0381] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0382] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0383] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0384] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0385] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0386] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0387] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0388] The system of the present invention collects video or image data of animals, analyzes the emotions of the animals from the data, and displays them in a format that is easy for humans to understand. Below, the processing of the system's program will be explained in natural language and in detail with concrete examples.

[0389] Retrieving data

[0390] Users take videos and photos of animals using a device (such as a smartphone or tablet), and this data is temporarily stored on the device.

[0391] Uploading data

[0392] The device uploads the captured video and image data to the server. At this time, the format and size of the data being uploaded are checked and it is converted into the appropriate format.

[0393] Data Preprocessing

[0394] The server performs preprocessing on the received data. In the case of video data, the video is broken down into frames, and the image and audio data are stored separately. Noise removal and data standardization are also performed at this stage.

[0395] Feature extraction

[0396] The server uses machine learning algorithms to extract animal features from the data. From the image data, it detects the animal's face and body parts and extracts features such as eye shape, mouth opening and closing position, and ear position. From the audio data, it extracts features such as pitch, frequency spectrum, and sound duration.

[0397] Emotion Analysis

[0398] The server inputs the features into a pre-trained machine learning model to analyze the animal's emotions. This model uses convolutional neural networks (CNNs) and recurrent neural networks (RNNs). As a result of the analysis, the animal's emotions (happiness, anger, anxiety, excitement, etc.) are estimated and tagged.

[0399] Generate visualization data

[0400] The server generates visualization data based on the analyzed emotion data. This visualization data includes icons and graphs that represent the animal's facial expressions. The generated visualization data is sent to the device in JSON format or similar.

[0401] Displaying the results

[0402] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, it displays tags that indicate the animal's emotions, as well as detailed information such as the shape of its eyes and whether its mouth is open or closed.

[0403] Gathering feedback

[0404] The user checks the displayed information and evaluates whether the animal's emotions have been correctly analyzed. Based on the evaluation, the user enters feedback, which is then sent to the server. The feedback is used to improve the accuracy of the system.

[0405] Specific examples

[0406] Dog Emotion Analysis

[0407] 1. The user takes videos of the dog with their smartphone and records the dog's behavior and sounds.

[0408] 2. The device uploads this video data to the server.

[0409] 3. The server extracts frames from the video, separates them from the audio data, and performs preprocessing.

[0410] 4. The server detects the dog's face and body from the image data and extracts features such as the shape of the eyes, whether the mouth is open or closed, and the position of the ears. It also extracts the pitch and frequency spectrum from the audio data.

[0411] 5. The server uses a machine learning model to analyze the dog's emotion as "joy."

[0412] 6. The server generates visualization data based on the analysis results and sends it to the terminal.

[0413] 7. The device will display to the user "Dog is happy" and also show icons such as a wagging tail and raised ears.

[0414] 8. The user checks this information and decides whether the dog is truly happy. If the result is appropriate, the user enters feedback and sends it to the server.

[0415] In this way, the system of the present invention analyzes the emotions of animals and provides them in a format that is easy for humans to understand, thereby realizing smooth communication between animals and humans.

[0416] The processing flow will be explained below.

[0417] Step 1: Capture video or images of your animal

[0418] Users use devices such as smartphones and tablets to take videos or images of animals, and this data includes the animals' facial expressions, movements, sounds, etc.

[0419] Step 2: Save and upload your data

[0420] The device temporarily stores the captured video or image data, then uploads the data to a server via the Internet. At this time, the data format and size are checked and, if necessary, converted to an appropriate format.

[0421] Step 3: Preprocessing the data

[0422] The server receives the uploaded data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates the audio data from the video and saves it. Next, the server performs noise reduction and standardization on the image and audio data.

[0423] Step 4: Feature extraction

[0424] The server extracts animal features from the preprocessed data. For image data, a face detection algorithm is used to detect the animal's face and body parts, and features such as eye shape, mouth opening and closing, and ear position are extracted. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[0425] Step 5: Sentiment Analysis

[0426] The server inputs the extracted features into a machine learning model to analyze the animal's emotions. This model has been pre-trained with a large amount of data and is designed to identify basic animal emotions (happiness, anger, anxiety, excitement, etc.). As a result of the analysis, the animal's emotion is tagged.

[0427] Step 6: Generate visualization data

[0428] The server generates visualization data based on the analyzed emotion data. This visualization data includes animal facial expression icons and charts that graphically represent the analysis results. The generated visualization data is sent to the device in an appropriate format, such as JSON.

[0429] Step 7: View the results

[0430] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, detailed information such as the shape of the eyes and the opening and closing state of the mouth is displayed along with icons and tags that indicate the animal's emotions.

[0431] Step 8: Gather feedback

[0432] The user checks the displayed emotion analysis results and evaluates whether the animal's emotions have been correctly analyzed. This evaluation is performed by comparing the emotion analysis results with the actual behavior and situation of the animal. The user inputs this evaluation result as feedback and sends it to the server.

[0433] Step 9: Relearn with feedback

[0434] The server analyzes the received feedback data and uses it to retrain the sentiment analysis model. This improves the system's analysis accuracy and also improves the accuracy of future sentiment analysis. The retrained model is then used for the next data analysis.

[0435] The above steps will realize a system that accurately analyzes animal emotions and provides them to users, promoting smooth communication between animals and humans.

[0436] Example 1

[0437] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0438] Conventional animal emotion analysis systems have had difficulty accurately analyzing animal emotions from video data. Furthermore, the means for displaying the analyzed emotion data in an easy-to-understand manner for users have been insufficient, making it difficult for users to intuitively understand the emotions of animals. To address these issues, there is a need for the development of a system that can analyze animal emotions with high accuracy and provide the data in a visually easy-to-understand format.

[0439] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0440] In this invention, the server includes means for decomposing video data into frames and storing the audio data and image data separately, means for removing noise from the data and standardizing it, and means for extracting animal facial expressions, gestures, and cries from the data. This makes it possible to analyze emotions from animal video data with high accuracy and provide the analysis results in a visually easy-to-understand format.

[0441] "Video or image data of animals" refers to video or photographic information of pets or wild animals, which can be used to record the movements and expressions of animals.

[0442] "Terminal" refers to an electronic device such as a smartphone, tablet, or personal computer, which is used to acquire, transmit, and receive data.

[0443] A "server" refers to a computer system that processes, stores, and analyzes data via a network, and receives and processes data from multiple terminals.

[0444] "Means for uploading data" refers to the process by which a user's device sends video or image data to a server via a network.

[0445] "Means of decomposing into frames" refers to the process of dividing video data into individual still images (frames), which allows for detailed image analysis.

[0446] "Denoising and standardisation measures" refers to the process of removing unnecessary information and outliers and converting data into a consistent format to improve data quality.

[0447] "Means for extracting animal facial expressions, gestures, and sounds" refers to the process of detecting specific features from image and audio data and extracting them as data for analysis.

[0448] "Features" refer to specific attributes and patterns extracted from image and audio data, which can be used to analyze an animal's emotions and state.

[0449] A "generative AI model" refers to a model trained using machine learning algorithms that can infer animal emotions from features.

[0450] The "means for generating visualization data" refers to the process of converting the analyzed emotion data into visual information such as icons and graphs in order to display it in a format that is easy for the user to understand.

[0451] "Means for transmitting visualized data to a terminal" refers to a process in which the server transmits the generated visualization data to a user's terminal via a network.

[0452] "Means for displaying data" refers to the process of displaying analysis results and visualized data on a user interface on a user's terminal.

[0453] The system of this invention analyzes animal emotions with high accuracy and provides the user with a visually easy-to-understand format. This system is mainly composed of three elements: the user's terminal, an internet connection, and a server.

[0454] Retrieving data

[0455] Users use devices such as smartphones and tablets to take videos and photos of animals. The device's camera app is used to do this, and the captured data is temporarily stored on the device. For example, a user may take a video of their pet dog and record its various behaviors and sounds.

[0456] Uploading data

[0457] The device uploads the captured video and image data to a server via the Internet. At this time, the data format (JPEG, MP4, etc.) and size are automatically checked, and if necessary, the data is converted to the appropriate format using a library such as FFmpeg.

[0458] Data Preprocessing

[0459] The server breaks down video data into frames and stores the audio and image data separately. Specifically, it uses the OpenCV library to break down the video into frames, and FFmpeg to extract and store the audio data. It also uses SciPy and Pandas to perform noise removal and normalization. For example, the server exports one video frame and the corresponding audio from a video of a dog.

[0460] Feature extraction

[0461] The server uses OpenCV to extract features such as the animal's face, eye shape, ear position, and mouth open / close state from image data. It also uses Librosa to extract pitch, frequency spectrum, and sound duration from audio data. For example, it can extract whether a dog's ears are erect from image data and the pitch of a bark from audio data.

[0462] Emotion Analysis

[0463] The server uses a pre-trained generative AI model (CNN or RNN built using TensorFlow or PyTorch) to analyze the animal's emotions based on the extracted features. This model estimates the animal's emotions from the features and tags them with emotions such as joy, anger, anxiety, and excitement. For example, it can determine that a dog is happy based on the position of its ears and the shape of its face.

[0464] Generate visualization data

[0465] The server generates visualization data using visualization libraries such as D3.js based on the analyzed emotion data. The generated data includes icons and graphs showing the animal's facial expressions, and is transferred in JSON format. For example, it generates an icon showing a dog's happy state and a graph showing changes in emotion.

[0466] Displaying the results

[0467] The device analyzes the JSON data received from the server and displays the results on the user interface. For example, it displays the message "The dog is happy" along with a tail-wagging icon and a smiling icon.

[0468] Gathering feedback

[0469] The user checks the displayed analysis results and evaluates whether they are appropriate. The evaluation results are sent to the server as feedback and used as learning data for the system. For example, if a user evaluates the results as "correct," that evaluation helps improve the accuracy of the system.

[0470] Specific examples

[0471] 1. The user takes videos of the dog with their smartphone and records the dog's behavior and sounds.

[0472] 2. The device uploads this video data to the server.

[0473] 3. The server extracts frames from the video, separates them from the audio data, and performs preprocessing.

[0474] 4. The server detects the dog's face and body from the image data and extracts features such as the shape of the eyes, whether the mouth is open or closed, and the position of the ears. It also extracts the pitch and frequency spectrum from the audio data.

[0475] 5. The server uses a generative AI model to analyze the dog's emotion as "joy."

[0476] 6. The server generates visualization data based on the analysis results and sends it to the terminal.

[0477] 7. The device will display to the user "Dog is happy" and also show icons of the dog wagging its tail and smiling.

[0478] 8. The user checks this information and decides whether the dog is truly happy or not, and sends the evaluation result to the server as feedback.

[0479] By utilizing generative AI models, it is possible to analyze the complex emotions of animals with high accuracy and present them to users in a format that is very easy to understand.

[0480] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0481] Processing Steps

[0482] Step 1: Get the data

[0483] Users use devices such as smartphones and tablets to take videos and photos of animals. This is done using the device's camera app, and the captured data is temporarily stored on the device. The user's operations record the behavior and sounds of the animal being photographed. The input for this process is the video and image data obtained through the camera app, and the output is data temporarily stored in the device's internal storage area.

[0484] Step 2: Upload your data

[0485] The device uploads the captured video and image data to a server via the Internet. At this time, the data format (JPEG, MP4, etc.) and size are automatically checked, and if there are any problems, it is converted to the appropriate format using a library such as FFmpeg. This process uses the video and image data stored on the device as input, and the converted and checked data is sent to the server as output.

[0486] Step 3: Preprocessing the data

[0487] The server breaks down the uploaded video data into frames, and stores the audio and image data separately. Here, the OpenCV library is used to break down the video into frames, and FFmpeg is used to extract and store the audio data. SciPy and Pandas are also used to remove noise from the data and standardize it. The input for this process is the video data uploaded to the server, and the output is frame-by-frame image data and separated audio data.

[0488] Step 4: Feature extraction

[0489] The server uses OpenCV to extract features such as the animal's face, eye shape, ear position, and mouth open / close state from the image data. It also uses Librosa to extract pitch, frequency spectrum, and sound duration from the audio data. The input to this process is the image data and audio data separated and saved in the previous step, and the output is each extracted feature. Specifically, it identifies the position of the dog's ears and eye shape from the image data, and the pitch of the bark from the audio data.

[0490] Step 5: Sentiment Analysis

[0491] The server uses a pre-trained generative AI model (CNN or RNN built using TensorFlow or PyTorch) to analyze the animal's emotions based on the extracted features. This model estimates the animal's emotions from the features and tags them with emotions such as joy, anger, anxiety, and excitement. The input to this process is the extracted features, and the output is estimated emotion data. Specifically, it determines the emotion as "joy" based on the position of the dog's ears and the shape of its face.

[0492] Step 6: Generate visualization data

[0493] The server generates visualized data based on the analyzed emotion data using a visualization library such as D3.js. The generated data includes icons and graphs showing the animal's facial expressions, and is transferred to the terminal in JSON format. The input to this process is the analyzed emotion data, and the output is the generated visualized data. Specifically, it generates an icon showing the dog's happy state and a graph showing changes in emotion.

[0494] Step 7: View the results

[0495] The terminal analyzes the JSON data received from the server and displays the results on the user interface. Specifically, it displays a message saying "The dog is happy" along with a tail-wagging icon, a smiling icon, etc. The input to this process is the JSON data received from the server, and the output is the visual information displayed on the user interface.

[0496] Step 8: Gather feedback

[0497] The user checks the displayed analysis results and evaluates whether they are appropriate. The evaluation results are sent as feedback to the server and used as learning data for the system. The inputs to this process are the user's evaluation and feedback, and the output is feedback data sent to the server. In concrete terms, the user evaluates the results as "correct," and the evaluation is sent to the server.

[0498] (Application example 1)

[0499] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0500] In conventional food delivery services, when delivery personnel visit homes containing animals, it is difficult for them to understand the emotions and state of the animals. As a result, the animals may feel anxious or stressed, making it difficult to provide a safe and comfortable delivery experience. The present invention aims to solve this problem by providing a system that allows delivery personnel to understand the emotional state of animals and take appropriate action.

[0501] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0502] In this invention, the server includes means for acquiring video or image data of the animal, means for extracting the animal's facial expressions, gestures, and cries from the data, means for analyzing the animal's emotions based on the extracted features, means for visualizing the analyzed emotional data in a format that is easy for humans to understand, means for transmitting the visualized data to a terminal, means for displaying the data on the terminal, and means for conveying the animal's emotions to a delivery person during delivery work, thereby enabling the delivery person to understand the animal's emotions and take appropriate action toward the animal.

[0503] "Animal video or image data" refers to digital data of videos or photographs of animals taken with a mobile phone or other device.

[0504] "Facial expressions" refer to changes in emotions shown by the position and shape of each part of an animal's face (e.g., eyes, mouth, ears, etc.).

[0505] A "gesture" refers to a specific action or movement that an animal makes with its body.

[0506] "Cry" refers to the sound that animals make, and is a way of expressing emotions and communicating.

[0507] "Features" refer to numerical values ​​or indicators extracted from image or audio data that are necessary for emotion analysis.

[0508] "Emotion analysis" is the process of using extracted features to estimate an animal's internal emotional state (joy, anger, anxiety, excitement, etc.).

[0509] "Visualization" refers to the process of displaying analyzed emotional data in visual formats such as graphs and icons.

[0510] "Terminal" refers to a digital device for receiving and displaying information, such as a smartphone or tablet.

[0511] "A means of communicating an animal's emotions to the delivery person during delivery work" is a system that notifies the delivery person of the animal's emotional state via a terminal during food delivery.

[0512] System Configuration

[0513] The system of the present invention collects video or image data of animals, analyzes the emotions of the animals from the data, and conveys the emotions to the delivery person. This system is composed of the following elements.

[0514] 1. Device

[0515] The devices used are mobile information terminals such as smartphones and tablets, which are used to acquire video and image data of animals and send them to a server.

[0516] 2. Server

[0517] The server receives video and image data sent from the device, preprocesses the data, extracts features, analyzes emotions, and visualizes the results. Specific technologies used include machine learning algorithms such as face detection and voice feature extraction, as well as convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

[0518] 3. User Interface

[0519] The device comes with a dedicated application that visually displays the analysis results to the delivery person, including icons and messages based on the animal's emotional state.

[0520] Processing Details

[0521] 1. Data Acquisition

[0522] The user (delivery person) takes videos and photos of their pet using their smartphone.

[0523] 2. Uploading data

[0524] The device uploads the captured video and image data to the server, where the data format and size are checked and converted into the appropriate format.

[0525] 3. Data Preprocessing

[0526] The server performs preprocessing on the received data, such as noise removal and data standardization. In the case of video data, it divides it into frames and separates the image data from the audio data.

[0527] 4. Feature Extraction

[0528] The server uses machine learning algorithms to extract features such as the shape of the animal's eyes, the degree to which its mouth is open or closed, and the position of its ears from the image data, as well as the pitch and frequency spectrum from the audio data.

[0529] 5. Emotion Analysis

[0530] The server uses pre-trained CNN or RNN models to analyze the animal's emotions from the extracted features, and the resulting emotion (joy, anger, anxiety, excitement, etc.) is tagged.

[0531] 6. Generate visualization data

[0532] The server generates visualization data based on the analyzed emotion data and sends it to the terminal in JSON format or similar.

[0533] 7. Displaying the results

[0534] The device uses the visualization data received from the server to display the emotional state of the animal on the user interface. For example, a tail-wagging icon is displayed along with the message "The dog is happy."

[0535] Specific examples

[0536] Delivery person visits your home scenario

[0537] When delivering food, the delivery person takes a photo with their smartphone of the dog waiting at home.

[0538] The terminal uploads the photo data to the server.

[0539] The server analyzes the shape of the dog's eyes and the position of its ears from the image data, and extracts sound characteristics from the audio data.

[0540] The server displays the dog's "anxiety" as the analysis result.

[0541] The device displays a warning message to the delivery person saying, "Dog is anxious."

[0542] Delivery personnel use this information to approach quietly and avoid causing stress to the dog.

[0543] Prompt Sentence Examples

[0544] "Analyze a photo of a dog, determine its emotion, and output it in JSON format."

[0545] "Analyze the dog's emotions from this audio data and display the results in a graph."

[0546] This system allows delivery personnel to properly understand the emotional state of animals and provide a safe and comfortable delivery experience.

[0547] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0548] Step 1: Get the data

[0549] Users take videos and image data of animals with their smartphones, and this data is temporarily stored on the device. The input is the taken video and image data, and the output is the temporarily stored data on the device.

[0550] Step 2: Upload your data

[0551] The device uploads the captured video and image data to the server. The input is data temporarily stored on the device, and the output is data sent to the server. At this time, the format and size of the data are checked and converted into the appropriate format.

[0552] Step 3: Preprocessing the data

[0553] The server performs preprocessing on the received data. The input is the data sent to the server, and the output is the preprocessed data. In the case of video data, it is broken down into frames and separated into image data and audio data. Noise removal and data standardization are also performed here.

[0554] Step 4: Feature extraction

[0555] The server uses machine learning algorithms to extract animal features from the data. The input is preprocessed data, and the output is the extracted features. Specifically, eye shape, mouth open / closed state, and ear position are extracted from image data, and pitch and frequency spectrum are extracted from audio data.

[0556] Step 5: Sentiment Analysis

[0557] The server inputs the features into a pre-trained generative AI model (CNN or RNN) and analyzes the animal's emotions. The input is the extracted features, and the output is the emotion analysis result. As a result of the analysis, the animal's emotion (joy, anger, anxiety, excitement, etc.) is estimated and tagged.

[0558] Step 6: Generate visualization data

[0559] The server generates visualization data based on the analyzed emotion data. The input is the emotion analysis results, and the output is visualization data in JSON format. The visualization data includes icons and graphs showing animal facial expressions.

[0560] Step 7: View the results

[0561] The terminal displays the results on the user interface based on the visualization data received from the server. The input is the visualization data sent from the server, and the output is visual information on the emotional state displayed on the terminal. For example, it might display "The dog is happy" and use an icon to show the dog wagging its tail.

[0562] Step 8: Gather feedback

[0563] The user checks the displayed information and evaluates whether the animal's emotion has been correctly analyzed. The input is the emotional state displayed on the device, and the output is the user's evaluation feedback. This feedback is sent to the server and used to improve the accuracy of the system.

[0564] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0565] This invention realizes two-way emotion analysis between animals and humans by combining a system that analyzes and visualizes animal emotions with an emotion engine that recognizes user emotions. Below, the processing of the system's program is explained in natural language and in detail with concrete examples.

[0566] Retrieving data

[0567] Users use a smartphone, tablet, or other device to take videos or images of animals, while simultaneously recording their own facial expressions and tone of voice. This data provides the basis for analyzing the emotions of both the animal and the user.

[0568] Data storage and upload

[0569] The device temporarily stores the captured video or image, along with the user's facial expressions and voice data. The data is then uploaded to a server via the Internet. At this time, the data format and size are checked and converted to the appropriate format.

[0570] Data Preprocessing

[0571] The server receives the uploaded data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates and saves audio data from the video. The user's facial expression and audio data are also preprocessed in the same way, with noise removal and standardization.

[0572] Feature extraction

[0573] The server extracts features of both the animal and the user from the preprocessed data. For image data, a face detection algorithm is used to detect the faces and body parts of the animal and the user, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[0574] Emotion Analysis

[0575] The server inputs the extracted features into a machine learning model to analyze the emotions of animals and users. This model is pre-trained with a large amount of data and is designed to identify basic emotions (happiness, anger, anxiety, excitement, etc.) of animals and users. As a result of the analysis, the emotions of animals and users are tagged.

[0576] Generate visualization data

[0577] The server generates visualization data based on the analyzed emotion data. This visualization data includes facial expression icons for animals and the user, as well as charts that graphically represent the analysis results. The generated visualization data is sent to the device in an appropriate format, such as JSON.

[0578] Displaying the results

[0579] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, detailed information such as the shape of the eyes and the degree to which the mouth is open or closed is displayed along with icons and tags that indicate the emotions of the animal and the user. This allows the results of two-way emotion analysis between the animal and the user to be confirmed.

[0580] Gathering feedback

[0581] The user checks the displayed emotion analysis results and evaluates whether the animal's and the user's emotions have been analyzed correctly. This evaluation is performed by comparing the emotion analysis results with the animal's actual behavior and situation, and the user's own emotions. The user inputs this evaluation result as feedback and sends it to the server.

[0582] Re-learning with feedback

[0583] The server analyzes the received feedback data and uses it to retrain the emotion analysis model for animals and users. This improves the system's analysis accuracy and also improves the accuracy of future emotion analysis. The retrained model is then used for the next data analysis.

[0584] Specific examples

[0585] Emotional analysis of dogs and users

[0586] 1. The user takes a video of the dog with their smartphone and records their own facial expressions and voice.

[0587] 2. The device uploads this video and audio data to the server.

[0588] 3. The server preprocesses the video and audio data and extracts features.

[0589] 4. The server analyzes the animal and user's emotions using a machine learning model and estimates that the dog is "happy" and the user is "pleased."

[0590] 5. The server generates visualization data based on the analysis results and sends it to the terminal.

[0591] 6. The device will display messages to the user such as "The dog is happy" and "You are happy," and also show icons such as a wagging tail and a smiling face.

[0592] 7. The user checks this information and evaluates whether the analysis results are correct, inputs the evaluation results as feedback, and sends them to the server.

[0593] In this way, the system of the present invention realizes smooth communication between animals and humans by analyzing the emotions of both animals and users in both directions and providing the results to the user.

[0594] The processing flow will be explained below.

[0595] Step 1: Get the data

[0596] Users use devices such as smartphones or tablets to take videos or images of animals, while simultaneously recording their own facial expressions and tone of voice, thereby collecting basic data for analyzing the emotions of both the animals and the user.

[0597] Step 2: Save and upload your data

[0598] The device temporarily stores the captured video or image data, along with the user's facial expressions and voice data. The data is then uploaded to a server via the Internet. At this time, the data format and size are checked and converted to the appropriate format.

[0599] Step 3: Preprocessing the data

[0600] The server receives the uploaded data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates and saves audio data from the video. The user's facial expression and audio data are also preprocessed in the same way, with noise removal and standardization.

[0601] Step 4: Feature extraction

[0602] The server extracts features of both the animal and the user from the preprocessed data. For image data, a face detection algorithm is used to detect the faces and body parts of the animal and the user, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[0603] Step 5: Sentiment Analysis

[0604] The server inputs the extracted features into a machine learning model to analyze the emotions of animals and users. This model is pre-trained with a large amount of data and is designed to identify basic emotions (happiness, anger, anxiety, excitement, etc.) of animals and users. As a result of the analysis, the emotions of animals and users are tagged.

[0605] Step 6: Generate visualization data

[0606] The server generates visualization data based on the analyzed emotion data. This visualization data includes facial expression icons for animals and the user, as well as charts that graphically represent the analysis results. The generated visualization data is sent to the device in an appropriate format, such as JSON.

[0607] Step 7: View the results

[0608] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, detailed information such as the shape of the eyes and the degree to which the mouth is open or closed is displayed along with icons and tags that indicate the emotions of the animal and the user. This allows the results of two-way emotion analysis between the animal and the user to be confirmed.

[0609] Step 8: Gather feedback

[0610] The user checks the displayed emotion analysis results and evaluates whether the animal's and the user's emotions have been analyzed correctly. This evaluation is performed by comparing the emotion analysis results with the animal's actual behavior and situation, and the user's own emotions. The user inputs this evaluation result as feedback and sends it to the server.

[0611] Step 9: Relearn with feedback

[0612] The server analyzes the received feedback data and uses it to retrain the emotion analysis model for animals and users. This improves the system's analysis accuracy and also improves the accuracy of future emotion analysis. The retrained model is then used for the next data analysis.

[0613] Through the above steps, the system of the present invention analyzes the emotions of animals and users in both directions and provides the results to the user, thereby realizing smooth communication between animals and humans.

[0614] Example 2

[0615] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0616] Conventional animal emotion analysis systems only analyze animal emotions, and are unable to analyze emotions in both directions with humans. Furthermore, there is no system that can simultaneously analyze the emotions of both humans and animals, visualize them, and promote two-way communication, making it difficult to smoothly communicate with animals.

[0617] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring video or image data of an animal, means for extracting the animal's facial expressions, gestures, and sounds from the data, means for simultaneously acquiring the user's facial expressions and voice data, means for transmitting user and animal data to the server, means for checking the format and size of the data and performing preprocessing, means for analyzing the emotions of the animal and the user based on the extracted features, means for visualizing the analyzed emotional data in a format that is easy for humans to understand, means for transmitting the visualized data to a terminal, means for displaying the data on the terminal, means for receiving feedback from the user and transmitting it to the server, and means for relearning a machine learning model using the feedback. This enables two-way emotion analysis between the animal and the user and smooth communication based on the analysis.

[0618] "Animal video or image data" refers to video or still image data that records the movements and expressions of animals captured by a user.

[0619] "Facial expressions, gestures, and sounds" refer to the facial expressions, body movements, and sounds that animals make, and are used to infer emotions.

[0620] "User facial expression and voice data" refers to data such as facial expressions, tone of voice, and timbre that are recorded when a user uses the system.

[0621] A "server" is a central computer system that receives data sent from terminals via the Internet, analyzes and stores it, and manages feedback.

[0622] "Preprocessing" refers to data processing operations such as noise removal and standardization that are performed on collected data before analysis.

[0623] "Features" are identifiable information extracted from data, such as eye shape, mouth open / closed state, ear position, voice pitch, and frequency spectrum, which are necessary for emotion analysis.

[0624] A "machine learning model" is an algorithm or program within a system that learns from large amounts of data in advance to analyze and identify emotions.

[0625] "Visualized data" refers to data such as graphical information, icons, and charts created to display the analyzed emotion results in an easy-to-understand manner.

[0626] A "terminal" is a device (smartphone, tablet, PC, etc.) used by a user to access the system.

[0627] "Feedback" refers to information on the user's evaluation of the analysis results, and is data used to improve the system and retrain the machine learning model.

[0628] "Retraining" refers to the process of reanalyzing data and updating models to improve the accuracy of machine learning models using feedback data collected by the system.

[0629] MODE FOR CARRYING OUT THE INVENTION

[0630] The present invention realizes two-way emotion analysis between animals and humans by combining a system that analyzes and visualizes animal emotions with an emotion engine that recognizes user emotions. Specific embodiments of the system are described below.

[0631] Hardware and Software Use

[0632] Users use devices such as smartphones and tablets, which are equipped with cameras and microphones that can simultaneously capture data such as facial and body expressions and voice.

[0633] The devices include smartphones and tablets running Android or iOS. A dedicated application is installed on the device, which has the ability to capture and temporarily store videos, images, user facial expressions, and audio data.

[0634] The server is a computer system built on the cloud that stores data, preprocesses it, extracts features, analyzes emotions using machine learning models, generates visualization data, and manages feedback. It can utilize common cloud services such as AWS (Amazon Web Services) and GCP (Google Cloud Platform).

[0635] Data Acquisition and Transmission

[0636] Users take videos or images of animals with their smartphone camera, and simultaneously record their own facial expressions and tone of voice. The captured data is temporarily stored in a dedicated directory on the device.

[0637] Data storage and upload

[0638] The device uploads the acquired data to a cloud server via the Internet, where the data format and size are checked and, if necessary, converted into an appropriate format before being sent.

[0639] Data Preprocessing

[0640] The server performs preprocessing on the received data before analyzing it. Specifically, it breaks down video data into still images for each frame, separates audio data and stores it separately. It also performs noise removal and standardization on image data to ensure uniform quality.

[0641] Feature extraction

[0642] The server extracts features of animals and users from the preprocessed data. For example, for image data, OpenCV is used to detect facial and body features, obtaining information such as eye shape, mouth open / closed state, and ear position. For audio data, a voice analysis library such as LibROSA is used to extract features such as pitch, frequency spectrum, and sound duration.

[0643] Emotion Analysis

[0644] The server then inputs the extracted features into a pre-trained machine learning model (using TensorFlow and Scikit-learn) to analyze the emotions of animals and users. The model is designed to identify basic emotions (such as joy, anger, anxiety, and excitement) of animals and users.

[0645] Generate visualization data

[0646] The server generates visualization data based on the analyzed emotion data. Specifically, it creates icons showing the animal and user's facial expressions, as well as charts that graphically represent the analysis results. The generated visualization data is sent to the device in JSON format or similar.

[0647] Displaying the results

[0648] The device displays the results on a user interface (UI) based on the visualization data received from the server. For example, detailed information is provided to the user along with icons and tags indicating the emotions of the animals and the user.

[0649] Gathering feedback and relearning

[0650] The user checks the displayed emotion analysis results and evaluates their accuracy. The evaluation results are then sent as feedback from the device to the server. The server then retrains the machine learning model based on the received feedback data to improve analysis accuracy.

[0651] Examples and prompts

[0652] For example, suppose a user takes a video of their dog with their smartphone and records their own facial expressions and voice. This data is uploaded to a server, where it undergoes preprocessing, feature extraction, and emotion analysis to analyze the animal's and user's emotions as "the dog is happy" and "the user is happy." The analysis results are visualized and sent to a device for the user to review.

[0653] Examples of prompts include natural language sentences such as "Analyze the dog's emotions and see if it is happy" and "Analyze the emotions in my tone of voice and tell me what you think."

[0654] As described above, the present invention is a system that realizes smooth communication between animals and humans by analyzing the emotions of animals and users in both directions and providing the results to the user.

[0655] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0656] Step 1: Get the data

[0657] The user uses the camera and microphone functions of their smartphone or tablet to capture video or images of the animal. At the same time, the user's facial expressions and tone of voice are also recorded. The specific inputs are the animal's video and audio data, and the user's video and audio data, which are temporarily stored in the device's internal storage. The output is these data files.

[0658] Step 2: Save and upload your data

[0659] The device saves the acquired video and audio data in a dedicated directory. When saving, a timestamp is added to the file name to make it unique. For example, "video_YYYYMMDD_HHMMSS.mp4". Next, the format and size of the saved data are checked, and the data is uploaded to a cloud server via the Internet. The input is the saved data file, and the output is the data transferred to the server.

[0660] Step 3: Preprocessing the data

[0661] The server preprocesses the received data before analyzing it. It breaks down the video data into frames and saves them as still images (e.g., "frame_001.jpg", "frame_002.jpg", etc.). It also separates the audio data from the video and saves it (e.g., "audio.wav"). It performs filtering to remove noise and standardize the data. The input is the uploaded raw data, and the output is preprocessed still image data and audio data.

[0662] Step 4: Feature extraction

[0663] The server extracts features of animals and users from the preprocessed data. For image data, a face detection algorithm (for example, the OpenCV library) is used to detect the faces and body parts of animals and users. Features extracted include eye shape, mouth open / closed state, and ear position. For audio data, an audio analysis library such as LibROSA is used to extract features such as pitch, frequency spectrum, and sound duration. The input is preprocessed still image data and audio data, and the output is the features of animals and users.

[0664] Step 5: Sentiment Analysis

[0665] The server inputs the features into a pre-trained machine learning model (using, for example, TensorFlow or Scikit-learn) to analyze the emotions of the animal and the user. This model identifies basic emotions (e.g., joy, anger, anxiety, excitement, etc.) of the animal and the user. As a result of the analysis, emotion tags are assigned to each animal and user. The input is the feature data, and the output is the analysis result with the emotion tag.

[0666] Step 6: Generate visualization data

[0667] The server generates visualized data based on the analyzed emotion data. Specifically, it creates facial expression icons for animals and the user, as well as charts and graphs that graphically represent the analysis results. The generated visualized data is saved in JSON format or similar and sent to the terminal via the Internet. The input is the analysis results with emotion tags, and the output is the visualized data.

[0668] Step 7: View the results

[0669] The terminal analyzes the visualization data received from the server and displays the results on a user interface (UI). Specifically, it displays icons and messages indicating the emotion tags of animals and users, icons of a dog wagging its tail, smiling faces, etc. The input is the visualization data, and the output is the analysis results displayed on the UI.

[0670] Step 8: Gather feedback

[0671] The user checks the displayed emotion analysis results and evaluates their accuracy. The evaluation results are input to the terminal as feedback and sent to the server. The input is the user's evaluation data, and the output is the feedback data sent to the server.

[0672] Step 9: Relearn with feedback

[0673] The server analyzes the received feedback data and uses it to retrain the sentiment analysis model, thereby improving the accuracy of the machine learning model. The retrained model is then used for the next data analysis. The input is the feedback data, and the output is the improved machine learning model after retraining.

[0674] (Application example 2)

[0675] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0676] While modern technology has advanced to analyze the emotions of animals and humans and facilitate communication between them, there are still no systems that provide specific suggestions that are useful in real life based on the analysis results. Furthermore, there are no adequate mechanisms for suggesting appropriate products based on pet emotions, collecting user feedback on those suggestions, and improving the accuracy of the model. This makes it difficult for users to understand their pets' emotions and provide optimal care.

[0677] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0678] In this invention, the server includes means for acquiring video or image data of an animal, means for extracting the animal's facial expressions, gestures, and cries from the data, means for analyzing the animal's emotions based on the extracted features, means for visualizing the analyzed emotional data in a format that is easy for humans to understand, means for transmitting the visualized data to a terminal, means for displaying the data on the terminal, means for suggesting related products based on the analyzed emotional data, and means for collecting user feedback on the product suggestion results and using the feedback to retrain the analytical model. This enables users to appropriately understand their pet's emotions and receive suggestions that are useful in their daily lives through specific product suggestions based on the animal's emotions and by collecting feedback.

[0679] "Video or image data" refers to visual data that records the movements and expressions of animals.

[0680] "Facial expressions, gestures, and sounds" are the facial movements, body movements, and sounds that an animal makes.

[0681] "Features" are attributes of data extracted to analyze animal emotions.

[0682] "Emotion data" is information about an animal's emotions analyzed based on its features.

[0683] "Visualization" refers to presenting analysis results in a format that is easy for humans to understand.

[0684] A "terminal" is a digital device such as a smartphone or tablet.

[0685] "Product suggestions" refers to recommending related products such as pet food and toys to users based on the analysis results.

[0686] "Feedback" means collecting user ratings and opinions.

[0687] "Retraining" is the process of improving the accuracy of an analytical model based on collected feedback.

[0688] "Server" means a central computer system for storing, analyzing, transmitting and receiving data.

[0689] "Data preprocessing" is the process of removing noise and standardizing data before analyzing it.

[0690] A "machine learning algorithm" is a computational method for extracting features from data and performing analysis.

[0691] "Face detection" is a technology that identifies the location of a face from image data.

[0692] "Audio feature extraction" is the process of extracting information such as pitch and frequency from audio data.

[0693] "Database linkage" refers to the exchange of information with a database that stores information on the products being proposed.

[0694] This invention is a system that analyzes the emotions of animals and suggests the most suitable related products based on the analysis. This system is realized using a smartphone, tablet, and server.

[0695] First, the user uses a smartphone or tablet to acquire video or image data of the animal. The user takes videos or images of their pet, simultaneously recording their own facial expressions and tone of voice. This data provides the foundation for analyzing the emotions of the animal and the user.

[0696] The device then temporarily stores the captured video or image, along with the user's facial expression and audio data. These data are then uploaded to a server via the Internet. The server checks the format and size of the data and converts it into an appropriate format if necessary.

[0697] The server preprocesses the received data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates the audio data from the video and saves it. This allows the server to process the image and audio data separately, removing noise and normalizing them.

[0698] The server then extracts features of the animal and the user from the preprocessed data. For image data, a face detection algorithm (e.g., Haar) is used to detect the faces and body parts of the animal and the user, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[0699] The server analyzes the animal and user's emotions using a pre-trained generative AI model based on the extracted features. As a result, the animal and user's emotions are tagged. Then, based on the analyzed emotion data, it suggests related products (e.g., specific pet food or toys) that match the animal's emotion.

[0700] The results of these suggestions are sent to the device in an appropriate format, such as JSON, and the device displays the results on its user interface. The user then checks the products suggested by the system and makes a purchasing decision based on them. The user then inputs feedback about the effectiveness of the suggested products and the pet's reaction, which is then sent to the server. The server then retrains the analysis model based on this feedback, improving the accuracy of the next data analysis.

[0701] For example, if a user takes a video of their dog and the app analyzes it, and determines that the dog is "happy," the app will suggest pet food or toys with a "joy" theme. The user can purchase the suggested products and provide feedback on their pet's subsequent reactions, thereby improving the accuracy of the entire system.

[0702] Example prompts to input to a generative AI model:

[0703] Write a Python program that, given a video path, analyzes a pet's emotions and suggests the best pet food and toys. The program uses OpenCV to process the video and analyzes emotions using a Keras emotion recognition model.

[0704] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0705] Step 1:

[0706] Users use a smartphone or tablet to capture video or image data of animals. The input is the captured video or image data, and the output is data temporarily stored on the device. The user takes a video of their pet and simultaneously records their own facial expressions and tone of voice.

[0707] Step 2:

[0708] The device temporarily stores the captured video or image, along with the user's facial expression and audio data. It then uploads this data to a server via the Internet. The input is the temporarily stored video, image, and audio data, and the output is the data uploaded to the server. The device also checks the format and size of the data and converts it to an appropriate format if necessary.

[0709] Step 3:

[0710] The server preprocesses the received data. In the case of video data, it breaks the video down into frames and saves them as still images. It also separates the audio data from the video and saves it. The input is the uploaded video data, and the output is the image data broken down into frames and the separated audio data. The server also performs noise removal and normalization on these data.

[0711] Step 4:

[0712] The server extracts features of animals and users from the preprocessed data. The input is preprocessed image and audio data, and the output is the extracted features. For image data, a face detection algorithm (e.g., Haar) is used to detect the faces and body parts of animals and users, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[0713] Step 5:

[0714] The server uses the extracted features to analyze the emotions of the animal and the user using a pre-trained generative AI model. The input is the extracted features, and the output is the analyzed emotion data. The generative AI model is designed to identify basic emotions (happiness, anger, anxiety, excitement, etc.). Depending on the emotion data, tagging is performed.

[0715] Step 6:

[0716] Based on the analysis results, the server suggests related products (e.g., specific pet food or toys) that match the animal's emotions. The input is the analyzed emotion data, and the output is data about the suggested products. The server also sends these suggestion data to the device in an appropriate format, such as JSON.

[0717] Step 7:

[0718] The terminal displays the results on a user interface (UI) based on the recommendation data received from the server. The input is data about the recommended products, and the output is visualized data that is displayed to the user. The user reviews the recommended products and makes a purchasing decision based on their content.

[0719] Step 8:

[0720] The user inputs the effects of the proposed product and the pet's reaction as feedback and sends it to the server. The input is feedback information, and the output is feedback data stored on the server.

[0721] Step 9:

[0722] The server retrains the sentiment analysis model based on the received feedback. The input is the feedback data, and the output is an updated sentiment analysis model. This improves the accuracy of the next data analysis.

[0723] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0724] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0725] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0726] [Third embodiment]

[0727] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0728] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0729] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0730] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0731] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0732] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0733] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0734] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0735] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0736] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0737] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0738] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0739] The system of the present invention collects video or image data of animals, analyzes the emotions of the animals from the data, and displays them in a format that is easy for humans to understand. Below, the processing of the system's program will be explained in natural language and in detail with concrete examples.

[0740] Retrieving data

[0741] Users take videos and photos of animals using a device (such as a smartphone or tablet), and this data is temporarily stored on the device.

[0742] Uploading data

[0743] The device uploads the captured video and image data to the server. At this time, the format and size of the data being uploaded are checked and it is converted into the appropriate format.

[0744] Data Preprocessing

[0745] The server performs preprocessing on the received data. In the case of video data, the video is broken down into frames, and the image and audio data are stored separately. Noise removal and data standardization are also performed at this stage.

[0746] Feature extraction

[0747] The server uses machine learning algorithms to extract animal features from the data. From the image data, it detects the animal's face and body parts and extracts features such as eye shape, mouth opening and closing position, and ear position. From the audio data, it extracts features such as pitch, frequency spectrum, and sound duration.

[0748] Emotion Analysis

[0749] The server inputs the features into a pre-trained machine learning model to analyze the animal's emotions. This model uses convolutional neural networks (CNNs) and recurrent neural networks (RNNs). As a result of the analysis, the animal's emotions (happiness, anger, anxiety, excitement, etc.) are estimated and tagged.

[0750] Generate visualization data

[0751] The server generates visualization data based on the analyzed emotion data. This visualization data includes icons and graphs that represent the animal's facial expressions. The generated visualization data is sent to the device in JSON format or similar.

[0752] Displaying the results

[0753] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, it displays tags that indicate the animal's emotions, as well as detailed information such as the shape of its eyes and whether its mouth is open or closed.

[0754] Gathering feedback

[0755] The user checks the displayed information and evaluates whether the animal's emotions have been correctly analyzed. Based on the evaluation, the user enters feedback, which is then sent to the server. The feedback is used to improve the accuracy of the system.

[0756] Specific examples

[0757] Dog Emotion Analysis

[0758] 1. The user takes videos of the dog with their smartphone and records the dog's behavior and sounds.

[0759] 2. The device uploads this video data to the server.

[0760] 3. The server extracts frames from the video, separates them from the audio data, and performs preprocessing.

[0761] 4. The server detects the dog's face and body from the image data and extracts features such as the shape of the eyes, whether the mouth is open or closed, and the position of the ears. It also extracts the pitch and frequency spectrum from the audio data.

[0762] 5. The server uses a machine learning model to analyze the dog's emotion as "joy."

[0763] 6. The server generates visualization data based on the analysis results and sends it to the terminal.

[0764] 7. The device will display to the user "Dog is happy" and also show icons such as a wagging tail and raised ears.

[0765] 8. The user checks this information and decides whether the dog is truly happy. If the result is appropriate, the user enters feedback and sends it to the server.

[0766] In this way, the system of the present invention analyzes the emotions of animals and provides them in a format that is easy for humans to understand, thereby realizing smooth communication between animals and humans.

[0767] The processing flow will be explained below.

[0768] Step 1: Capture video or images of your animal

[0769] Users use devices such as smartphones and tablets to take videos or images of animals, and this data includes the animals' facial expressions, movements, sounds, etc.

[0770] Step 2: Save and upload your data

[0771] The device temporarily stores the captured video or image data, then uploads the data to a server via the Internet. At this time, the data format and size are checked and, if necessary, converted to an appropriate format.

[0772] Step 3: Preprocessing the data

[0773] The server receives the uploaded data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates the audio data from the video and saves it. Next, the server performs noise reduction and standardization on the image and audio data.

[0774] Step 4: Feature extraction

[0775] The server extracts animal features from the preprocessed data. For image data, a face detection algorithm is used to detect the animal's face and body parts, and features such as eye shape, mouth opening and closing, and ear position are extracted. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[0776] Step 5: Sentiment Analysis

[0777] The server inputs the extracted features into a machine learning model to analyze the animal's emotions. This model has been pre-trained with a large amount of data and is designed to identify basic animal emotions (happiness, anger, anxiety, excitement, etc.). As a result of the analysis, the animal's emotion is tagged.

[0778] Step 6: Generate visualization data

[0779] The server generates visualization data based on the analyzed emotion data. This visualization data includes animal facial expression icons and charts that graphically represent the analysis results. The generated visualization data is sent to the device in an appropriate format, such as JSON.

[0780] Step 7: View the results

[0781] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, detailed information such as the shape of the eyes and the opening and closing state of the mouth is displayed along with icons and tags that indicate the animal's emotions.

[0782] Step 8: Gather feedback

[0783] The user checks the displayed emotion analysis results and evaluates whether the animal's emotions have been correctly analyzed. This evaluation is performed by comparing the emotion analysis results with the actual behavior and situation of the animal. The user inputs this evaluation result as feedback and sends it to the server.

[0784] Step 9: Relearn with feedback

[0785] The server analyzes the received feedback data and uses it to retrain the sentiment analysis model. This improves the system's analysis accuracy and also improves the accuracy of future sentiment analysis. The retrained model is then used for the next data analysis.

[0786] The above steps will realize a system that accurately analyzes animal emotions and provides them to users, promoting smooth communication between animals and humans.

[0787] Example 1

[0788] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0789] Conventional animal emotion analysis systems have had difficulty accurately analyzing animal emotions from video data. Furthermore, the means for displaying the analyzed emotion data in an easy-to-understand manner for users have been insufficient, making it difficult for users to intuitively understand the emotions of animals. To address these issues, there is a need for the development of a system that can analyze animal emotions with high accuracy and provide the data in a visually easy-to-understand format.

[0790] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0791] In this invention, the server includes means for decomposing video data into frames and storing the audio data and image data separately, means for removing noise from the data and standardizing it, and means for extracting animal facial expressions, gestures, and cries from the data. This makes it possible to analyze emotions from animal video data with high accuracy and provide the analysis results in a visually easy-to-understand format.

[0792] "Video or image data of animals" refers to video or photographic information of pets or wild animals, which can be used to record the movements and expressions of animals.

[0793] "Terminal" refers to an electronic device such as a smartphone, tablet, or personal computer, which is used to acquire, transmit, and receive data.

[0794] A "server" refers to a computer system that processes, stores, and analyzes data via a network, and receives and processes data from multiple terminals.

[0795] "Means for uploading data" refers to the process by which a user's device sends video or image data to a server via a network.

[0796] "Means of decomposing into frames" refers to the process of dividing video data into individual still images (frames), which allows for detailed image analysis.

[0797] "Denoising and standardisation measures" refers to the process of removing unnecessary information and outliers and converting data into a consistent format to improve data quality.

[0798] "Means for extracting animal facial expressions, gestures, and sounds" refers to the process of detecting specific features from image and audio data and extracting them as data for analysis.

[0799] "Features" refer to specific attributes and patterns extracted from image and audio data, which can be used to analyze an animal's emotions and state.

[0800] A "generative AI model" refers to a model trained using machine learning algorithms that can infer animal emotions from features.

[0801] The "means for generating visualization data" refers to the process of converting the analyzed emotion data into visual information such as icons and graphs in order to display it in a format that is easy for the user to understand.

[0802] "Means for transmitting visualized data to a terminal" refers to a process in which the server transmits the generated visualization data to a user's terminal via a network.

[0803] "Means for displaying data" refers to the process of displaying analysis results and visualized data on a user interface on a user's terminal.

[0804] The system of this invention analyzes animal emotions with high accuracy and provides the user with a visually easy-to-understand format. This system is mainly composed of three elements: the user's terminal, an internet connection, and a server.

[0805] Retrieving data

[0806] Users use devices such as smartphones and tablets to take videos and photos of animals. The device's camera app is used to do this, and the captured data is temporarily stored on the device. For example, a user may take a video of their pet dog and record its various behaviors and sounds.

[0807] Uploading data

[0808] The device uploads the captured video and image data to a server via the Internet. At this time, the data format (JPEG, MP4, etc.) and size are automatically checked, and if necessary, the data is converted to the appropriate format using a library such as FFmpeg.

[0809] Data Preprocessing

[0810] The server breaks down video data into frames and stores the audio and image data separately. Specifically, it uses the OpenCV library to break down the video into frames, and FFmpeg to extract and store the audio data. It also uses SciPy and Pandas to perform noise removal and normalization. For example, the server exports one video frame and the corresponding audio from a video of a dog.

[0811] Feature extraction

[0812] The server uses OpenCV to extract features such as the animal's face, eye shape, ear position, and mouth open / close state from image data. It also uses Librosa to extract pitch, frequency spectrum, and sound duration from audio data. For example, it can extract whether a dog's ears are erect from image data and the pitch of a bark from audio data.

[0813] Emotion Analysis

[0814] The server uses a pre-trained generative AI model (CNN or RNN built using TensorFlow or PyTorch) to analyze the animal's emotions based on the extracted features. This model estimates the animal's emotions from the features and tags them with emotions such as joy, anger, anxiety, and excitement. For example, it can determine that a dog is happy based on the position of its ears and the shape of its face.

[0815] Generate visualization data

[0816] The server generates visualization data using visualization libraries such as D3.js based on the analyzed emotion data. The generated data includes icons and graphs showing the animal's facial expressions, and is transferred in JSON format. For example, it generates an icon showing a dog's happy state and a graph showing changes in emotion.

[0817] Displaying the results

[0818] The device analyzes the JSON data received from the server and displays the results on the user interface. For example, it displays the message "The dog is happy" along with a tail-wagging icon and a smiling icon.

[0819] Gathering feedback

[0820] The user checks the displayed analysis results and evaluates whether they are appropriate. The evaluation results are sent to the server as feedback and used as learning data for the system. For example, if a user evaluates the results as "correct," that evaluation helps improve the accuracy of the system.

[0821] Specific examples

[0822] 1. The user takes videos of the dog with their smartphone and records the dog's behavior and sounds.

[0823] 2. The device uploads this video data to the server.

[0824] 3. The server extracts frames from the video, separates them from the audio data, and performs preprocessing.

[0825] 4. The server detects the dog's face and body from the image data and extracts features such as the shape of the eyes, whether the mouth is open or closed, and the position of the ears. It also extracts the pitch and frequency spectrum from the audio data.

[0826] 5. The server uses a generative AI model to analyze the dog's emotion as "joy."

[0827] 6. The server generates visualization data based on the analysis results and sends it to the terminal.

[0828] 7. The device will display to the user "Dog is happy" and also show icons of the dog wagging its tail and smiling.

[0829] 8. The user checks this information and decides whether the dog is truly happy or not, and sends the evaluation result to the server as feedback.

[0830] By utilizing generative AI models, it is possible to analyze the complex emotions of animals with high accuracy and present them to users in a format that is very easy to understand.

[0831] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0832] Processing Steps

[0833] Step 1: Get the data

[0834] Users use devices such as smartphones and tablets to take videos and photos of animals. This is done using the device's camera app, and the captured data is temporarily stored on the device. The user's operations record the behavior and sounds of the animal being photographed. The input for this process is the video and image data obtained through the camera app, and the output is data temporarily stored in the device's internal storage area.

[0835] Step 2: Upload your data

[0836] The device uploads the captured video and image data to a server via the Internet. At this time, the data format (JPEG, MP4, etc.) and size are automatically checked, and if there are any problems, it is converted to the appropriate format using a library such as FFmpeg. This process uses the video and image data stored on the device as input, and the converted and checked data is sent to the server as output.

[0837] Step 3: Preprocessing the data

[0838] The server breaks down the uploaded video data into frames, and stores the audio and image data separately. Here, the OpenCV library is used to break down the video into frames, and FFmpeg is used to extract and store the audio data. SciPy and Pandas are also used to remove noise from the data and standardize it. The input for this process is the video data uploaded to the server, and the output is frame-by-frame image data and separated audio data.

[0839] Step 4: Feature extraction

[0840] The server uses OpenCV to extract features such as the animal's face, eye shape, ear position, and mouth open / close state from the image data. It also uses Librosa to extract pitch, frequency spectrum, and sound duration from the audio data. The input to this process is the image data and audio data separated and saved in the previous step, and the output is each extracted feature. Specifically, it identifies the position of the dog's ears and eye shape from the image data, and the pitch of the bark from the audio data.

[0841] Step 5: Sentiment Analysis

[0842] The server uses a pre-trained generative AI model (CNN or RNN built using TensorFlow or PyTorch) to analyze the animal's emotions based on the extracted features. This model estimates the animal's emotions from the features and tags them with emotions such as joy, anger, anxiety, and excitement. The input to this process is the extracted features, and the output is estimated emotion data. Specifically, it determines the emotion as "joy" based on the position of the dog's ears and the shape of its face.

[0843] Step 6: Generate visualization data

[0844] The server generates visualized data based on the analyzed emotion data using a visualization library such as D3.js. The generated data includes icons and graphs showing the animal's facial expressions, and is transferred to the terminal in JSON format. The input to this process is the analyzed emotion data, and the output is the generated visualized data. Specifically, it generates an icon showing the dog's happy state and a graph showing changes in emotion.

[0845] Step 7: View the results

[0846] The terminal analyzes the JSON data received from the server and displays the results on the user interface. Specifically, it displays a message saying "The dog is happy" along with a tail-wagging icon, a smiling icon, etc. The input to this process is the JSON data received from the server, and the output is the visual information displayed on the user interface.

[0847] Step 8: Gather feedback

[0848] The user checks the displayed analysis results and evaluates whether they are appropriate. The evaluation results are sent as feedback to the server and used as learning data for the system. The inputs to this process are the user's evaluation and feedback, and the output is feedback data sent to the server. In concrete terms, the user evaluates the results as "correct," and the evaluation is sent to the server.

[0849] (Application example 1)

[0850] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0851] In conventional food delivery services, when delivery personnel visit homes containing animals, it is difficult for them to understand the emotions and state of the animals. As a result, the animals may feel anxious or stressed, making it difficult to provide a safe and comfortable delivery experience. The present invention aims to solve this problem by providing a system that allows delivery personnel to understand the emotional state of animals and take appropriate action.

[0852] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0853] In this invention, the server includes means for acquiring video or image data of the animal, means for extracting the animal's facial expressions, gestures, and cries from the data, means for analyzing the animal's emotions based on the extracted features, means for visualizing the analyzed emotional data in a format that is easy for humans to understand, means for transmitting the visualized data to a terminal, means for displaying the data on the terminal, and means for conveying the animal's emotions to a delivery person during delivery work, thereby enabling the delivery person to understand the animal's emotions and take appropriate action toward the animal.

[0854] "Animal video or image data" refers to digital data of videos or photographs of animals taken with a mobile phone or other device.

[0855] "Facial expressions" refer to changes in emotions shown by the position and shape of each part of an animal's face (e.g., eyes, mouth, ears, etc.).

[0856] A "gesture" refers to a specific action or movement that an animal makes with its body.

[0857] "Cry" refers to the sound that animals make, and is a way of expressing emotions and communicating.

[0858] "Features" refer to numerical values ​​or indicators extracted from image or audio data that are necessary for emotion analysis.

[0859] "Emotion analysis" is the process of using extracted features to estimate an animal's internal emotional state (joy, anger, anxiety, excitement, etc.).

[0860] "Visualization" refers to the process of displaying analyzed emotional data in visual formats such as graphs and icons.

[0861] "Terminal" refers to a digital device for receiving and displaying information, such as a smartphone or tablet.

[0862] "A means of communicating an animal's emotions to the delivery person during delivery work" is a system that notifies the delivery person of the animal's emotional state via a terminal during food delivery.

[0863] System Configuration

[0864] The system of the present invention collects video or image data of animals, analyzes the emotions of the animals from the data, and conveys the emotions to the delivery person. This system is composed of the following elements.

[0865] 1. Device

[0866] The devices used are mobile information terminals such as smartphones and tablets, which are used to acquire video and image data of animals and send them to a server.

[0867] 2. Server

[0868] The server receives video and image data sent from the device, preprocesses the data, extracts features, analyzes emotions, and visualizes the results. Specific technologies used include machine learning algorithms such as face detection and voice feature extraction, as well as convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

[0869] 3. User Interface

[0870] The device comes with a dedicated application that visually displays the analysis results to the delivery person, including icons and messages based on the animal's emotional state.

[0871] Processing Details

[0872] 1. Data Acquisition

[0873] The user (delivery person) takes videos and photos of their pet using their smartphone.

[0874] 2. Uploading data

[0875] The device uploads the captured video and image data to the server, where the data format and size are checked and converted into the appropriate format.

[0876] 3. Data Preprocessing

[0877] The server performs preprocessing on the received data, such as noise removal and data standardization. In the case of video data, it divides it into frames and separates the image data from the audio data.

[0878] 4. Feature Extraction

[0879] The server uses machine learning algorithms to extract features such as the shape of the animal's eyes, the degree to which its mouth is open or closed, and the position of its ears from the image data, as well as the pitch and frequency spectrum from the audio data.

[0880] 5. Emotion Analysis

[0881] The server uses pre-trained CNN or RNN models to analyze the animal's emotions from the extracted features, and the resulting emotion (joy, anger, anxiety, excitement, etc.) is tagged.

[0882] 6. Generate visualization data

[0883] The server generates visualization data based on the analyzed emotion data and sends it to the terminal in JSON format or similar.

[0884] 7. Displaying the results

[0885] The device uses the visualization data received from the server to display the emotional state of the animal on the user interface. For example, a tail-wagging icon is displayed along with the message "The dog is happy."

[0886] Specific examples

[0887] Delivery person visits your home scenario

[0888] When delivering food, the delivery person takes a photo with their smartphone of the dog waiting at home.

[0889] The terminal uploads the photo data to the server.

[0890] The server analyzes the shape of the dog's eyes and the position of its ears from the image data, and extracts sound characteristics from the audio data.

[0891] The server displays the dog's "anxiety" as the analysis result.

[0892] The device displays a warning message to the delivery person saying, "Dog is anxious."

[0893] Delivery personnel use this information to approach quietly and avoid causing stress to the dog.

[0894] Prompt Sentence Examples

[0895] "Analyze a photo of a dog, determine its emotion, and output it in JSON format."

[0896] "Analyze the dog's emotions from this audio data and display the results in a graph."

[0897] This system allows delivery personnel to properly understand the emotional state of animals and provide a safe and comfortable delivery experience.

[0898] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0899] Step 1: Get the data

[0900] Users take videos and image data of animals with their smartphones, and this data is temporarily stored on the device. The input is the taken video and image data, and the output is the temporarily stored data on the device.

[0901] Step 2: Upload your data

[0902] The device uploads the captured video and image data to the server. The input is data temporarily stored on the device, and the output is data sent to the server. At this time, the format and size of the data are checked and converted into the appropriate format.

[0903] Step 3: Preprocessing the data

[0904] The server performs preprocessing on the received data. The input is the data sent to the server, and the output is the preprocessed data. In the case of video data, it is broken down into frames and separated into image data and audio data. Noise removal and data standardization are also performed here.

[0905] Step 4: Feature extraction

[0906] The server uses machine learning algorithms to extract animal features from the data. The input is preprocessed data, and the output is the extracted features. Specifically, eye shape, mouth open / closed state, and ear position are extracted from image data, and pitch and frequency spectrum are extracted from audio data.

[0907] Step 5: Sentiment Analysis

[0908] The server inputs the features into a pre-trained generative AI model (CNN or RNN) and analyzes the animal's emotions. The input is the extracted features, and the output is the emotion analysis result. As a result of the analysis, the animal's emotion (joy, anger, anxiety, excitement, etc.) is estimated and tagged.

[0909] Step 6: Generate visualization data

[0910] The server generates visualization data based on the analyzed emotion data. The input is the emotion analysis results, and the output is visualization data in JSON format. The visualization data includes icons and graphs showing animal facial expressions.

[0911] Step 7: View the results

[0912] The terminal displays the results on the user interface based on the visualization data received from the server. The input is the visualization data sent from the server, and the output is visual information on the emotional state displayed on the terminal. For example, it might display "The dog is happy" and use an icon to show the dog wagging its tail.

[0913] Step 8: Gather feedback

[0914] The user checks the displayed information and evaluates whether the animal's emotion has been correctly analyzed. The input is the emotional state displayed on the device, and the output is the user's evaluation feedback. This feedback is sent to the server and used to improve the accuracy of the system.

[0915] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0916] This invention realizes two-way emotion analysis between animals and humans by combining a system that analyzes and visualizes animal emotions with an emotion engine that recognizes user emotions. Below, the processing of the system's program is explained in natural language and in detail with concrete examples.

[0917] Retrieving data

[0918] Users use a smartphone, tablet, or other device to take videos or images of animals, while simultaneously recording their own facial expressions and tone of voice. This data provides the basis for analyzing the emotions of both the animal and the user.

[0919] Data storage and upload

[0920] The device temporarily stores the captured video or image, along with the user's facial expressions and voice data. The data is then uploaded to a server via the Internet. At this time, the data format and size are checked and converted to the appropriate format.

[0921] Data Preprocessing

[0922] The server receives the uploaded data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates and saves audio data from the video. The user's facial expression and audio data are also preprocessed in the same way, with noise removal and standardization.

[0923] Feature extraction

[0924] The server extracts features of both the animal and the user from the preprocessed data. For image data, a face detection algorithm is used to detect the faces and body parts of the animal and the user, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[0925] Emotion Analysis

[0926] The server inputs the extracted features into a machine learning model to analyze the emotions of animals and users. This model is pre-trained with a large amount of data and is designed to identify basic emotions (happiness, anger, anxiety, excitement, etc.) of animals and users. As a result of the analysis, the emotions of animals and users are tagged.

[0927] Generate visualization data

[0928] The server generates visualization data based on the analyzed emotion data. This visualization data includes facial expression icons for animals and the user, as well as charts that graphically represent the analysis results. The generated visualization data is sent to the device in an appropriate format, such as JSON.

[0929] Displaying the results

[0930] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, detailed information such as the shape of the eyes and the degree to which the mouth is open or closed is displayed along with icons and tags that indicate the emotions of the animal and the user. This allows the results of two-way emotion analysis between the animal and the user to be confirmed.

[0931] Gathering feedback

[0932] The user checks the displayed emotion analysis results and evaluates whether the animal's and the user's emotions have been analyzed correctly. This evaluation is performed by comparing the emotion analysis results with the animal's actual behavior and situation, and the user's own emotions. The user inputs this evaluation result as feedback and sends it to the server.

[0933] Re-learning with feedback

[0934] The server analyzes the received feedback data and uses it to retrain the emotion analysis model for animals and users. This improves the system's analysis accuracy and also improves the accuracy of future emotion analysis. The retrained model is then used for the next data analysis.

[0935] Specific examples

[0936] Emotional analysis of dogs and users

[0937] 1. The user takes a video of the dog with their smartphone and records their own facial expressions and voice.

[0938] 2. The device uploads this video and audio data to the server.

[0939] 3. The server preprocesses the video and audio data and extracts features.

[0940] 4. The server analyzes the animal and user's emotions using a machine learning model and estimates that the dog is "happy" and the user is "pleased."

[0941] 5. The server generates visualization data based on the analysis results and sends it to the terminal.

[0942] 6. The device will display messages to the user such as "The dog is happy" and "You are happy," and also show icons such as a wagging tail and a smiling face.

[0943] 7. The user checks this information and evaluates whether the analysis results are correct, inputs the evaluation results as feedback, and sends them to the server.

[0944] In this way, the system of the present invention realizes smooth communication between animals and humans by analyzing the emotions of both animals and users in both directions and providing the results to the user.

[0945] The processing flow will be explained below.

[0946] Step 1: Get the data

[0947] Users use devices such as smartphones or tablets to take videos or images of animals, while simultaneously recording their own facial expressions and tone of voice, thereby collecting basic data for analyzing the emotions of both the animals and the user.

[0948] Step 2: Save and upload your data

[0949] The device temporarily stores the captured video or image data, along with the user's facial expressions and voice data. The data is then uploaded to a server via the Internet. At this time, the data format and size are checked and converted to the appropriate format.

[0950] Step 3: Preprocessing the data

[0951] The server receives the uploaded data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates and saves audio data from the video. The user's facial expression and audio data are also preprocessed in the same way, with noise removal and standardization.

[0952] Step 4: Feature extraction

[0953] The server extracts features of both the animal and the user from the preprocessed data. For image data, a face detection algorithm is used to detect the faces and body parts of the animal and the user, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[0954] Step 5: Sentiment Analysis

[0955] The server inputs the extracted features into a machine learning model to analyze the emotions of animals and users. This model is pre-trained with a large amount of data and is designed to identify basic emotions (happiness, anger, anxiety, excitement, etc.) of animals and users. As a result of the analysis, the emotions of animals and users are tagged.

[0956] Step 6: Generate visualization data

[0957] The server generates visualization data based on the analyzed emotion data. This visualization data includes facial expression icons for animals and the user, as well as charts that graphically represent the analysis results. The generated visualization data is sent to the device in an appropriate format, such as JSON.

[0958] Step 7: View the results

[0959] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, detailed information such as the shape of the eyes and the degree to which the mouth is open or closed is displayed along with icons and tags that indicate the emotions of the animal and the user. This allows the results of two-way emotion analysis between the animal and the user to be confirmed.

[0960] Step 8: Gather feedback

[0961] The user checks the displayed emotion analysis results and evaluates whether the animal's and the user's emotions have been analyzed correctly. This evaluation is performed by comparing the emotion analysis results with the animal's actual behavior and situation, and the user's own emotions. The user inputs this evaluation result as feedback and sends it to the server.

[0962] Step 9: Relearn with feedback

[0963] The server analyzes the received feedback data and uses it to retrain the emotion analysis model for animals and users. This improves the system's analysis accuracy and also improves the accuracy of future emotion analysis. The retrained model is then used for the next data analysis.

[0964] Through the above steps, the system of the present invention analyzes the emotions of animals and users in both directions and provides the results to the user, thereby realizing smooth communication between animals and humans.

[0965] Example 2

[0966] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0967] Conventional animal emotion analysis systems only analyze animal emotions, and are unable to analyze emotions in both directions with humans. Furthermore, there is no system that can simultaneously analyze the emotions of both humans and animals, visualize them, and promote two-way communication, making it difficult to smoothly communicate with animals.

[0968] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring video or image data of an animal, means for extracting the animal's facial expressions, gestures, and sounds from the data, means for simultaneously acquiring the user's facial expressions and voice data, means for transmitting user and animal data to the server, means for checking the format and size of the data and performing preprocessing, means for analyzing the emotions of the animal and the user based on the extracted features, means for visualizing the analyzed emotional data in a format that is easy for humans to understand, means for transmitting the visualized data to a terminal, means for displaying the data on the terminal, means for receiving feedback from the user and transmitting it to the server, and means for relearning a machine learning model using the feedback. This enables two-way emotion analysis between the animal and the user and smooth communication based on the analysis.

[0969] "Animal video or image data" refers to video or still image data that records the movements and expressions of animals captured by a user.

[0970] "Facial expressions, gestures, and sounds" refer to the facial expressions, body movements, and sounds that animals make, and are used to infer emotions.

[0971] "User facial expression and voice data" refers to data such as facial expressions, tone of voice, and timbre that are recorded when a user uses the system.

[0972] A "server" is a central computer system that receives data sent from terminals via the Internet, analyzes and stores it, and manages feedback.

[0973] "Preprocessing" refers to data processing operations such as noise removal and standardization that are performed on collected data before analysis.

[0974] "Features" are identifiable information extracted from data, such as eye shape, mouth open / closed state, ear position, voice pitch, and frequency spectrum, which are necessary for emotion analysis.

[0975] A "machine learning model" is an algorithm or program within a system that learns from large amounts of data in advance to analyze and identify emotions.

[0976] "Visualized data" refers to data such as graphical information, icons, and charts created to display the analyzed emotion results in an easy-to-understand manner.

[0977] A "terminal" is a device (smartphone, tablet, PC, etc.) used by a user to access the system.

[0978] "Feedback" refers to information on the user's evaluation of the analysis results, and is data used to improve the system and retrain the machine learning model.

[0979] "Retraining" refers to the process of reanalyzing data and updating models to improve the accuracy of machine learning models using feedback data collected by the system.

[0980] MODE FOR CARRYING OUT THE INVENTION

[0981] The present invention realizes two-way emotion analysis between animals and humans by combining a system that analyzes and visualizes animal emotions with an emotion engine that recognizes user emotions. Specific embodiments of the system are described below.

[0982] Hardware and Software Use

[0983] Users use devices such as smartphones and tablets, which are equipped with cameras and microphones that can simultaneously capture data such as facial and body expressions and voice.

[0984] The devices include smartphones and tablets running Android or iOS. A dedicated application is installed on the device, which has the ability to capture and temporarily store videos, images, user facial expressions, and audio data.

[0985] The server is a computer system built on the cloud that stores data, preprocesses it, extracts features, analyzes emotions using machine learning models, generates visualization data, and manages feedback. It can utilize common cloud services such as AWS (Amazon Web Services) and GCP (Google Cloud Platform).

[0986] Data Acquisition and Transmission

[0987] Users take videos or images of animals with their smartphone camera, and simultaneously record their own facial expressions and tone of voice. The captured data is temporarily stored in a dedicated directory on the device.

[0988] Data storage and upload

[0989] The device uploads the acquired data to a cloud server via the Internet, where the data format and size are checked and, if necessary, converted into an appropriate format before being sent.

[0990] Data Preprocessing

[0991] The server performs preprocessing on the received data before analyzing it. Specifically, it breaks down video data into still images for each frame, separates audio data and stores it separately. It also performs noise removal and standardization on image data to ensure uniform quality.

[0992] Feature extraction

[0993] The server extracts features of animals and users from the preprocessed data. For example, for image data, OpenCV is used to detect facial and body features, obtaining information such as eye shape, mouth open / closed state, and ear position. For audio data, a voice analysis library such as LibROSA is used to extract features such as pitch, frequency spectrum, and sound duration.

[0994] Emotion Analysis

[0995] The server then inputs the extracted features into a pre-trained machine learning model (using TensorFlow and Scikit-learn) to analyze the emotions of animals and users. The model is designed to identify basic emotions (such as joy, anger, anxiety, and excitement) of animals and users.

[0996] Generate visualization data

[0997] The server generates visualization data based on the analyzed emotion data. Specifically, it creates icons showing the animal and user's facial expressions, as well as charts that graphically represent the analysis results. The generated visualization data is sent to the device in JSON format or similar.

[0998] Displaying the results

[0999] The device displays the results on a user interface (UI) based on the visualization data received from the server. For example, detailed information is provided to the user along with icons and tags indicating the emotions of the animals and the user.

[1000] Gathering feedback and relearning

[1001] The user checks the displayed emotion analysis results and evaluates their accuracy. The evaluation results are then sent as feedback from the device to the server. The server then retrains the machine learning model based on the received feedback data to improve analysis accuracy.

[1002] Examples and prompts

[1003] For example, suppose a user takes a video of their dog with their smartphone and records their own facial expressions and voice. This data is uploaded to a server, where it undergoes preprocessing, feature extraction, and emotion analysis to analyze the animal's and user's emotions as "the dog is happy" and "the user is happy." The analysis results are visualized and sent to a device for the user to review.

[1004] Examples of prompts include natural language sentences such as "Analyze the dog's emotions and see if it is happy" and "Analyze the emotions in my tone of voice and tell me what you think."

[1005] As described above, the present invention is a system that realizes smooth communication between animals and humans by analyzing the emotions of animals and users in both directions and providing the results to the user.

[1006] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1007] Step 1: Get the data

[1008] The user uses the camera and microphone functions of their smartphone or tablet to capture video or images of the animal. At the same time, the user's facial expressions and tone of voice are also recorded. The specific inputs are the animal's video and audio data, and the user's video and audio data, which are temporarily stored in the device's internal storage. The output is these data files.

[1009] Step 2: Save and upload your data

[1010] The device saves the acquired video and audio data in a dedicated directory. When saving, a timestamp is added to the file name to make it unique. For example, "video_YYYYMMDD_HHMMSS.mp4". Next, the format and size of the saved data are checked, and the data is uploaded to a cloud server via the Internet. The input is the saved data file, and the output is the data transferred to the server.

[1011] Step 3: Preprocessing the data

[1012] The server preprocesses the received data before analyzing it. It breaks down the video data into frames and saves them as still images (e.g., "frame_001.jpg", "frame_002.jpg", etc.). It also separates the audio data from the video and saves it (e.g., "audio.wav"). It performs filtering to remove noise and standardize the data. The input is the uploaded raw data, and the output is preprocessed still image data and audio data.

[1013] Step 4: Feature extraction

[1014] The server extracts features of animals and users from the preprocessed data. For image data, a face detection algorithm (for example, the OpenCV library) is used to detect the faces and body parts of animals and users. Features extracted include eye shape, mouth open / closed state, and ear position. For audio data, an audio analysis library such as LibROSA is used to extract features such as pitch, frequency spectrum, and sound duration. The input is preprocessed still image data and audio data, and the output is the features of animals and users.

[1015] Step 5: Sentiment Analysis

[1016] The server inputs the features into a pre-trained machine learning model (using, for example, TensorFlow or Scikit-learn) to analyze the emotions of the animal and the user. This model identifies basic emotions (e.g., joy, anger, anxiety, excitement, etc.) of the animal and the user. As a result of the analysis, emotion tags are assigned to each animal and user. The input is the feature data, and the output is the analysis result with the emotion tag.

[1017] Step 6: Generate visualization data

[1018] The server generates visualized data based on the analyzed emotion data. Specifically, it creates facial expression icons for animals and the user, as well as charts and graphs that graphically represent the analysis results. The generated visualized data is saved in JSON format or similar and sent to the terminal via the Internet. The input is the analysis results with emotion tags, and the output is the visualized data.

[1019] Step 7: View the results

[1020] The terminal analyzes the visualization data received from the server and displays the results on a user interface (UI). Specifically, it displays icons and messages indicating the emotion tags of animals and users, icons of a dog wagging its tail, smiling faces, etc. The input is the visualization data, and the output is the analysis results displayed on the UI.

[1021] Step 8: Gather feedback

[1022] The user checks the displayed emotion analysis results and evaluates their accuracy. The evaluation results are input to the terminal as feedback and sent to the server. The input is the user's evaluation data, and the output is the feedback data sent to the server.

[1023] Step 9: Relearn with feedback

[1024] The server analyzes the received feedback data and uses it to retrain the sentiment analysis model, thereby improving the accuracy of the machine learning model. The retrained model is then used for the next data analysis. The input is the feedback data, and the output is the improved machine learning model after retraining.

[1025] (Application example 2)

[1026] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1027] While modern technology has advanced to analyze the emotions of animals and humans and facilitate communication between them, there are still no systems that provide specific suggestions that are useful in real life based on the analysis results. Furthermore, there are no adequate mechanisms for suggesting appropriate products based on pet emotions, collecting user feedback on those suggestions, and improving the accuracy of the model. This makes it difficult for users to understand their pets' emotions and provide optimal care.

[1028] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1029] In this invention, the server includes means for acquiring video or image data of an animal, means for extracting the animal's facial expressions, gestures, and cries from the data, means for analyzing the animal's emotions based on the extracted features, means for visualizing the analyzed emotional data in a format that is easy for humans to understand, means for transmitting the visualized data to a terminal, means for displaying the data on the terminal, means for suggesting related products based on the analyzed emotional data, and means for collecting user feedback on the product suggestion results and using the feedback to retrain the analytical model. This enables users to appropriately understand their pet's emotions and receive suggestions that are useful in their daily lives through specific product suggestions based on the animal's emotions and by collecting feedback.

[1030] "Video or image data" refers to visual data that records the movements and expressions of animals.

[1031] "Facial expressions, gestures, and sounds" are the facial movements, body movements, and sounds that an animal makes.

[1032] "Features" are attributes of data extracted to analyze animal emotions.

[1033] "Emotion data" is information about an animal's emotions analyzed based on its features.

[1034] "Visualization" refers to presenting analysis results in a format that is easy for humans to understand.

[1035] A "terminal" is a digital device such as a smartphone or tablet.

[1036] "Product suggestions" refers to recommending related products such as pet food and toys to users based on the analysis results.

[1037] "Feedback" means collecting user ratings and opinions.

[1038] "Retraining" is the process of improving the accuracy of an analytical model based on collected feedback.

[1039] "Server" means a central computer system for storing, analyzing, transmitting and receiving data.

[1040] "Data preprocessing" is the process of removing noise and standardizing data before analyzing it.

[1041] A "machine learning algorithm" is a computational method for extracting features from data and performing analysis.

[1042] "Face detection" is a technology that identifies the location of a face from image data.

[1043] "Audio feature extraction" is the process of extracting information such as pitch and frequency from audio data.

[1044] "Database linkage" refers to the exchange of information with a database that stores information on the products being proposed.

[1045] This invention is a system that analyzes the emotions of animals and suggests the most suitable related products based on the analysis. This system is realized using a smartphone, tablet, and server.

[1046] First, the user uses a smartphone or tablet to acquire video or image data of the animal. The user takes videos or images of their pet, simultaneously recording their own facial expressions and tone of voice. This data provides the foundation for analyzing the emotions of the animal and the user.

[1047] The device then temporarily stores the captured video or image, along with the user's facial expression and audio data. These data are then uploaded to a server via the Internet. The server checks the format and size of the data and converts it into an appropriate format if necessary.

[1048] The server preprocesses the received data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates the audio data from the video and saves it. This allows the server to process the image and audio data separately, removing noise and normalizing them.

[1049] The server then extracts features of the animal and the user from the preprocessed data. For image data, a face detection algorithm (e.g., Haar) is used to detect the faces and body parts of the animal and the user, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[1050] The server analyzes the animal and user's emotions using a pre-trained generative AI model based on the extracted features. As a result, the animal and user's emotions are tagged. Then, based on the analyzed emotion data, it suggests related products (e.g., specific pet food or toys) that match the animal's emotion.

[1051] The results of these suggestions are sent to the device in an appropriate format, such as JSON, and the device displays the results on its user interface. The user then checks the products suggested by the system and makes a purchasing decision based on them. The user then inputs feedback about the effectiveness of the suggested products and the pet's reaction, which is then sent to the server. The server then retrains the analysis model based on this feedback, improving the accuracy of the next data analysis.

[1052] For example, if a user takes a video of their dog and the app analyzes it, and determines that the dog is "happy," the app will suggest pet food or toys with a "joy" theme. The user can purchase the suggested products and provide feedback on their pet's subsequent reactions, thereby improving the accuracy of the entire system.

[1053] Example prompts to input to a generative AI model:

[1054] Write a Python program that, given a video path, analyzes a pet's emotions and suggests the best pet food and toys. The program uses OpenCV to process the video and analyzes emotions using a Keras emotion recognition model.

[1055] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1056] Step 1:

[1057] Users use a smartphone or tablet to capture video or image data of animals. The input is the captured video or image data, and the output is data temporarily stored on the device. The user takes a video of their pet and simultaneously records their own facial expressions and tone of voice.

[1058] Step 2:

[1059] The device temporarily stores the captured video or image, along with the user's facial expression and audio data. It then uploads this data to a server via the Internet. The input is the temporarily stored video, image, and audio data, and the output is the data uploaded to the server. The device also checks the format and size of the data and converts it to an appropriate format if necessary.

[1060] Step 3:

[1061] The server preprocesses the received data. In the case of video data, it breaks the video down into frames and saves them as still images. It also separates the audio data from the video and saves it. The input is the uploaded video data, and the output is the image data broken down into frames and the separated audio data. The server also performs noise removal and normalization on these data.

[1062] Step 4:

[1063] The server extracts features of animals and users from the preprocessed data. The input is preprocessed image and audio data, and the output is the extracted features. For image data, a face detection algorithm (e.g., Haar) is used to detect the faces and body parts of animals and users, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[1064] Step 5:

[1065] The server uses the extracted features to analyze the emotions of the animal and the user using a pre-trained generative AI model. The input is the extracted features, and the output is the analyzed emotion data. The generative AI model is designed to identify basic emotions (happiness, anger, anxiety, excitement, etc.). Depending on the emotion data, tagging is performed.

[1066] Step 6:

[1067] Based on the analysis results, the server suggests related products (e.g., specific pet food or toys) that match the animal's emotions. The input is the analyzed emotion data, and the output is data about the suggested products. The server also sends these suggestion data to the device in an appropriate format, such as JSON.

[1068] Step 7:

[1069] The terminal displays the results on a user interface (UI) based on the recommendation data received from the server. The input is data about the recommended products, and the output is visualized data that is displayed to the user. The user reviews the recommended products and makes a purchasing decision based on their content.

[1070] Step 8:

[1071] The user inputs the effects of the proposed product and the pet's reaction as feedback and sends it to the server. The input is feedback information, and the output is feedback data stored on the server.

[1072] Step 9:

[1073] The server retrains the sentiment analysis model based on the received feedback. The input is the feedback data, and the output is an updated sentiment analysis model. This improves the accuracy of the next data analysis.

[1074] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1075] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1076] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1077] [Fourth embodiment]

[1078] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1079] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1080] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1081] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1082] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1083] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1084] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1085] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1086] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1087] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1088] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1089] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1090] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1091] The system of the present invention collects video or image data of animals, analyzes the emotions of the animals from the data, and displays them in a format that is easy for humans to understand. Below, the processing of the system's program will be explained in natural language and in detail with concrete examples.

[1092] Retrieving data

[1093] Users take videos and photos of animals using a device (such as a smartphone or tablet), and this data is temporarily stored on the device.

[1094] Uploading data

[1095] The device uploads the captured video and image data to the server. At this time, the format and size of the data being uploaded are checked and it is converted into the appropriate format.

[1096] Data Preprocessing

[1097] The server performs preprocessing on the received data. In the case of video data, the video is broken down into frames, and the image and audio data are stored separately. Noise removal and data standardization are also performed at this stage.

[1098] Feature extraction

[1099] The server uses machine learning algorithms to extract animal features from the data. From the image data, it detects the animal's face and body parts and extracts features such as eye shape, mouth opening and closing position, and ear position. From the audio data, it extracts features such as pitch, frequency spectrum, and sound duration.

[1100] Emotion Analysis

[1101] The server inputs the features into a pre-trained machine learning model to analyze the animal's emotions. This model uses convolutional neural networks (CNNs) and recurrent neural networks (RNNs). As a result of the analysis, the animal's emotions (happiness, anger, anxiety, excitement, etc.) are estimated and tagged.

[1102] Generate visualization data

[1103] The server generates visualization data based on the analyzed emotion data. This visualization data includes icons and graphs that represent the animal's facial expressions. The generated visualization data is sent to the device in JSON format or similar.

[1104] Displaying the results

[1105] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, it displays tags that indicate the animal's emotions, as well as detailed information such as the shape of its eyes and whether its mouth is open or closed.

[1106] Gathering feedback

[1107] The user checks the displayed information and evaluates whether the animal's emotions have been correctly analyzed. Based on the evaluation, the user enters feedback, which is then sent to the server. The feedback is used to improve the accuracy of the system.

[1108] Specific examples

[1109] Dog Emotion Analysis

[1110] 1. The user takes videos of the dog with their smartphone and records the dog's behavior and sounds.

[1111] 2. The device uploads this video data to the server.

[1112] 3. The server extracts frames from the video, separates them from the audio data, and performs preprocessing.

[1113] 4. The server detects the dog's face and body from the image data and extracts features such as the shape of the eyes, whether the mouth is open or closed, and the position of the ears. It also extracts the pitch and frequency spectrum from the audio data.

[1114] 5. The server uses a machine learning model to analyze the dog's emotion as "joy."

[1115] 6. The server generates visualization data based on the analysis results and sends it to the terminal.

[1116] 7. The device will display to the user "Dog is happy" and also show icons such as a wagging tail and raised ears.

[1117] 8. The user checks this information and decides whether the dog is truly happy. If the result is appropriate, the user enters feedback and sends it to the server.

[1118] In this way, the system of the present invention analyzes the emotions of animals and provides them in a format that is easy for humans to understand, thereby realizing smooth communication between animals and humans.

[1119] The processing flow will be explained below.

[1120] Step 1: Capture video or images of your animal

[1121] Users use devices such as smartphones and tablets to take videos or images of animals, and this data includes the animals' facial expressions, movements, sounds, etc.

[1122] Step 2: Save and upload your data

[1123] The device temporarily stores the captured video or image data, then uploads the data to a server via the Internet. At this time, the data format and size are checked and, if necessary, converted to an appropriate format.

[1124] Step 3: Preprocessing the data

[1125] The server receives the uploaded data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates the audio data from the video and saves it. Next, the server performs noise reduction and standardization on the image and audio data.

[1126] Step 4: Feature extraction

[1127] The server extracts animal features from the preprocessed data. For image data, a face detection algorithm is used to detect the animal's face and body parts, and features such as eye shape, mouth opening and closing, and ear position are extracted. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[1128] Step 5: Sentiment Analysis

[1129] The server inputs the extracted features into a machine learning model to analyze the animal's emotions. This model has been pre-trained with a large amount of data and is designed to identify basic animal emotions (happiness, anger, anxiety, excitement, etc.). As a result of the analysis, the animal's emotion is tagged.

[1130] Step 6: Generate visualization data

[1131] The server generates visualization data based on the analyzed emotion data. This visualization data includes animal facial expression icons and charts that graphically represent the analysis results. The generated visualization data is sent to the device in an appropriate format, such as JSON.

[1132] Step 7: View the results

[1133] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, detailed information such as the shape of the eyes and the opening and closing state of the mouth is displayed along with icons and tags that indicate the animal's emotions.

[1134] Step 8: Gather feedback

[1135] The user checks the displayed emotion analysis results and evaluates whether the animal's emotions have been correctly analyzed. This evaluation is performed by comparing the emotion analysis results with the actual behavior and situation of the animal. The user inputs this evaluation result as feedback and sends it to the server.

[1136] Step 9: Relearn with feedback

[1137] The server analyzes the received feedback data and uses it to retrain the sentiment analysis model. This improves the system's analysis accuracy and also improves the accuracy of future sentiment analysis. The retrained model is then used for the next data analysis.

[1138] The above steps will realize a system that accurately analyzes animal emotions and provides them to users, promoting smooth communication between animals and humans.

[1139] Example 1

[1140] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1141] Conventional animal emotion analysis systems have had difficulty accurately analyzing animal emotions from video data. Furthermore, the means for displaying the analyzed emotion data in an easy-to-understand manner for users have been insufficient, making it difficult for users to intuitively understand the emotions of animals. To address these issues, there is a need for the development of a system that can analyze animal emotions with high accuracy and provide the data in a visually easy-to-understand format.

[1142] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1143] In this invention, the server includes means for decomposing video data into frames and storing the audio data and image data separately, means for removing noise from the data and standardizing it, and means for extracting animal facial expressions, gestures, and cries from the data. This makes it possible to analyze emotions from animal video data with high accuracy and provide the analysis results in a visually easy-to-understand format.

[1144] "Video or image data of animals" refers to video or photographic information of pets or wild animals, which can be used to record the movements and expressions of animals.

[1145] "Terminal" refers to an electronic device such as a smartphone, tablet, or personal computer, which is used to acquire, transmit, and receive data.

[1146] A "server" refers to a computer system that processes, stores, and analyzes data via a network, and receives and processes data from multiple terminals.

[1147] "Means for uploading data" refers to the process by which a user's device sends video or image data to a server via a network.

[1148] "Means of decomposing into frames" refers to the process of dividing video data into individual still images (frames), which allows for detailed image analysis.

[1149] "Denoising and standardisation measures" refers to the process of removing unnecessary information and outliers and converting data into a consistent format to improve data quality.

[1150] "Means for extracting animal facial expressions, gestures, and sounds" refers to the process of detecting specific features from image and audio data and extracting them as data for analysis.

[1151] "Features" refer to specific attributes and patterns extracted from image and audio data, which can be used to analyze an animal's emotions and state.

[1152] A "generative AI model" refers to a model trained using machine learning algorithms that can infer animal emotions from features.

[1153] The "means for generating visualization data" refers to the process of converting the analyzed emotion data into visual information such as icons and graphs in order to display it in a format that is easy for the user to understand.

[1154] "Means for transmitting visualized data to a terminal" refers to a process in which the server transmits the generated visualization data to a user's terminal via a network.

[1155] "Means for displaying data" refers to the process of displaying analysis results and visualized data on a user interface on a user's terminal.

[1156] The system of this invention analyzes animal emotions with high accuracy and provides the user with a visually easy-to-understand format. This system is mainly composed of three elements: the user's terminal, an internet connection, and a server.

[1157] Retrieving data

[1158] Users use devices such as smartphones and tablets to take videos and photos of animals. The device's camera app is used to do this, and the captured data is temporarily stored on the device. For example, a user may take a video of their pet dog and record its various behaviors and sounds.

[1159] Uploading data

[1160] The device uploads the captured video and image data to a server via the Internet. At this time, the data format (JPEG, MP4, etc.) and size are automatically checked, and if necessary, the data is converted to the appropriate format using a library such as FFmpeg.

[1161] Data Preprocessing

[1162] The server breaks down video data into frames and stores the audio and image data separately. Specifically, it uses the OpenCV library to break down the video into frames, and FFmpeg to extract and store the audio data. It also uses SciPy and Pandas to perform noise removal and normalization. For example, the server exports one video frame and the corresponding audio from a video of a dog.

[1163] Feature extraction

[1164] The server uses OpenCV to extract features such as the animal's face, eye shape, ear position, and mouth open / close state from image data. It also uses Librosa to extract pitch, frequency spectrum, and sound duration from audio data. For example, it can extract whether a dog's ears are erect from image data and the pitch of a bark from audio data.

[1165] Emotion Analysis

[1166] The server uses a pre-trained generative AI model (CNN or RNN built using TensorFlow or PyTorch) to analyze the animal's emotions based on the extracted features. This model estimates the animal's emotions from the features and tags them with emotions such as joy, anger, anxiety, and excitement. For example, it can determine that a dog is happy based on the position of its ears and the shape of its face.

[1167] Generate visualization data

[1168] The server generates visualization data using visualization libraries such as D3.js based on the analyzed emotion data. The generated data includes icons and graphs showing the animal's facial expressions, and is transferred in JSON format. For example, it generates an icon showing a dog's happy state and a graph showing changes in emotion.

[1169] Displaying the results

[1170] The device analyzes the JSON data received from the server and displays the results on the user interface. For example, it displays the message "The dog is happy" along with a tail-wagging icon and a smiling icon.

[1171] Gathering feedback

[1172] The user checks the displayed analysis results and evaluates whether they are appropriate. The evaluation results are sent to the server as feedback and used as learning data for the system. For example, if a user evaluates the results as "correct," that evaluation helps improve the accuracy of the system.

[1173] Specific examples

[1174] 1. The user takes videos of the dog with their smartphone and records the dog's behavior and sounds.

[1175] 2. The device uploads this video data to the server.

[1176] 3. The server extracts frames from the video, separates them from the audio data, and performs preprocessing.

[1177] 4. The server detects the dog's face and body from the image data and extracts features such as the shape of the eyes, whether the mouth is open or closed, and the position of the ears. It also extracts the pitch and frequency spectrum from the audio data.

[1178] 5. The server uses a generative AI model to analyze the dog's emotion as "joy."

[1179] 6. The server generates visualization data based on the analysis results and sends it to the terminal.

[1180] 7. The device will display to the user "Dog is happy" and also show icons of the dog wagging its tail and smiling.

[1181] 8. The user checks this information and decides whether the dog is truly happy or not, and sends the evaluation result to the server as feedback.

[1182] By utilizing generative AI models, it is possible to analyze the complex emotions of animals with high accuracy and present them to users in a format that is very easy to understand.

[1183] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1184] Processing Steps

[1185] Step 1: Get the data

[1186] Users use devices such as smartphones and tablets to take videos and photos of animals. This is done using the device's camera app, and the captured data is temporarily stored on the device. The user's operations record the behavior and sounds of the animal being photographed. The input for this process is the video and image data obtained through the camera app, and the output is data temporarily stored in the device's internal storage area.

[1187] Step 2: Upload your data

[1188] The device uploads the captured video and image data to a server via the Internet. At this time, the data format (JPEG, MP4, etc.) and size are automatically checked, and if there are any problems, it is converted to the appropriate format using a library such as FFmpeg. This process uses the video and image data stored on the device as input, and the converted and checked data is sent to the server as output.

[1189] Step 3: Preprocessing the data

[1190] The server breaks down the uploaded video data into frames, and stores the audio and image data separately. Here, the OpenCV library is used to break down the video into frames, and FFmpeg is used to extract and store the audio data. SciPy and Pandas are also used to remove noise from the data and standardize it. The input for this process is the video data uploaded to the server, and the output is frame-by-frame image data and separated audio data.

[1191] Step 4: Feature extraction

[1192] The server uses OpenCV to extract features such as the animal's face, eye shape, ear position, and mouth open / close state from the image data. It also uses Librosa to extract pitch, frequency spectrum, and sound duration from the audio data. The input to this process is the image data and audio data separated and saved in the previous step, and the output is each extracted feature. Specifically, it identifies the position of the dog's ears and eye shape from the image data, and the pitch of the bark from the audio data.

[1193] Step 5: Sentiment Analysis

[1194] The server uses a pre-trained generative AI model (CNN or RNN built using TensorFlow or PyTorch) to analyze the animal's emotions based on the extracted features. This model estimates the animal's emotions from the features and tags them with emotions such as joy, anger, anxiety, and excitement. The input to this process is the extracted features, and the output is estimated emotion data. Specifically, it determines the emotion as "joy" based on the position of the dog's ears and the shape of its face.

[1195] Step 6: Generate visualization data

[1196] The server generates visualized data based on the analyzed emotion data using a visualization library such as D3.js. The generated data includes icons and graphs showing the animal's facial expressions, and is transferred to the terminal in JSON format. The input to this process is the analyzed emotion data, and the output is the generated visualized data. Specifically, it generates an icon showing the dog's happy state and a graph showing changes in emotion.

[1197] Step 7: View the results

[1198] The terminal analyzes the JSON data received from the server and displays the results on the user interface. Specifically, it displays a message saying "The dog is happy" along with a tail-wagging icon, a smiling icon, etc. The input to this process is the JSON data received from the server, and the output is the visual information displayed on the user interface.

[1199] Step 8: Gather feedback

[1200] The user checks the displayed analysis results and evaluates whether they are appropriate. The evaluation results are sent as feedback to the server and used as learning data for the system. The inputs to this process are the user's evaluation and feedback, and the output is feedback data sent to the server. In concrete terms, the user evaluates the results as "correct," and the evaluation is sent to the server.

[1201] (Application example 1)

[1202] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1203] In conventional food delivery services, when delivery personnel visit homes containing animals, it is difficult for them to understand the emotions and state of the animals. As a result, the animals may feel anxious or stressed, making it difficult to provide a safe and comfortable delivery experience. The present invention aims to solve this problem by providing a system that allows delivery personnel to understand the emotional state of animals and take appropriate action.

[1204] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1205] In this invention, the server includes means for acquiring video or image data of the animal, means for extracting the animal's facial expressions, gestures, and cries from the data, means for analyzing the animal's emotions based on the extracted features, means for visualizing the analyzed emotional data in a format that is easy for humans to understand, means for transmitting the visualized data to a terminal, means for displaying the data on the terminal, and means for conveying the animal's emotions to a delivery person during delivery work, thereby enabling the delivery person to understand the animal's emotions and take appropriate action toward the animal.

[1206] "Animal video or image data" refers to digital data of videos or photographs of animals taken with a mobile phone or other device.

[1207] "Facial expressions" refer to changes in emotions shown by the position and shape of each part of an animal's face (e.g., eyes, mouth, ears, etc.).

[1208] A "gesture" refers to a specific action or movement that an animal makes with its body.

[1209] "Cry" refers to the sound that animals make, and is a way of expressing emotions and communicating.

[1210] "Features" refer to numerical values ​​or indicators extracted from image or audio data that are necessary for emotion analysis.

[1211] "Emotion analysis" is the process of using extracted features to estimate an animal's internal emotional state (joy, anger, anxiety, excitement, etc.).

[1212] "Visualization" refers to the process of displaying analyzed emotional data in visual formats such as graphs and icons.

[1213] "Terminal" refers to a digital device for receiving and displaying information, such as a smartphone or tablet.

[1214] "A means of communicating an animal's emotions to the delivery person during delivery work" is a system that notifies the delivery person of the animal's emotional state via a terminal during food delivery.

[1215] System Configuration

[1216] The system of the present invention collects video or image data of animals, analyzes the emotions of the animals from the data, and conveys the emotions to the delivery person. This system is composed of the following elements.

[1217] 1. Device

[1218] The devices used are mobile information terminals such as smartphones and tablets, which are used to acquire video and image data of animals and send them to a server.

[1219] 2. Server

[1220] The server receives video and image data sent from the device, preprocesses the data, extracts features, analyzes emotions, and visualizes the results. Specific technologies used include machine learning algorithms such as face detection and voice feature extraction, as well as convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

[1221] 3. User Interface

[1222] The device comes with a dedicated application that visually displays the analysis results to the delivery person, including icons and messages based on the animal's emotional state.

[1223] Processing Details

[1224] 1. Data Acquisition

[1225] The user (delivery person) takes videos and photos of their pet using their smartphone.

[1226] 2. Uploading data

[1227] The device uploads the captured video and image data to the server, where the data format and size are checked and converted into the appropriate format.

[1228] 3. Data Preprocessing

[1229] The server performs preprocessing on the received data, such as noise removal and data standardization. In the case of video data, it divides it into frames and separates the image data from the audio data.

[1230] 4. Feature Extraction

[1231] The server uses machine learning algorithms to extract features such as the shape of the animal's eyes, the degree to which its mouth is open or closed, and the position of its ears from the image data, as well as the pitch and frequency spectrum from the audio data.

[1232] 5. Emotion Analysis

[1233] The server uses pre-trained CNN or RNN models to analyze the animal's emotions from the extracted features, and the resulting emotion (joy, anger, anxiety, excitement, etc.) is tagged.

[1234] 6. Generate visualization data

[1235] The server generates visualization data based on the analyzed emotion data and sends it to the terminal in JSON format or similar.

[1236] 7. Displaying the results

[1237] The device uses the visualization data received from the server to display the emotional state of the animal on the user interface. For example, a tail-wagging icon is displayed along with the message "The dog is happy."

[1238] Specific examples

[1239] Delivery person visits your home scenario

[1240] When delivering food, the delivery person takes a photo with their smartphone of the dog waiting at home.

[1241] The terminal uploads the photo data to the server.

[1242] The server analyzes the shape of the dog's eyes and the position of its ears from the image data, and extracts sound characteristics from the audio data.

[1243] The server displays the dog's "anxiety" as the analysis result.

[1244] The device displays a warning message to the delivery person saying, "Dog is anxious."

[1245] Delivery personnel use this information to approach quietly and avoid causing stress to the dog.

[1246] Prompt Sentence Examples

[1247] "Analyze a photo of a dog, determine its emotion, and output it in JSON format."

[1248] "Analyze the dog's emotions from this audio data and display the results in a graph."

[1249] This system allows delivery personnel to properly understand the emotional state of animals and provide a safe and comfortable delivery experience.

[1250] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1251] Step 1: Get the data

[1252] Users take videos and image data of animals with their smartphones, and this data is temporarily stored on the device. The input is the taken video and image data, and the output is the temporarily stored data on the device.

[1253] Step 2: Upload your data

[1254] The device uploads the captured video and image data to the server. The input is data temporarily stored on the device, and the output is data sent to the server. At this time, the format and size of the data are checked and converted into the appropriate format.

[1255] Step 3: Preprocessing the data

[1256] The server performs preprocessing on the received data. The input is the data sent to the server, and the output is the preprocessed data. In the case of video data, it is broken down into frames and separated into image data and audio data. Noise removal and data standardization are also performed here.

[1257] Step 4: Feature extraction

[1258] The server uses machine learning algorithms to extract animal features from the data. The input is preprocessed data, and the output is the extracted features. Specifically, eye shape, mouth open / closed state, and ear position are extracted from image data, and pitch and frequency spectrum are extracted from audio data.

[1259] Step 5: Sentiment Analysis

[1260] The server inputs the features into a pre-trained generative AI model (CNN or RNN) and analyzes the animal's emotions. The input is the extracted features, and the output is the emotion analysis result. As a result of the analysis, the animal's emotion (joy, anger, anxiety, excitement, etc.) is estimated and tagged.

[1261] Step 6: Generate visualization data

[1262] The server generates visualization data based on the analyzed emotion data. The input is the emotion analysis results, and the output is visualization data in JSON format. The visualization data includes icons and graphs showing animal facial expressions.

[1263] Step 7: View the results

[1264] The terminal displays the results on the user interface based on the visualization data received from the server. The input is the visualization data sent from the server, and the output is visual information on the emotional state displayed on the terminal. For example, it might display "The dog is happy" and use an icon to show the dog wagging its tail.

[1265] Step 8: Gather feedback

[1266] The user checks the displayed information and evaluates whether the animal's emotion has been correctly analyzed. The input is the emotional state displayed on the device, and the output is the user's evaluation feedback. This feedback is sent to the server and used to improve the accuracy of the system.

[1267] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1268] This invention realizes two-way emotion analysis between animals and humans by combining a system that analyzes and visualizes animal emotions with an emotion engine that recognizes user emotions. Below, the processing of the system's program is explained in natural language and in detail with concrete examples.

[1269] Retrieving data

[1270] Users use a smartphone, tablet, or other device to take videos or images of animals, while simultaneously recording their own facial expressions and tone of voice. This data provides the basis for analyzing the emotions of both the animal and the user.

[1271] Data storage and upload

[1272] The device temporarily stores the captured video or image, along with the user's facial expressions and voice data. The data is then uploaded to a server via the Internet. At this time, the data format and size are checked and converted to the appropriate format.

[1273] Data Preprocessing

[1274] The server receives the uploaded data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates and saves audio data from the video. The user's facial expression and audio data are also preprocessed in the same way, with noise removal and standardization.

[1275] Feature extraction

[1276] The server extracts features of both the animal and the user from the preprocessed data. For image data, a face detection algorithm is used to detect the faces and body parts of the animal and the user, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[1277] Emotion Analysis

[1278] The server inputs the extracted features into a machine learning model to analyze the emotions of animals and users. This model is pre-trained with a large amount of data and is designed to identify basic emotions (happiness, anger, anxiety, excitement, etc.) of animals and users. As a result of the analysis, the emotions of animals and users are tagged.

[1279] Generate visualization data

[1280] The server generates visualization data based on the analyzed emotion data. This visualization data includes facial expression icons for animals and the user, as well as charts that graphically represent the analysis results. The generated visualization data is sent to the device in an appropriate format, such as JSON.

[1281] Displaying the results

[1282] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, detailed information such as the shape of the eyes and the degree to which the mouth is open or closed is displayed along with icons and tags that indicate the emotions of the animal and the user. This allows the results of two-way emotion analysis between the animal and the user to be confirmed.

[1283] Gathering feedback

[1284] The user checks the displayed emotion analysis results and evaluates whether the animal's and the user's emotions have been analyzed correctly. This evaluation is performed by comparing the emotion analysis results with the animal's actual behavior and situation, and the user's own emotions. The user inputs this evaluation result as feedback and sends it to the server.

[1285] Re-learning with feedback

[1286] The server analyzes the received feedback data and uses it to retrain the emotion analysis model for animals and users. This improves the system's analysis accuracy and also improves the accuracy of future emotion analysis. The retrained model is then used for the next data analysis.

[1287] Specific examples

[1288] Emotional analysis of dogs and users

[1289] 1. The user takes a video of the dog with their smartphone and records their own facial expressions and voice.

[1290] 2. The device uploads this video and audio data to the server.

[1291] 3. The server preprocesses the video and audio data and extracts features.

[1292] 4. The server analyzes the animal and user's emotions using a machine learning model and estimates that the dog is "happy" and the user is "pleased."

[1293] 5. The server generates visualization data based on the analysis results and sends it to the terminal.

[1294] 6. The device will display messages to the user such as "The dog is happy" and "You are happy," and also show icons such as a wagging tail and a smiling face.

[1295] 7. The user checks this information and evaluates whether the analysis results are correct, inputs the evaluation results as feedback, and sends them to the server.

[1296] In this way, the system of the present invention realizes smooth communication between animals and humans by analyzing the emotions of both animals and users in both directions and providing the results to the user.

[1297] The processing flow will be explained below.

[1298] Step 1: Get the data

[1299] Users use devices such as smartphones or tablets to take videos or images of animals, while simultaneously recording their own facial expressions and tone of voice, thereby collecting basic data for analyzing the emotions of both the animals and the user.

[1300] Step 2: Save and upload your data

[1301] The device temporarily stores the captured video or image data, along with the user's facial expressions and voice data. The data is then uploaded to a server via the Internet. At this time, the data format and size are checked and converted to the appropriate format.

[1302] Step 3: Preprocessing the data

[1303] The server receives the uploaded data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates and saves audio data from the video. The user's facial expression and audio data are also preprocessed in the same way, with noise removal and standardization.

[1304] Step 4: Feature extraction

[1305] The server extracts features of both the animal and the user from the preprocessed data. For image data, a face detection algorithm is used to detect the faces and body parts of the animal and the user, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[1306] Step 5: Sentiment Analysis

[1307] The server inputs the extracted features into a machine learning model to analyze the emotions of animals and users. This model is pre-trained with a large amount of data and is designed to identify basic emotions (happiness, anger, anxiety, excitement, etc.) of animals and users. As a result of the analysis, the emotions of animals and users are tagged.

[1308] Step 6: Generate visualization data

[1309] The server generates visualization data based on the analyzed emotion data. This visualization data includes facial expression icons for animals and the user, as well as charts that graphically represent the analysis results. The generated visualization data is sent to the device in an appropriate format, such as JSON.

[1310] Step 7: View the results

[1311] The device displays the results on a user interface (UI) based on the visualization data received from the server. Specifically, detailed information such as the shape of the eyes and the degree to which the mouth is open or closed is displayed along with icons and tags that indicate the emotions of the animal and the user. This allows the results of two-way emotion analysis between the animal and the user to be confirmed.

[1312] Step 8: Gather feedback

[1313] The user checks the displayed emotion analysis results and evaluates whether the animal's and the user's emotions have been analyzed correctly. This evaluation is performed by comparing the emotion analysis results with the animal's actual behavior and situation, and the user's own emotions. The user inputs this evaluation result as feedback and sends it to the server.

[1314] Step 9: Relearn with feedback

[1315] The server analyzes the received feedback data and uses it to retrain the emotion analysis model for animals and users. This improves the system's analysis accuracy and also improves the accuracy of future emotion analysis. The retrained model is then used for the next data analysis.

[1316] Through the above steps, the system of the present invention analyzes the emotions of animals and users in both directions and provides the results to the user, thereby realizing smooth communication between animals and humans.

[1317] Example 2

[1318] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1319] Conventional animal emotion analysis systems only analyze animal emotions, and are unable to analyze emotions in both directions with humans. Furthermore, there is no system that can simultaneously analyze the emotions of both humans and animals, visualize them, and promote two-way communication, making it difficult to smoothly communicate with animals.

[1320] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring video or image data of an animal, means for extracting the animal's facial expressions, gestures, and sounds from the data, means for simultaneously acquiring the user's facial expressions and voice data, means for transmitting user and animal data to the server, means for checking the format and size of the data and performing preprocessing, means for analyzing the emotions of the animal and the user based on the extracted features, means for visualizing the analyzed emotional data in a format that is easy for humans to understand, means for transmitting the visualized data to a terminal, means for displaying the data on the terminal, means for receiving feedback from the user and transmitting it to the server, and means for relearning a machine learning model using the feedback. This enables two-way emotion analysis between the animal and the user and smooth communication based on the analysis.

[1321] "Animal video or image data" refers to video or still image data that records the movements and expressions of animals captured by a user.

[1322] "Facial expressions, gestures, and sounds" refer to the facial expressions, body movements, and sounds that animals make, and are used to infer emotions.

[1323] "User facial expression and voice data" refers to data such as facial expressions, tone of voice, and timbre that are recorded when a user uses the system.

[1324] A "server" is a central computer system that receives data sent from terminals via the Internet, analyzes and stores it, and manages feedback.

[1325] "Preprocessing" refers to data processing operations such as noise removal and standardization that are performed on collected data before analysis.

[1326] "Features" are identifiable information extracted from data, such as eye shape, mouth open / closed state, ear position, voice pitch, and frequency spectrum, which are necessary for emotion analysis.

[1327] A "machine learning model" is an algorithm or program within a system that learns from large amounts of data in advance to analyze and identify emotions.

[1328] "Visualized data" refers to data such as graphical information, icons, and charts created to display the analyzed emotion results in an easy-to-understand manner.

[1329] A "terminal" is a device (smartphone, tablet, PC, etc.) used by a user to access the system.

[1330] "Feedback" refers to information on the user's evaluation of the analysis results, and is data used to improve the system and retrain the machine learning model.

[1331] "Retraining" refers to the process of reanalyzing data and updating models to improve the accuracy of machine learning models using feedback data collected by the system.

[1332] MODE FOR CARRYING OUT THE INVENTION

[1333] The present invention realizes two-way emotion analysis between animals and humans by combining a system that analyzes and visualizes animal emotions with an emotion engine that recognizes user emotions. Specific embodiments of the system are described below.

[1334] Hardware and Software Use

[1335] Users use devices such as smartphones and tablets, which are equipped with cameras and microphones that can simultaneously capture data such as facial and body expressions and voice.

[1336] The devices include smartphones and tablets running Android or iOS. A dedicated application is installed on the device, which has the ability to capture and temporarily store videos, images, user facial expressions, and audio data.

[1337] The server is a computer system built on the cloud that stores data, preprocesses it, extracts features, analyzes emotions using machine learning models, generates visualization data, and manages feedback. It can utilize common cloud services such as AWS (Amazon Web Services) and GCP (Google Cloud Platform).

[1338] Data Acquisition and Transmission

[1339] Users take videos or images of animals with their smartphone camera, and simultaneously record their own facial expressions and tone of voice. The captured data is temporarily stored in a dedicated directory on the device.

[1340] Data storage and upload

[1341] The device uploads the acquired data to a cloud server via the Internet, where the data format and size are checked and, if necessary, converted into an appropriate format before being sent.

[1342] Data Preprocessing

[1343] The server performs preprocessing on the received data before analyzing it. Specifically, it breaks down video data into still images for each frame, separates audio data and stores it separately. It also performs noise removal and standardization on image data to ensure uniform quality.

[1344] Feature extraction

[1345] The server extracts features of animals and users from the preprocessed data. For example, for image data, OpenCV is used to detect facial and body features, obtaining information such as eye shape, mouth open / closed state, and ear position. For audio data, a voice analysis library such as LibROSA is used to extract features such as pitch, frequency spectrum, and sound duration.

[1346] Emotion Analysis

[1347] The server then inputs the extracted features into a pre-trained machine learning model (using TensorFlow and Scikit-learn) to analyze the emotions of animals and users. The model is designed to identify basic emotions (such as joy, anger, anxiety, and excitement) of animals and users.

[1348] Generate visualization data

[1349] The server generates visualization data based on the analyzed emotion data. Specifically, it creates icons showing the animal and user's facial expressions, as well as charts that graphically represent the analysis results. The generated visualization data is sent to the device in JSON format or similar.

[1350] Displaying the results

[1351] The device displays the results on a user interface (UI) based on the visualization data received from the server. For example, detailed information is provided to the user along with icons and tags indicating the emotions of the animals and the user.

[1352] Gathering feedback and relearning

[1353] The user checks the displayed emotion analysis results and evaluates their accuracy. The evaluation results are then sent as feedback from the device to the server. The server then retrains the machine learning model based on the received feedback data to improve analysis accuracy.

[1354] Examples and prompts

[1355] For example, suppose a user takes a video of their dog with their smartphone and records their own facial expressions and voice. This data is uploaded to a server, where it undergoes preprocessing, feature extraction, and emotion analysis to analyze the animal's and user's emotions as "the dog is happy" and "the user is happy." The analysis results are visualized and sent to a device for the user to review.

[1356] Examples of prompts include natural language sentences such as "Analyze the dog's emotions and see if it is happy" and "Analyze the emotions in my tone of voice and tell me what you think."

[1357] As described above, the present invention is a system that realizes smooth communication between animals and humans by analyzing the emotions of animals and users in both directions and providing the results to the user.

[1358] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1359] Step 1: Get the data

[1360] The user uses the camera and microphone functions of their smartphone or tablet to capture video or images of the animal. At the same time, the user's facial expressions and tone of voice are also recorded. The specific inputs are the animal's video and audio data, and the user's video and audio data, which are temporarily stored in the device's internal storage. The output is these data files.

[1361] Step 2: Save and upload your data

[1362] The device saves the acquired video and audio data in a dedicated directory. When saving, a timestamp is added to the file name to make it unique. For example, "video_YYYYMMDD_HHMMSS.mp4". Next, the format and size of the saved data are checked, and the data is uploaded to a cloud server via the Internet. The input is the saved data file, and the output is the data transferred to the server.

[1363] Step 3: Preprocessing the data

[1364] The server preprocesses the received data before analyzing it. It breaks down the video data into frames and saves them as still images (e.g., "frame_001.jpg", "frame_002.jpg", etc.). It also separates the audio data from the video and saves it (e.g., "audio.wav"). It performs filtering to remove noise and standardize the data. The input is the uploaded raw data, and the output is preprocessed still image data and audio data.

[1365] Step 4: Feature extraction

[1366] The server extracts features of animals and users from the preprocessed data. For image data, a face detection algorithm (for example, the OpenCV library) is used to detect the faces and body parts of animals and users. Features extracted include eye shape, mouth open / closed state, and ear position. For audio data, an audio analysis library such as LibROSA is used to extract features such as pitch, frequency spectrum, and sound duration. The input is preprocessed still image data and audio data, and the output is the features of animals and users.

[1367] Step 5: Sentiment Analysis

[1368] The server inputs the features into a pre-trained machine learning model (using, for example, TensorFlow or Scikit-learn) to analyze the emotions of the animal and the user. This model identifies basic emotions (e.g., joy, anger, anxiety, excitement, etc.) of the animal and the user. As a result of the analysis, emotion tags are assigned to each animal and user. The input is the feature data, and the output is the analysis result with the emotion tag.

[1369] Step 6: Generate visualization data

[1370] The server generates visualized data based on the analyzed emotion data. Specifically, it creates facial expression icons for animals and the user, as well as charts and graphs that graphically represent the analysis results. The generated visualized data is saved in JSON format or similar and sent to the terminal via the Internet. The input is the analysis results with emotion tags, and the output is the visualized data.

[1371] Step 7: View the results

[1372] The terminal analyzes the visualization data received from the server and displays the results on a user interface (UI). Specifically, it displays icons and messages indicating the emotion tags of animals and users, icons of a dog wagging its tail, smiling faces, etc. The input is the visualization data, and the output is the analysis results displayed on the UI.

[1373] Step 8: Gather feedback

[1374] The user checks the displayed emotion analysis results and evaluates their accuracy. The evaluation results are input to the terminal as feedback and sent to the server. The input is the user's evaluation data, and the output is the feedback data sent to the server.

[1375] Step 9: Relearn with feedback

[1376] The server analyzes the received feedback data and uses it to retrain the sentiment analysis model, thereby improving the accuracy of the machine learning model. The retrained model is then used for the next data analysis. The input is the feedback data, and the output is the improved machine learning model after retraining.

[1377] (Application example 2)

[1378] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1379] While modern technology has advanced to analyze the emotions of animals and humans and facilitate communication between them, there are still no systems that provide specific suggestions that are useful in real life based on the analysis results. Furthermore, there are no adequate mechanisms for suggesting appropriate products based on pet emotions, collecting user feedback on those suggestions, and improving the accuracy of the model. This makes it difficult for users to understand their pets' emotions and provide optimal care.

[1380] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1381] In this invention, the server includes means for acquiring video or image data of an animal, means for extracting the animal's facial expressions, gestures, and cries from the data, means for analyzing the animal's emotions based on the extracted features, means for visualizing the analyzed emotional data in a format that is easy for humans to understand, means for transmitting the visualized data to a terminal, means for displaying the data on the terminal, means for suggesting related products based on the analyzed emotional data, and means for collecting user feedback on the product suggestion results and using the feedback to retrain the analytical model. This enables users to appropriately understand their pet's emotions and receive suggestions that are useful in their daily lives through specific product suggestions based on the animal's emotions and by collecting feedback.

[1382] "Video or image data" refers to visual data that records the movements and expressions of animals.

[1383] "Facial expressions, gestures, and sounds" are the facial movements, body movements, and sounds that an animal makes.

[1384] "Features" are attributes of data extracted to analyze animal emotions.

[1385] "Emotion data" is information about an animal's emotions analyzed based on its features.

[1386] "Visualization" refers to presenting analysis results in a format that is easy for humans to understand.

[1387] A "terminal" is a digital device such as a smartphone or tablet.

[1388] "Product suggestions" refers to recommending related products such as pet food and toys to users based on the analysis results.

[1389] "Feedback" means collecting user ratings and opinions.

[1390] "Retraining" is the process of improving the accuracy of an analytical model based on collected feedback.

[1391] "Server" means a central computer system for storing, analyzing, transmitting and receiving data.

[1392] "Data preprocessing" is the process of removing noise and standardizing data before analyzing it.

[1393] A "machine learning algorithm" is a computational method for extracting features from data and performing analysis.

[1394] "Face detection" is a technology that identifies the location of a face from image data.

[1395] "Audio feature extraction" is the process of extracting information such as pitch and frequency from audio data.

[1396] "Database linkage" refers to the exchange of information with a database that stores information on the products being proposed.

[1397] This invention is a system that analyzes the emotions of animals and suggests the most suitable related products based on the analysis. This system is realized using a smartphone, tablet, and server.

[1398] First, the user uses a smartphone or tablet to acquire video or image data of the animal. The user takes videos or images of their pet, simultaneously recording their own facial expressions and tone of voice. This data provides the foundation for analyzing the emotions of the animal and the user.

[1399] The device then temporarily stores the captured video or image, along with the user's facial expression and audio data. These data are then uploaded to a server via the Internet. The server checks the format and size of the data and converts it into an appropriate format if necessary.

[1400] The server preprocesses the received data. In the case of video data, the video is broken down into frames and saved as still images. The server also separates the audio data from the video and saves it. This allows the server to process the image and audio data separately, removing noise and normalizing them.

[1401] The server then extracts features of the animal and the user from the preprocessed data. For image data, a face detection algorithm (e.g., Haar) is used to detect the faces and body parts of the animal and the user, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[1402] The server analyzes the animal and user's emotions using a pre-trained generative AI model based on the extracted features. As a result, the animal and user's emotions are tagged. Then, based on the analyzed emotion data, it suggests related products (e.g., specific pet food or toys) that match the animal's emotion.

[1403] The results of these suggestions are sent to the device in an appropriate format, such as JSON, and the device displays the results on its user interface. The user then checks the products suggested by the system and makes a purchasing decision based on them. The user then inputs feedback about the effectiveness of the suggested products and the pet's reaction, which is then sent to the server. The server then retrains the analysis model based on this feedback, improving the accuracy of the next data analysis.

[1404] For example, if a user takes a video of their dog and the app analyzes it, and determines that the dog is "happy," the app will suggest pet food or toys with a "joy" theme. The user can purchase the suggested products and provide feedback on their pet's subsequent reactions, thereby improving the accuracy of the entire system.

[1405] Example prompts to input to a generative AI model:

[1406] Write a Python program that, given a video path, analyzes a pet's emotions and suggests the best pet food and toys. The program uses OpenCV to process the video and analyzes emotions using a Keras emotion recognition model.

[1407] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1408] Step 1:

[1409] Users use a smartphone or tablet to capture video or image data of animals. The input is the captured video or image data, and the output is data temporarily stored on the device. The user takes a video of their pet and simultaneously records their own facial expressions and tone of voice.

[1410] Step 2:

[1411] The device temporarily stores the captured video or image, along with the user's facial expression and audio data. It then uploads this data to a server via the Internet. The input is the temporarily stored video, image, and audio data, and the output is the data uploaded to the server. The device also checks the format and size of the data and converts it to an appropriate format if necessary.

[1412] Step 3:

[1413] The server preprocesses the received data. In the case of video data, it breaks the video down into frames and saves them as still images. It also separates the audio data from the video and saves it. The input is the uploaded video data, and the output is the image data broken down into frames and the separated audio data. The server also performs noise removal and normalization on these data.

[1414] Step 4:

[1415] The server extracts features of animals and users from the preprocessed data. The input is preprocessed image and audio data, and the output is the extracted features. For image data, a face detection algorithm (e.g., Haar) is used to detect the faces and body parts of animals and users, and extract features such as eye shape, mouth opening and closing, and ear position. For audio data, features are extracted by analyzing characteristics such as pitch, frequency spectrum, and sound duration.

[1416] Step 5:

[1417] The server uses the extracted features to analyze the emotions of the animal and the user using a pre-trained generative AI model. The input is the extracted features, and the output is the analyzed emotion data. The generative AI model is designed to identify basic emotions (happiness, anger, anxiety, excitement, etc.). Depending on the emotion data, tagging is performed.

[1418] Step 6:

[1419] Based on the analysis results, the server suggests related products (e.g., specific pet food or toys) that match the animal's emotions. The input is the analyzed emotion data, and the output is data about the suggested products. The server also sends these suggestion data to the device in an appropriate format, such as JSON.

[1420] Step 7:

[1421] The terminal displays the results on a user interface (UI) based on the recommendation data received from the server. The input is data about the recommended products, and the output is visualized data that is displayed to the user. The user reviews the recommended products and makes a purchasing decision based on their content.

[1422] Step 8:

[1423] The user inputs the effects of the proposed product and the pet's reaction as feedback and sends it to the server. The input is feedback information, and the output is feedback data stored on the server.

[1424] Step 9:

[1425] The server retrains the sentiment analysis model based on the received feedback. The input is the feedback data, and the output is an updated sentiment analysis model. This improves the accuracy of the next data analysis.

[1426] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1427] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1428] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1429] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1430] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1431] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1432] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1433] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1434] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1435] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1436] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1437] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1438] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1439] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1440] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1441] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1442] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1443] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1444] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1445] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1446] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1447] The following is further disclosed regarding the above embodiment.

[1448] (Claim 1)

[1449] a means for acquiring video or image data of the animal;

[1450] means for extracting animal facial expressions, gestures, and sounds from said data;

[1451] means for analyzing the emotion of the animal based on the extracted feature amount;

[1452] means for visualizing the analyzed emotion data in a format that is easy for humans to understand;

[1453] means for transmitting the visualized data to a terminal;

[1454] means for displaying the data in the terminal;

[1455] A system including:

[1456] (Claim 2)

[1457] 2. The system according to claim 1, further comprising means for performing noise removal and standardization as preprocessing of the data.

[1458] (Claim 3)

[1459] 2. The system according to claim 1, further comprising means for using a face detection and audio feature extraction algorithm as a machine learning algorithm for extracting the animal features.

[1460] (Claim 4)

[1461] 10. The system of claim 1, further comprising means for displaying the analyzed emotion data in visual icon or graph form.

[1462] (Claim 5)

[1463] 10. The system of claim 1, further comprising means for collecting feedback data from the user and using the feedback data to retrain the machine learning model.

[1464] "Example 1"

[1465] (Claim 1)

[1466] a means for acquiring video or image data of the animal;

[1467] means for uploading the data from the terminal to a server;

[1468] A means for decomposing video data into frames and storing audio data and image data separately;

[1469] means for denoising and normalising said data;

[1470] means for extracting animal facial expressions, gestures, and sounds from said data;

[1471] means for analyzing the emotion of the animal based on the extracted feature amount;

[1472] means for generating data that visualizes the analyzed emotion data;

[1473] means for transmitting the visualized data to a terminal;

[1474] means for displaying the data in the terminal;

[1475] A system including:

[1476] (Claim 2)

[1477] The system of claim 1, further comprising means for using a generative AI model as a machine learning algorithm for extracting the features of the animal.

[1478] (Claim 3)

[1479] 2. The system according to claim 1, further comprising means for performing noise removal and standardization as preprocessing of the data.

[1480] "Application Example 1"

[1481] (Claim 1)

[1482] a means for acquiring video or image data of the animal;

[1483] means for extracting animal facial expressions, gestures, and sounds from said data;

[1484] means for analyzing the emotion of the animal based on the extracted feature amount;

[1485] means for visualizing the analyzed emotion data in a format that is easy for humans to understand;

[1486] means for transmitting the visualized data to a terminal;

[1487] means for displaying the data in the terminal;

[1488] A way to communicate the emotions of animals to delivery staff during delivery work,

[1489] A system including:

[1490] (Claim 2)

[1491] 2. The system according to claim 1, further comprising means for performing noise removal and standardization as preprocessing of the data.

[1492] (Claim 3)

[1493] 2. The system according to claim 1, further comprising means for using a face detection and audio feature extraction algorithm as a machine learning algorithm for extracting the animal features.

[1494] "Example 2: Combining Emotion Engines"

[1495] (Claim 1)

[1496] a means for acquiring video or image data of the animal;

[1497] means for extracting animal facial expressions, gestures, and sounds from said data;

[1498] means for simultaneously acquiring facial expression and voice data of a user;

[1499] means for transmitting user and animal data to a server;

[1500] means for checking the format and size of said data and for pre-processing;

[1501] means for analyzing emotions of animals and users based on the extracted feature amounts;

[1502] means for visualizing the analyzed emotion data in a format that is easy for humans to understand;

[1503] means for transmitting the visualized data to a terminal;

[1504] means for displaying the data in the terminal;

[1505] means for obtaining and transmitting feedback from the user to the server;

[1506] means for retraining a machine learning model using the feedback;

[1507] A system including:

[1508] (Claim 2)

[1509] 2. The system according to claim 1, further comprising means for performing noise removal and standardization as preprocessing of the data.

[1510] (Claim 3)

[1511] The system according to claim 1, further comprising means for using a face detection and voice feature extraction algorithm as a machine learning algorithm for extracting the features of the animal and the user.

[1512] "Application example 2 when combining emotion engines"

[1513] (Claim 1)

[1514] a means for acquiring video or image data of the animal;

[1515] means for extracting animal facial expressions, gestures, and sounds from said data;

[1516] means for analyzing the emotion of the animal based on the extracted feature amount;

[1517] means for visualizing the analyzed emotion data in a format that is easy for humans to understand;

[1518] means for transmitting the visualized data to a terminal;

[1519] means for displaying the data in the terminal;

[1520] means for suggesting related products based on the analyzed emotion data;

[1521] a means for collecting user feedback on the product proposal results and using the feedback to retrain the analysis model;

[1522] A system including:

[1523] (Claim 2)

[1524] 2. The system according to claim 1, further comprising means for performing noise removal and standardization as preprocessing of the data.

[1525] (Claim 3)

[1526] 2. The system according to claim 1, further comprising means for using a face detection and audio feature extraction algorithm as a machine learning algorithm for extracting the animal features.

[1527] (Claim 4)

[1528] 2. The system according to claim 1, further comprising means for linking with a database of related products based on the product suggestion results. [Explanation of symbols]

[1529] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for acquiring video or image data of the animal; means for extracting animal facial expressions, gestures, and sounds from said data; means for analyzing the emotion of the animal based on the extracted feature amount; means for visualizing the analyzed emotion data in a format that is easy for humans to understand; means for transmitting the visualized data to a terminal; means for displaying the data in the terminal; A system including:

2. 2. The system according to claim 1, further comprising means for preprocessing the data by noise removal and standardization.

3. The system according to claim 1 , further comprising means for using a face detection and audio feature extraction algorithm as a machine learning algorithm for extracting the animal features.

4. 10. The system of claim 1, further comprising means for displaying the analyzed emotion data in visual icon or graph form.

5. The system of claim 1 , further comprising means for collecting feedback data from the user and using the feedback data to retrain the machine learning model.

Images