Animal language conversion method, apparatus, electronic device, storage medium, and program

Abstract

The present invention provides a method, apparatus, electronic device, storage medium, and program for converting animal language. [Solution] The method includes acquiring multimodal data relating to an animal. The multimodal data includes animal vocal data, animal behavior data, and animal vital data. The method also includes preprocessing the multimodal data to obtain fused multimodal data, recognizing the animal's current emotions based on the fused multimodal data to obtain an animal emotion recognition result, performing semantic mapping and language translation on the emotion recognition result to convert the animal language into human language and obtain a language conversion result.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as machine learning, deep learning, and natural language processing, and particularly relates to an animal language conversion method, device, electronic device, storage medium, and program.

Background Art

[0002] Current market technologies attempt to decode the emotional world of animals through some basic animal vocalization and behavior translation devices and pet emotion analysis tools using artificial intelligence image recognition technology. However, these methods are often limited to a superficial interpretation of animal behavior. These technical means cannot deeply penetrate into the complex emotional aspects of animals and cannot achieve deep and real-time emotional understanding and mutual communication between humans and animals.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The present disclosure provides an animal language conversion method, device, electronic device, storage medium, and program.

Means for Solving the Problems

[0004] According to one aspect of the present disclosure, an animal language conversion method is provided, and the method includes: acquiring multimodal data related to an animal, where the multimodal data includes animal voice data, animal behavior data, and animal vital data; preprocessing the multimodal data to obtain fused multimodal data; recognizing the current emotion of the animal based on the fused multimodal data to obtain an animal emotion recognition result; performing semantic mapping and language translation on the emotion recognition result to convert animal language into human language and obtain a language conversion result.

[0005] According to another aspect of this disclosure, an animal language conversion device is provided, the device is, An acquisition module for acquiring multimodal data relating to animals, wherein the multimodal data includes animal vocal data, animal behavior data, and animal vital data. A preprocessing module for preprocessing the aforementioned multimodal data and obtaining fused multimodal data, An emotion recognition module for recognizing the animal's current emotions based on the aforementioned fused multimodal data and obtaining the animal's emotion recognition result, The system includes a conversion module that performs semantic mapping and language translation on the emotion recognition results, converts animal language into human language, and obtains a language conversion result.

[0006] A third aspect of this disclosure provides an electronic device, which is At least one processor, The system comprises at least one processor and memory that is communicated with, The memory stores instructions that are executable by the at least one processor, and when such instructions are executed by the at least one processor, the at least one processor performs any one of the methods in the embodiments of the present disclosure.

[0007] A fourth aspect of the present disclosure provides a non-temporary computer-readable storage medium storing computer instructions for causing a computer to perform any one of the methods of the embodiments of the present disclosure.

[0008] A fifth aspect of this disclosure provides a program, when executed by a processor, for causing any of the embodiments of this disclosure to perform.

[0009] This disclosure provides an animal language conversion method, apparatus, device, and storage medium, which can achieve comprehensive capture and accurate recognition of animal emotions by acquiring multimodal data such as animal voice, behavior, and vital signs. Next, through semantic mapping and language translation technology, it significantly enhances human-animal communication by converting animal emotional states and intentions into human-understandable language, increasing the accuracy of understanding animal emotions and the real-time nature of interaction, and providing humans with a new method of communication with animals. In other words, this solution can accurately recognize an animal's current emotional state and convert it into human language, thereby realizing deeper emotional exchange and understanding between animals and humans, and increasing the accuracy and efficiency of interspecies communication.

[0010] It should be understood that the content contained herein is not intended to describe any key points or important features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Further details of other features of this disclosure are provided in the specification below.

[0011] The attached drawings are for the purpose of better understanding the solutions of this disclosure and do not constitute a limitation of this disclosure. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic diagram illustrating the steps of the animal language conversion method according to an embodiment of the present disclosure. [Figure 2] This is a flowchart illustrating the collection of multimodal data according to an embodiment of the present disclosure. [Figure 3] This is a flowchart illustrating the preprocessing of multimodal data according to an embodiment of the present disclosure. [Figure 4] This is a flowchart illustrating how to obtain the emotion recognition results of an animal according to the embodiments of this disclosure. [Figure 5] This is a flowchart illustrating the conversion of animal language to human language according to an embodiment of the present disclosure. [Figure 6]This is a flowchart illustrating how to update emotion labels according to an embodiment of the present disclosure. [Figure 7] This is a block diagram of the principle of an animal language conversion device according to an embodiment of the present disclosure. [Figure 8] This is a block diagram of an electronic device for realizing the animal language conversion method according to the embodiments of this disclosure. [Modes for carrying out the invention]

[0013] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the accompanying drawings. These drawings include various details of the embodiments of the present disclosure to aid understanding, and should be considered as illustrative only. Accordingly, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, well-known descriptions of functions and structures are omitted in the following description for clarity and brevity.

[0014] Currently, technologies related to human-animal communication on the market can be broadly divided into the following two types.

[0015] The first method involves using a simple translation device for animal sounds and behaviors, which primarily implements the translation function of animal emotions based on a "voiceprint database + simple algorithm." Typical examples include several commercially available pet behavior and voice recognition devices, which can capture animal sounds and some typical behaviors using relatively simple sensors, and can perform a simple mapping of emotions by matching them to a pre-built emotion library, for example, recognizing that a dog's bark expresses a request for food, anger, etc.

[0016] The second method is a pet emotion analysis tool based on AI (Artificial Intelligence) image recognition. That is, by introducing image processing and AI technology, it helps users understand the emotional reactions of pets. For example, some companies capture the faces and actions of animals in camera image data, combine them with existing deep learning training models, and analyze different expressions of animal faces to recognize specific expressions such as laughing, anger, frustration, confusion, etc.

[0017] The above two methods have some limitations in terms of animal emotion understanding and translation. First, the emotion translation is single, relying on preset voiceprint data and action classification, unable to continuously track emotional changes, lacking accuracy under complex scenes. Second, multi-modal fusion analysis is insufficient, overly relying on a single information source, restricting the comprehensiveness and accuracy of emotion translation. Also, timing detection is insufficient, lacking the ability to consistently track emotional states, and the perception of emotional changes becomes dull. Moreover, self-adaptive learning and tuning mechanisms are insufficient, making optimization and iteration difficult when facing unknown emotional patterns, restricting the flexible scalability of the system. Finally, edge computing cannot be performed, resulting in a lack of real-time performance, affecting the immediate communication between humans and animals. These limitations commonly lead to the deficiencies in realizing deep and real-time cross-species emotional communication in the prior art.

[0018] To solve at least one of the above technical problems, the present disclosure provides an animal emotion recognition method.

[0019] As shown in FIG. 1, FIG. 1 is a flowchart diagram of an animal language conversion method according to an embodiment of the present disclosure. This method can be applied to the server side and includes the following.

[0020] In step S101, multi-modal data related to an animal is acquired, and the multi-modal data includes animal voice data, animal behavior data, and animal vital data.

[0021] Specifically, acquiring multimodal data about animals means collecting different types of information through multiple sensors in order to fully understand the animal's state and emotions. Here, "animal vocal data" refers to various sounds emitted by animals captured via audio sensors, which can reflect the animal's emotions and needs; "animal behavioral data" refers to the movement and posture of the animal's limbs recorded via video cameras, which supports the analysis of its behavioral patterns and emotional expressions; and "animal vital data" refers to physiological indicators of the animal monitored by physiological sensors such as heart rate and body temperature, which can provide the physiological basis for the animal's emotional state.

[0022] By acquiring multimodal data from animals in this way, including sound, behavior, and symptom data, it is possible to achieve a comprehensive capture of animal emotions and behaviors, providing more accurate sentiment analysis and behavioral understanding, thereby strengthening communication and interaction between humans and animals, and improving animal welfare and humans' ability to respond to animal behavior.

[0023] In step S102, the multimodal data is preprocessed to obtain fused multimodal data.

[0024] Specifically, preprocessing multimodal data means performing a series of processing steps on raw data collected from different sources (e.g., audio, video, physiological sensors, etc.), which can include noise reduction, normalization, and feature extraction of multimodal data to facilitate analysis and understanding. On the other hand, "fused multimodal data" means integrating these preprocessed data into a single unified dataset, a process that involves temporal alignment (ensuring all data is temporally synchronized), feature fusion (merging features from different modalities into a single unified feature vector), and data synchronization (handling differences in timestamps and sampling rates of different data streams). Such preprocessing and fusion can provide a single, comprehensive data view, making subsequent sentiment recognition and language translation more accurate and efficient.

[0025] In this way, by preprocessing multimodal data such as animal sounds, behaviors, and vital signs, and then fusing this data into a single unified dataset, data consistency and availability can be improved, emotion recognition and language translation can be made more accurate, the system's overall understanding of animal behavior and emotional states can be enhanced, and the efficiency and effectiveness of interspecies communication can be increased.

[0026] In step S103, the current emotions of the animal are recognized based on the fused multimodal data, and the results of the animal's emotion recognition are obtained.

[0027] Specifically, after obtaining fused multimodal data, the system recognizes the animal's current emotions based on this fused data. Specifically, it uses a comprehensive dataset integrating information such as animal voice, behavior, and vital signs, and analyzes and determines the animal's emotional state using machine learning and deep learning techniques. Here, "emotion recognition" means recognizing the animal's emotional state, such as anxiety, excitement, or relaxation, by analyzing features extracted from this multimodal data. To obtain the animal emotion recognition result, for example, the fused data is input into a trained emotion recognition model. This model can output the animal emotion recognition result by comparing it with the features of known emotional states, thus providing a basis for subsequent language conversion and human-machine interaction. This animal emotion recognition model can be pre-trained individually for animal emotion recognition. The training process and the inference process for recognizing animal emotions are consistent, and the training of the animal emotion recognition model will be described later.

[0028] Thus, analyzing fused multimodal data, including animal vocalizations, behaviors, and physiological vitals, is advantageous for accurately recognizing an animal's current emotional state. This process not only improves the accuracy of understanding and responding to animal emotions but also strengthens human-animal communication, allowing humans to better decipher animal needs and emotions, thereby enhancing animal welfare and the intimacy of human-animal relationships.

[0029] In step S104, semantic mapping and language translation are performed on the emotion recognition results to convert animal language into human language and obtain the language conversion result.

[0030] Specifically, "performing semantic mapping and language translation on emotion recognition results" refers to the process of converting animal emotional states obtained by analyzing multimodal data, such as anxiety, excitement, or joy, into linguistic expressions that humans can understand. More specifically, this process involves using a pre-trained language model and deep learning techniques to establish a correspondence between animal emotional characteristics and corresponding expressions in human language, i.e., "semantic mapping." Subsequently, these mapping results are converted into concrete text or audio output to achieve "language translation." Finally, the "language conversion result" means that animal emotions and intentions are converted into a form of human language, allowing human users to understand the animal's "language" more intuitively, thereby enabling effective interspecies communication.

[0031] In this way, by performing semantic mapping and language translation on emotion recognition results, it is possible to convert animal nonverbal communication into a language that humans can understand. This process not only breaks down communication barriers between humans and animals, but also greatly improves humanity's understanding of animal emotions and needs, harmonizes human-animal interactions, and provides new perspectives and tools for animal welfare and behavioral research.

[0032] This disclosure provides an animal language conversion method, apparatus, device, and storage medium that can achieve comprehensive capture and accurate recognition of animal emotions by acquiring multimodal data such as animal voice, behavior, and vital signs. Next, through semantic mapping and language translation technology, it significantly enhances human-animal communication by converting animal emotional states and intentions into human-understandable language, increasing the accuracy of understanding animal emotions and the real-time nature of interaction, and providing humans with a new method of communication with animals. In other words, this solution can accurately recognize the current emotional state of an animal and convert it into human language, thereby realizing deeper emotional exchange and understanding between animals and humans, and increasing the accuracy and efficiency of interspecies communication.

[0033] In several selectable embodiments, obtaining multimodal data about animals is possible. Collecting sound wave information emitted by animals to obtain animal vocal data, To collect animal body language and movement changes and obtain animal behavior data, This includes collecting physical biometric indicators from animals and obtaining animal vital data.

[0034] Specifically, this system captures animal vocalizations through audio collectors, collecting sound wave information to obtain "animal vocal data." Simultaneously, it collects animal body language and movement changes via visual sensors such as video cameras to form "animal behavior data," which reflects the animal's activity and nonverbal behavior. Furthermore, it monitors "animal vital data" such as heart rate and body temperature using physiological sensors, and these physical biometric indicators provide important information for understanding the animal's physiological state and emotions. By integrating this multimodal data, it is possible to comprehensively capture the animal's communication methods and emotional state, laying the foundation for further data analysis and emotion recognition.

[0035] To facilitate understanding of the solutions of the embodiments of this disclosure, refer, for example, to Figure 2, which is a flowchart illustrating the collection of multimodal data according to the embodiments of this disclosure. First, an audio collector is used to capture animal sounds in real time, and the audio collector then transmits the captured audio data to a data processing module, through which noise processing can be performed using corresponding audio frequency filters to reduce background interference. For example, for dog barks, dimensional information such as pitch, amplitude, time, frequency, and breakpoint changes are sampled together to capture all information about the bark.

[0036] Regarding the collection of animal behavioral data, the limb movements and behavioral expressions of animals are acquired through imaging devices (such as high-definition cameras and infrared cameras), and postural analysis such as tail wagging, jumping, and lying on one side can be performed, and specific physical characteristics of the animal (such as the erection of ear rings and pupil dilation) can be captured. Next, the captured video / image data is transmitted to a data processing module.

[0037] The collection of animal vital data can be achieved by collecting heart rate and body temperature using high-precision contact or non-contact temperature detection sensors, and then transmitting the collected vital data to a data processing module.

[0038] In this way, by collecting sound wave information, limb language and movement changes, and physical biometric indicators emitted by animals, relatively comprehensive animal vocal, behavioral, and vital data can be collected. The integration of this multidimensional information provides a rich and accurate data base for a deep understanding and analysis of animal emotions and behavior, thereby enabling humans to more accurately decipher animals' communication intentions and physiological states, strengthening nonverbal communication between humans and animals, improving the effectiveness of animal nursing and training, and opening up new avenues for animal health monitoring and behavioral research.

[0039] In several selectable embodiments, preprocessing multimodal data to obtain fused multimodal data is possible. The process involves denoising multimodal data to clean it and obtain the cleaned multimodal data. The process involves normalizing the cleaned multimodal data to obtain normalized multimodal data, and This includes performing timing alignment and fusion processing on normalized multimodal data to obtain fused multimodal data.

[0040] Specifically, denoising and cleaning multimodal data means using signal processing techniques to remove noise and interference from audio and video data, thereby improving data quality and obtaining cleaned multimodal data. Next, "normalizing the cleaned multimodal data" means converting data from different sources and scales into a unified format or scale so that machine learning models can process it more effectively. This step results in normalized multimodal data. Finally, "performing timing alignment and fusion processing on the normalized multimodal data" refers to temporally aligning data from different modalities and merging them into a single unified dataset. This step ensures the temporal consistency of the data, integrates information from different sensors, and ultimately results in fused multimodal data, providing an accurate and comprehensive data foundation for subsequent sentiment recognition and behavioral analysis.

[0041] To facilitate understanding of the solutions in the embodiments of this disclosure, refer, for example, to Figure 3, which is a flowchart illustrating the preprocessing of multimodal data according to one embodiment of this disclosure. First, the audio data, image data, and body temperature and heart rate data in the multimodal data are preprocessed and normalized, respectively. The data processing includes noise reduction and cleaning to process invalid portions in the audio and visual information. For example, noise that humans may produce (wind noise, speech, etc.) is filtered out to make the audio signal easier to recognize. In images, for example, background movement and objects in video frames are sorted out to maintain only key information such as changes in animal behavior. Next, the data is normalized. Whether it is the audio signal, video, or vital information, it is necessary to normalize it, convert it to a unified standard, and represent it as a feature vector that can be processed by a machine learning algorithm.

[0042] Finally, regarding timing alignment for multimodal data and fused data, data alignment is a prerequisite for data fusion and must resolve temporal and spatial differences between signal acquisitions from different modalities. For example, a single dog barking event often has a corresponding time difference with the time of limb movement or body temperature fluctuations. These data need to be timing-calibrated before input so that the multimodal input in the translation task has a consistent reference point.

[0043] By applying noise reduction, normalization, timing alignment, and fusion processing to multimodal data in this way, the quality, consistency, and availability of the data can be significantly improved, and the features extracted from animal voices, behaviors, and vital signs can be made more accurate and reliable. This process not only enhances the accuracy of data analysis but also improves the accuracy of the model's recognition of animal emotions and behaviors, providing a solid data foundation for efficient and accurate interspecies communication.

[0044] In several selectable embodiments, recognizing an animal's current emotion based on fused multimodal data and obtaining the animal's emotion recognition result is possible. Using a deep learning model, we perform speech feature extraction, visual / action feature extraction, and vital sign change analysis on fused multimodal data to obtain multimodal feature vectors. This includes using a generative adversarial network to analyze the sentiment of multimodal features and obtain the result of animal emotion recognition.

[0045] Specifically, "performing speech feature extraction, visual / behavioral feature extraction, and vital sign change analysis on fused multimodal data using deep learning models" refers to processing and analyzing multimodal datasets that integrate speech, visual, and physiological data using advanced deep learning techniques, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). This process involves extracting speech features such as tone, rhythm, and volume from audio signals, visual / behavioral features such as posture and behavioral patterns from video data, and analyzing vital sign changes such as heart rate and body temperature fluctuations from physiological data. These features can collectively form a single multimodal feature vector that comprehensively reflects the physiological and behavioral state of the animal. Next, "performing sentiment analysis on multimodal features using generative adversarial networks" means using deep learning frameworks such as Generative Adversarial Networks (GANs) to enable the network to recognize and distinguish different emotional states through an adversarial training process. This process ultimately produces results in recognizing animal emotions, translating the animal's complex emotions and behaviors into understandable emotion labels, providing a basis for further semantic mapping and linguistic translation.

[0046] To facilitate understanding of the solutions of the embodiments of this disclosure, refer, for example, to Figure 4, which is a flowchart illustrating how to obtain animal emotion recognition results according to one embodiment of this disclosure. After obtaining fused multimodal data, fine-grained feature extraction is performed on the data of each modality via a deep learning model, specifically extracting vocal features, visual behavior features, and vital change analysis via the deep learning model to obtain feature vector combinations. Next, based on emotion recognition, a large-scale generative adversarial network (GAN) model is used to analyze voiceprint features, behavior changes, and vital changes in the data to obtain emotion classification labels. For example, if the model detects that an animal's low-frequency barking is accompanied by limb tension and pupil dilation, the model recognizes that the animal is in a highly alert state through contrasting reasoning and further infers potentially underlying psychological activity (fear or embarrassment) through feature sequencing.

[0047] In this way, by comprehensively analyzing fused multimodal data using deep learning models, it is possible to accurately extract features such as animal vocalizations, visual behaviors, and vital changes, form multimodal feature vectors, and then perform deep sentiment analysis on these features using generative adversarial networks to obtain results in recognizing animal emotions. This process not only enhances the accuracy and depth of emotion recognition but also strengthens our understanding of animal behavior and psychological states, providing powerful technical support for more effective human-animal communication.

[0048] In several selectable embodiments, semantic mapping and language translation are performed on the emotion recognition results, and animal language is converted into human language to obtain the language conversion result. This involves extracting emotion labels and speech features from emotion recognition results, and converting the speech features into standardized speech vectors. Using a pre-trained language model, we can semantically map emotion labels to speech vectors and obtain emotional intent. This includes using a language generator to translate emotional intentions into language, generating a corresponding human language, and obtaining the language conversion result.

[0049] Specifically, a "pre-trained language model" refers to an artificial intelligence model trained on large amounts of data that can understand and process natural language. This model is used to recognize the emotional intentions of animals by performing a "semantic mapping" of animal emotion labels, i.e., the emotional states of animals obtained from an emotion recognition module, with speech vectors, thereby corresponding the animal's speech features with emotional meanings in human language. Next, a "language generator" is the process of converting the nonverbal communication of animals into language that humans can understand, based on these emotional intentions. This process involves converting the emotions and intentions of animals into concrete text or speech output, i.e., "language conversion results," enabling humans to intuitively understand the "language" of animals and realize effective communication between humans and animals.

[0050] To facilitate understanding of the solutions in the embodiments of this disclosure, refer, for example, to Figure 5, which is a flowchart illustrating the conversion of animal language to human language in an embodiment of this disclosure. After the emotion recognition module obtains the emotion recognition result, it provides the animal's emotion label and vocal features, and then transmits this information to the vocal mapping module, which is responsible for converting the animal's vocal features into human-understandable vocal representations. Next, the human semantic output module receives the converted vocal information and outputs it as a language conversion result, finally presenting these results to the user to achieve real-time conversion from animal language to human language and emotional communication. The entire process relates to emotion recognition, feature extraction and mapping, and human language generation, and aims to facilitate effective communication between humans and animals.

[0051] In this way, by using a pre-trained language model to perform semantic mapping between emotion labels and speech vectors, it is possible to accurately capture and understand the emotional intentions of animals, and then use a language generator to convert these emotional intentions into human language. This process not only enables accurate decoding of animal emotions, but also translates animals' nonverbal communication into human-understandable language, greatly promoting communication and understanding between humans and animals, and increasing the transparency of animal emotional expression and the efficiency of interaction.

[0052] In some selectable embodiments, the method further If specific audio data is detected and no emotion matching history exists, label the specific audio data and obtain an updated emotion label. This includes dynamically updating sample data based on updated sentiment labels in order to adjust model parameters based on updated sample data.

[0053] Specifically, if specific animal vocal data is detected and this vocal data does not exist in the historical emotion mapping record, i.e., if it does not correspond to a previous emotion label, a labeling process is triggered. Here, "labeling" means describing the emotional state when the animal makes the vocalization and artificially assigning an emotion label to these specific vocal data in order to "obtain an updated emotion label." Subsequently, this newly labeled emotion label is incorporated into the sample database, enabling "dynamic updating of sample data based on the updated emotion label." This updated sample data is used to adjust the model parameters, i.e., "adjusting the model parameters based on the updated sample data," thereby optimizing and improving the model's emotion recognition ability for newly appearing vocal data, ensuring that the system can adapt to new or unfamiliar animal vocalizations, and enhancing recognition accuracy and the system's self-adaptive learning ability.

[0054] To facilitate understanding of the solutions in the embodiments of this disclosure, for example, when the system encounters an undeterminable vocalization pattern or anomaly, it prompts the user to input relevant labels or recognition information. For example, if the system detects a particular action + sound combination and there is no matching history for an obvious emotion expression, the user can annotate this sound through the interface, e.g., "help" or "hungry." After the user manually labels it, the system adjusts the data parameters of the label corresponding to the current audio and action and updates the weights of the current model. Once the user has finished labeling, the system updates the dataset in a timely manner, continuously optimizing and retraining the relevant generative model so that the system gradually improves its recognition rate of similar or new categories of emotion events.

[0055] In this way, by detecting specific voice data and artificially labeling it when there is insufficient emotion matching history, and updating the emotion label based on that, the sample database can be continuously expanded and enriched, and model parameters can be dynamically updated to enhance the system's emotion recognition ability for new voice data, increasing the model's adaptability and accuracy, and ensuring that the interspecies communication system continuously evolves to better understand and respond to the communication intentions of animals.

[0056] In some selectable embodiments, the method further Collect multimodal data within a preset time window to obtain data on the animal's emotional changes, To extract features from emotional change data and obtain emotional change characteristics, This includes updating the sentiment label based on the difference between the sentiment change characteristics of the current time window and the sentiment change characteristics of the previous time window.

[0057] Specifically, a "preset time window" refers to a specific period set up for analyzing changes in animal emotions. "Multimodal data collection" within this period includes information such as animal vocalizations, behaviors, and vital signs to gather data on the animal's emotional state. Next, through a "feature extraction" process, key information that can represent changes in animal emotions—namely, "emotional change features"—is recognized from this data. Then, the differences between the emotional features extracted in the current time window and those from the previous time window are compared. If these differences indicate a significant change in the animal's emotional state, the "emotional label" is updated based on that change to reflect the animal's latest emotional state. This process is related to continuous monitoring and dynamic analysis of the animal's emotional state, ensuring the real-time nature and accuracy of emotion recognition.

[0058] In this way, multimodal data of animals is collected within a preset time window to comprehensively capture the emotional changes of animals, and these data are feature-extracted to recognize the characteristics of emotional changes. Then, by dynamically updating the emotional labels based on the difference between the current and previous time window's emotional characteristics, accurate real-time monitoring and response to the animal's emotional state are achieved. This not only improves the accuracy of emotion recognition but also enhances the real-time nature and depth of human-animal communication, allowing humans to better understand and respond to the emotional needs of animals.

[0059] In several selectable embodiments, collecting multimodal data within a preset time window and obtaining data on changes in animal emotions is possible. Collecting multimodal data within a preset time window, This includes inputting multimodal data into an emotion period recognition model and obtaining emotion change data from animals using the emotion period recognition model.

[0060] Specifically, a "preset time window" refers to a specific period set up to monitor and analyze changes in animal emotions. During this period, "multimodal data" is collected, including different types of information such as animal vocalizations, behaviors, and vital signs. This data is then fed into an "emotional period recognition model," which is an algorithm specifically designed to process and analyze time-series data, such as a long-term short-term memory network (LSTM) or gated regressive units (GRU). The model analyzes this multimodal data, recognizes and extracts features related to the animal's emotional state, and thus "obtains animal emotional change data." This data reflects the animal's emotional state within a continuous time window and provides a basis for further emotion recognition and label updates. In this way, the solution can achieve dynamic tracking and accurate recognition of animal emotional states.

[0061] In this way, by collecting multimodal data of animals within a preset time window and inputting this data into an emotion period recognition model, accurate capture and analysis of animal emotion changes can be achieved. This method allows for the recognition and understanding of the emotional dynamics of animals over a continuous period, thereby providing richer and more accurate emotion change data. This data not only enhances the depth and real-time nature of emotion recognition but also improves the quality and efficiency of communication between humans and animals, enabling humans to respond more precisely and in a timely manner to the emotional needs and behavioral changes of animals.

[0062] In several selectable embodiments, updating the sentiment label based on the difference between the sentiment change characteristics of the current time window and the sentiment change characteristics of the previous time window is possible. The Euclidean distance difference between the emotion change features of the current time window and the emotion change features of the previous time window is calculated to obtain the emotion difference. This includes upgrading the emotion label and obtaining an updated emotion label if the emotion difference exceeds a preset emotion difference threshold.

[0063] Specifically, "emotional change features in the current time window" refers to emotion-related features extracted from multimodal data of animals over a specific period, while "emotional change features in the previous time window" refers to features corresponding to the previous period. By calculating the "Euclidean distance difference" between the emotional features of the two time windows, that is, by measuring the distance between two points in a multidimensional space, a single quantitative "emotional difference" can be obtained.

[0064] The Euclidean distance difference between the emotion change feature in the current time window and the emotion change feature in the previous time window satisfies the equation [D(W_i,W_{i-1})=\sqrt{\sum_{n=1}^{N}(x_{i,n}-x_{i-1,n})^2}], where D(W_i,W_{i-1}): represents the Euclidean distance between the emotion change features of the i-th time window and the (i-1)-th time window, used to quantify the degree of change in emotion features within two consecutive time windows. x_{i,n} is a value representing the nth feature within the i-th time window, which may include pitch, rhythm, and amplitude of speech, frequency of behavior, and changes in vital signs such as heart rate. x_{i-1,n} represents the value of the nth feature within the (i-1)-th time window. sum represents the addition operation, and sqrt represents the operation of calculating the squared value.

[0065] By calculating the Euclidean distance difference between the emotion change features of the current time window and the emotion change features of the previous time window, an emotion difference is obtained that reflects the degree of change in the animal's emotional state within two consecutive time windows. If this difference exceeds a "preset emotion difference threshold," i.e., a single predetermined limit, indicating a significant change in the animal's emotional state, an "emotional label upgrade" is triggered. Based on the emotion difference and change pattern, the emotion label is updated from one state (e.g., "vigilant") to a label more appropriate for the other current emotional state (e.g., "anxious"), resulting in an "updated emotion label." This process makes emotion recognition more dynamic and accurate, allowing for real-time responses to significant changes in animal emotions.

[0066] In this way, by calculating the Euclidean distance difference between the emotion change features of the current time window and the emotion change features of the previous time window, the degree of change in the animal's emotional state can be quantified, and the emotion difference can be obtained. When the emotion difference exceeds a preset threshold, the emotion label can be automatically upgraded to obtain an updated emotion label. This method makes emotion recognition more sensitive and accurate, allowing for the capture and response to significant changes in animal emotions in real time, thereby improving the quality and efficiency of communication between humans and animals, and ensuring that humans can understand and respond to the emotional needs of animals in a timely and appropriate manner.

[0067] In some selectable embodiments, the method further Evaluate the emotion weights corresponding to each time window and obtain an emotion weight score. This includes accumulating sentiment weight scores corresponding to similar sentiment time windows over consecutive periods, and updating the sentiment label if the accumulated result is greater than a set upgrade threshold.

[0068] Specifically, the "emotional weight score" refers to the process of quantitatively evaluating the emotional state of an animal within each time window, where each time window contains multimodal data collected over a specific period. By analyzing this data, emotional features are extracted, different weights are assigned based on the intensity or prominence of the features, and then the emotional weight score for each time window is calculated. Next, the weight scores of time windows with similar emotional features over consecutive periods are accumulated. If this accumulated score exceeds a "set upgrade threshold," a predefined number used to determine whether the emotional state is sufficiently prominent to trigger a label update, the emotional label is updated to reflect the change in the animal's emotional state. This process makes emotion recognition more dynamic and accurate, allowing for real-time responses to continuous changes in animal emotions.

[0069] In this way, by weighting and evaluating emotional states within each time window, this embodiment can quantify and track emotional changes in animals over consecutive periods. Weight scores of time windows with similar emotional characteristics are accumulated, and if the accumulated result exceeds a preset upgrade threshold, the emotional label is automatically updated to more accurately reflect the animal's current emotional state. This method enhances the sensitivity and adaptability of emotion recognition, making human-animal communication more accurate and timely, thereby improving the ability to understand and respond to the animal's emotional needs.

[0070] To facilitate understanding of the embodiments of the present disclosure, Figure 6 shows a flowchart illustrating the updating of emotion labels in an embodiment of the present disclosure. Animal bark audio and body temperature data are collected through voice collection and a body temperature sensor. This data is then sent to a data processing and fusion module for analysis to recognize audio features and changes in body temperature. The analysis results are used for emotion feedback, dynamically adjusting the animal's emotion label based on the emotional meaning of the audio and body temperature changes, for example, upgrading the "alert" state to "anxious." This process involves real-time monitoring and continuous emotional state evaluation to ensure that changes in the animal's emotion are reflected in the label update in a timely manner, thereby improving the accuracy of understanding and responding to the animal's emotional state.

[0071] In this way, by collecting animal emotional change data within a continuous time window and extracting emotional change features from this data, subtle changes in animal emotions can be monitored and quantified in real time. By comparing the differences between these change features and baseline emotional features, and updating the emotion label when the amount of change exceeds a preset threshold, dynamic changes in animal emotions can be captured more accurately. Furthermore, continuous tracking of animal emotional states and real-time responses are achieved, improving the sensitivity and accuracy of emotion recognition and enhancing the effectiveness of emotional communication between humans and animals.

[0072] Below are examples of the apparatus of the present application that can be used to implement the animal emotion recognition method described in the above-described embodiments of the present application. For details not disclosed in the embodiments of the apparatus of the present application, refer to the above-described embodiments of the animal emotion recognition method of the present application.

[0073] Based on a similar technical concept, this disclosure also provides an animal language translation device 700, which, as shown in Figure 7, An acquisition module 701 for acquiring multimodal data relating to animals, wherein the multimodal data includes animal vocal data, animal behavior data, and animal vital data, and the acquisition module 701... A preprocessing module 702 for preprocessing multimodal data and obtaining fused multimodal data, An emotion recognition module 703 for recognizing the current emotions of an animal based on fused multimodal data and obtaining the animal's emotion recognition result, The system includes a conversion module 704 that performs semantic mapping and language translation on emotion recognition results, converts animal language into human language, and obtains a language conversion result.

[0074] In several selectable embodiments, the acquisition module 701 acquires multimodal data about animals.

[0075] Collecting sound wave information emitted by animals to obtain animal vocal data, To collect animal body language and movement changes and obtain animal behavior data, This includes collecting physical biometric indicators from animals and obtaining animal vital data.

[0076] In several selectable embodiments, the preprocessing module 702 preprocesses the multimodal data to obtain fused multimodal data. The process involves denoising multimodal data to clean it and obtain the cleaned multimodal data. The process involves normalizing the cleaned multimodal data to obtain normalized multimodal data, and This includes performing timing alignment and fusion processing on normalized multimodal data to obtain fused multimodal data.

[0077] In several selectable embodiments, the emotion recognition module 703 recognizes the animal's current emotion based on fused multimodal data and obtains the animal's emotion recognition result. Using a deep learning model, we perform speech feature extraction, visual / action feature extraction, and vital sign change analysis on fused multimodal data to obtain multimodal feature vectors. This includes using a generative adversarial network to analyze the sentiment of multimodal features and obtain the result of animal emotion recognition.

[0078] In several selectable embodiments, the conversion module 704 performs semantic mapping and language translation on the emotion recognition result, converting animal language to human language and obtaining a language conversion result. This involves extracting emotion labels and speech features from emotion recognition results, and converting the speech features into standardized speech vectors. Using a pre-trained language model, we can semantically map emotion labels to speech vectors and obtain emotional intent. This includes using a language generator to translate emotional intentions into language, generating a corresponding human language, and obtaining the language conversion result.

[0079] In some selectable embodiments, the device further comprises a first update module used to label specific voice data when specific voice data is detected and no emotion matching history exists, to obtain an updated emotion label, and to dynamically update sample data based on the updated emotion label in order to adjust model parameters based on the updated sample data.

[0080] In some selectable embodiments, the apparatus further comprises a second update module used to collect multimodal data within a preset time window to obtain animal emotion change data, to feature extract the emotion change data to obtain emotion change features, and to update emotion labels based on the difference between the emotion change features of the current time window and the emotion change features of the previous time window.

[0081] In several selectable embodiments, the second update module collects multimodal data within a preset time window and obtains animal emotion change data. Collecting multimodal data within a preset time window, This includes inputting multimodal data into an emotion period recognition model and obtaining emotion change data from animals using the emotion period recognition model.

[0082] In several selectable embodiments, the second update module updates the sentiment label based on the difference between the sentiment change features of the current time window and the sentiment change features of the previous time window. The Euclidean distance difference between the emotion change features of the current time window and the emotion change features of the previous time window is calculated to obtain the emotion difference. This includes upgrading the emotion label and obtaining an updated emotion label if the emotion difference exceeds a preset emotion difference threshold.

[0083] In some selectable embodiments, the second update module is: Evaluate the emotion weights corresponding to each time window and obtain an emotion weight score. This is used to accumulate sentiment weight scores corresponding to similar sentiment time windows over consecutive periods, and to update the sentiment label if the accumulated result exceeds a set upgrade threshold.

[0084] The technical solutions disclosed herein ensure that the acquisition, storage, and application of users' personal information comply with applicable laws and regulations and do not violate public order or morality.

[0085] According to embodiments of the present disclosure, the present disclosure further provides electronic devices, non-temporary computer-readable storage media, and program products.

[0086] Figure 8 shows a schematic block diagram showing how an electronic device 800 according to an embodiment of the present disclosure can be implemented. An electronic device refers to any form of digital computer, such as a laptop computer, desktop computer, workstation, personal digital assistant, server, blade server, mainframe computer, and other compatible computers. An electronic device further refers to any form of mobile device, such as a personal digital assistant, cellular phone, intelligent phone, wearable device, and other similar computer equipment. The components, their connections, and functions described in this disclosure are illustrative and do not limit the implementation of anything described or specified in this disclosure.

[0087] As shown in Figure 8, device 800 includes a computing unit 801 that can perform various appropriate operations and processes based on computer program instructions stored in read-only memory (ROM) 802 or computer program instructions loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 can further store various programs and data necessary for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are connected to each other via bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0088] Multiple components in device 800 are connected to an I / O interface 805, which includes an input unit 806 such as a keyboard and mouse, an output unit 807 such as various displays and speakers, a storage unit 808 such as a magnetic disk or optical disk, and a communication unit 809 such as a network card, modem, or wireless communication transceiver. The communication unit 809 allows device 800 to exchange information / data with other devices via computer networks such as the Internet and / or various carrier networks.

[0089] The computing unit 801 may be a variety of general-purpose and / or dedicated processing components having processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, a computing unit that executes various machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs each of the methods described above, for example, the animal language translation method. For example, in some embodiments, the animal language translation method can be implemented as a computer software program tangibly contained in a machine-readable medium such as a memory unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed into device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the detection method described above can be performed. In addition, in other embodiments, the computing unit 801 may be configured to perform the animal language translation method by any other suitable method (e.g., firmware).

[0090] Various embodiments of the systems or technologies described in this disclosure can be implemented by digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standards (ASSPs), systems-on-a-chip (SOCs), complex-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. Each of these embodiments may be implemented by one or more computer programs that run and / or interpret on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0091] Program code for performing the methods of this disclosure can be written in any combination of one or more programming languages. These program codes are provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programming data processing device, so that when the program code is executed by the processor or controller, it can perform the functions / operations defined in the flowcharts and / or block diagrams. The program code may run entirely in a mainscan, partially in a mainscan, partially as an independent soft encapsulation and partially in a remote mainscan, or entirely in a remote mainscan or server.

[0092] In this disclosure, machine-readable media may be tangible media containing or storing programs used by or in conjunction with instruction execution systems, devices, or equipment. Machine-readable media may be machine-readable signal media or machine-readable storage media. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any suitable combination of the contents described above. Further specific examples of machine-readable storage media include one or more wired electrical connections, portable computer disk cartridges, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any combination of the contents described above.

[0093] To provide user interaction, a computer may implement the systems and technologies described herein, which may include a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor), a keyboard and pointing device for the user to provide input to the computer (e.g., a mouse or trackball). Other types of devices may also be used to provide user interaction; for example, the feedback provided to the user may be any form of sensor feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be accepted in any form (e.g., acoustic input, voice input, tactile input).

[0094] The systems and technologies described herein can be implemented in computing systems that include background components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include front-end components (e.g., user computers having a graphics user interface or network browser, through which users can interact with embodiments of the systems and technologies described herein), or in any combination of such background components, middleware components, or front-end components. Components of the system can be connected to one another via digital data communication in any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0095] A computer system can include a client and a server. Typically, the client and server are geographically separated and interact via a communication network. The client-server relationship is created by a computer program that operates on the corresponding computer. The server may be a cloud server, a server in a distributed system, or a server incorporating blockchain technology, etc.

[0096] It should be understood that steps can be newly ranked, added, or deleted using the various forms of flows shown above. For example, each step described in this disclosure may be executed in parallel, sequentially, or in a different order. This disclosure is not limited to this, as long as the technical solutions disclosed herein can achieve the desired results.

[0097] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, subcombinations, and substitutions are possible due to design considerations and other factors. Any changes, equivalent substitutions, and improvements within the gist and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for converting to animal language, The acquisition of multimodal data relating to animals, wherein the multimodal data includes animal vocal data, animal behavioral data, and animal vital data. The aforementioned multimodal data is preprocessed to obtain fused multimodal data, Based on the aforementioned fused multimodal data, the current emotions of the animal are recognized, and the results of the animal's emotion recognition are obtained. This includes performing semantic mapping and language translation on the aforementioned emotion recognition results, converting animal language into human language, and obtaining language conversion results. A method for translating animal languages.

2. Obtaining multimodal data on the aforementioned animals is To collect sound wave information emitted by the animal and obtain animal sound data, To collect the limb language and movement changes of the aforementioned animals and obtain the aforementioned animal behavior data, This includes collecting physical biomarkers of the animals and obtaining vital data of the animals. The method for converting animal language according to claim 1.

3. Preprocessing the aforementioned multimodal data to obtain fused multimodal data is, The aforementioned multimodal data is subjected to noise reduction processing to perform data cleaning, and the cleaned multimodal data is obtained. The cleaned multimodal data is normalized to obtain normalized multimodal data. This includes performing timing alignment and fusion processing on the normalized multimodal data to obtain fused multimodal data. The method for converting animal language according to claim 1.

4. Recognizing the animal's current emotions based on the aforementioned fused multimodal data and obtaining the result of recognizing the animal's emotions is possible. Using a deep learning model, speech feature extraction, visual / action feature extraction, and vital sign change analysis are performed on the fused multimodal data to obtain multimodal feature vectors. This includes performing sentiment analysis on the multimodal features using a generative adversarial network to obtain the result of the animal's emotion recognition. The method for converting animal language according to claim 1.

5. Performing semantic mapping and language translation on the aforementioned emotion recognition results, converting animal language into human language, and obtaining the language conversion result is, The process involves extracting emotion labels and speech features from the emotion recognition results, and converting the speech features into standardized speech vectors. Using a pre-trained language model, the emotional labels and the speech vectors are semantically mapped to obtain the emotional intent. This includes translating the aforementioned emotional intentions into language using a language generator, generating a corresponding human language, and obtaining a language conversion result. The method for converting animal language according to claim 1.

6. The aforementioned animal language conversion method further includes: If specific audio data is detected and no emotion matching history exists, the specific audio data is labeled to obtain an updated emotion label. This includes dynamically updating the sample data based on the updated sentiment labels in order to adjust the model parameters based on the updated sample data, The method for converting animal language according to claim 1.

7. The aforementioned animal language conversion method further includes: Collect multimodal data within a preset time window to obtain data on the animal's emotional changes, The aforementioned emotional change data is feature-extracted to obtain emotional change features, This includes updating the sentiment label based on the difference between the sentiment change characteristics of the current time window and the sentiment change characteristics of the previous time window. The method for converting animal language according to claim 1.

8. Collecting multimodal data within the aforementioned preset time window and obtaining data on the animal's emotional changes is, Collecting multimodal data within a preset time window, This includes inputting the multimodal data into an emotion period recognition model and obtaining emotion change data of animals using the emotion period recognition model. The method for converting animal language according to claim 7.

9. Updating the sentiment label based on the difference between the sentiment change characteristics of the current time window and the sentiment change characteristics of the previous time window is: The Euclidean distance difference between the emotion change features of the current time window and the emotion change features of the previous time window is calculated to obtain the emotion difference. If the aforementioned emotion difference exceeds a preset emotion difference threshold, the emotion label is upgraded and an updated emotion label is obtained, including: The method for converting animal language according to claim 7.

10. The aforementioned animal language conversion method further includes: Evaluate the emotion weights corresponding to each time window and obtain an emotion weight score. This includes accumulating sentiment weight scores corresponding to similar sentiment time windows over consecutive periods, and updating the sentiment label if the accumulated result is greater than a set upgrade threshold. The method for converting animal language according to claim 9.

11. It is an animal language conversion device, An acquisition module for acquiring multimodal data relating to animals, wherein the multimodal data includes animal sound data, animal behavior data, and animal vital data. A preprocessing module for preprocessing the aforementioned multimodal data and obtaining fused multimodal data, An emotion recognition module for recognizing the animal's current emotions based on the aforementioned fused multimodal data and obtaining the animal's emotion recognition result, The system includes a conversion module that performs semantic mapping and language translation on the emotion recognition results, converts animal language into human language, and obtains a language conversion result. Animal language translation device.

12. At least one processor, The system comprises at least one processor and a memory that is communicated with by it, The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor performs the method according to any one of claims 1 to 10. Electronic devices.

13. A non-temporary computer-readable storage medium storing computer instructions that cause a computer to perform the method described in any one of claims 1 to 10.

14. A program, when executed by a processor in a computer, for implementing the method according to any one of claims 1 to 10.

Images