A method and system for cross-species bidirectional translation of human pets based on multi-modal data

Abstract

The present application relates to the technical field of natural language processing, and more particularly to a pet cross-species bidirectional translation method and system based on multi-modal data. The method comprises: using a pre-trained discriminant model to classify the behavior of pet-side multi-modal signals, and converting the classification results into human language; synchronously extracting emotional feature vectors in human-side multi-modal signals, and analyzing them into acoustic modulation parameters; calling a generation model to modulate biological acoustic carriers, and outputting induced speech with emotional features; monitoring the actual behavior of pets after induction in real time, and retrieving corresponding predicted behavior reactions from a cross-species behavior correlation library; calculating the geometric similarity of the predicted and actual behaviors through a reinforcement learning algorithm, and using the geometric similarity as a reward value to synchronously iterate and optimize the discriminant model recognition weight and the generation model synthesis parameter. The present application overcomes the defects of fixed recognition benchmark, lack of emotional expression and inability to adapt to individuals in the prior art, and improves the intention recognition accuracy and the effectiveness of interaction guidance.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method and system for bidirectional human-pet cross-species translation based on multimodal data. Background Technology

[0002] With the rapid development of the pet economy, intelligent human-pet interaction has become a research hotspot in the field of cross-modal information processing. Currently, pet behavior translation systems, such as Trainini, mostly employ a one-way mapping logic. This involves collecting audio or video signals from the pet and using a pre-trained model to convert them into human language labels. While this approach achieves high accuracy in one-way translation from pet language to human language, the one-way nature of its model architecture prevents humans from conveying feedback information to pets in a manner consistent with the biological characteristics of the species.

[0003] On the other hand, existing interactive feedback methods mostly remain at the simple audio playback stage, lacking in-depth analysis of the real-time interactive intentions and emotional intensity of the human side. The induction signals output by traditional systems are often pre-recorded waveforms with fixed acoustic parameters. Due to the lack of emotional envelope characteristics, they are difficult to resonate effectively with the pet's auditory neural network, and long-term use can easily lead to biological fatigue in pets. In addition, existing systems generally lack optimization mechanisms based on the pet's actual behavioral feedback, and cannot dynamically adjust translation weights and synthesis strategies according to the differences in the responses of specific individuals. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides a human-pet cross-species bidirectional translation method and system based on multimodal data, aiming to improve the problems of unidirectional asymmetry in existing cross-species translation logic, lack of induced emotional signals, and inability to conduct co-evolution for specific individuals due to the lack of feedback mechanisms.

[0005] In a first aspect, the present invention provides the following technical solution: a human-pet cross-species bidirectional translation method based on multimodal data, comprising: S1. Construct a pre-trained discriminant model to classify the collected multimodal signals from the pet side into behaviors, and use a natural language processing model to convert the classification results into human language output. S2. Simultaneously collect human-side action and speech signals, perform sentiment analysis through the natural language processing model, and extract sentiment feature vectors representing human interaction intentions; S3. Input the emotional feature vector into the generation model, combine it with the bioacoustic carrier of the target species, and synthesize and output the induced speech with emotional features; S4. Predict the predicted behavioral response of the pet after receiving the prompting voice, and simultaneously monitor the actual behavioral response of the pet after receiving the prompting voice. S5. Calculate the error between the predicted behavioral response and the actual behavioral response, and use a reinforcement learning algorithm to synchronously iteratively optimize the classification weights of the discriminant model and the synthesis parameters of the generative model.

[0006] Preferably, in step S1, the behavioral classification of the collected pet-side multimodal signals includes: The collected real-time sound frequencies of the pets are matched with the simultaneously captured pet body movements in a time sequence to form a sound-image correlation data pair; Obtain the spatial orientation information of the pet's current environment, and superimpose the spatial orientation information as a background parameter into the sound-image association data pair; The superimposed data is input into the discrimination model, which determines the pet's specific behavioral intentions by recognizing the pet's sounds, body movements, and interactions with environmental objects.

[0007] Preferably, in step S2, the extraction of the emotional feature vector representing the human interaction intention includes: Collect human speech and action signals, and use the natural language processing model to identify the meaning of human language in the input information and the corresponding body movements; The parsed language meaning and body movements are encapsulated into an emotional feature vector.

[0008] Preferably, in step S3, the bioacoustic carrier of the target species is extracted from an existing dataset, including: Annotated audio samples of the target species are retrieved from the existing dataset, and signal processing is used to separate the individual timbre features from the environmental noise features in the audio samples. The audio samples after feature stripping are fitted with a minimum residual, and the fitted baseline acoustic curve is used as the bioacoustic carrier.

[0009] Preferably, in step S3, the synthesis and output of the induced speech with emotional features includes: The emotional feature vector is parsed into the envelope, fundamental frequency, and harmonic components of the synthesized waveform; The generative model is invoked to modulate the waveform of a preset bioacoustic carrier, and an induction signal that retains emotional characteristics and conforms to the auditory frequency band of the target species is output.

[0010] Preferably, in step S4, the predicted behavioral response of the pet after receiving the induced voice includes: The original human semantic labels corresponding to the induced speech output by the generation model are synchronized in real time. Based on the semantic tags, a matching baseline behavior template is retrieved from a pre-defined cross-species behavior association database to serve as the predicted behavioral response.

[0011] Preferably, in step S5, calculating the error between the predicted behavioral response and the actual behavioral response includes: The predicted behavioral response and the actual behavioral response are mapped to the same high-dimensional behavioral representation space, generating predicted behavioral vectors and actual behavioral vectors; Calculate the similarity between the predicted behavior vector and the actual behavior vector in the representation space to obtain a quantitative score representing the convergence of behaviors.

[0012] Preferably, in step S5, the synchronous iterative optimization of the classification weights of the discriminant model and the synthesis parameters of the generative model using a reinforcement learning algorithm includes: The quantized score is used as the reward value for reinforcement learning, and the quantized score is set to be positively correlated with the reward value; With the goal of maximizing the reward value, the recognition feature weights of the discrimination model and the acoustic synthesis parameters of the generation model are adjusted simultaneously to make the system's judgment criteria for the pet's intentions and the generation model's inducement strategy more coordinated.

[0013] Secondly, the present invention provides the following technical solution: a human-pet cross-species bidirectional translation system based on multimodal data, used to implement any of the above-mentioned human-pet cross-species bidirectional translation methods, the system comprising: The pet behavior classification module is used to build a pre-trained discriminant model to classify the collected multimodal signals from pets and to convert the classification results into human language output through a natural language processing model. The emotion feature extraction module is used to simultaneously collect human action and speech signals, perform emotion analysis through the natural language processing model, and extract emotion feature vectors that represent human interaction intentions. The induced speech synthesis module is used to input the emotional feature vector into the generation model, combine it with the bioacoustic carrier of the target species, and synthesize and output induced speech with emotional features. The pet behavior monitoring module is used to predict the pet's behavioral response after receiving the prompting voice, and simultaneously monitor the pet's actual behavioral response after receiving the prompting voice. The model synchronous iteration module is used to calculate the error between the predicted behavioral response and the actual behavioral response, and to synchronously iteratively optimize the classification weights of the discriminant model and the synthesis parameters of the generative model using a reinforcement learning algorithm.

[0014] The present invention has the following beneficial effects: 1. This invention realizes the deep simulation and generation of animal language from human semantics. By mapping human emotional vectors to the physical modulation parameters of bioacoustic carriers, the synthesized signal retains the auditory characteristics of specific species and incorporates real-time emotional intent, realizing the induced expression of humans to pets in cross-species interaction.

[0015] 2. This invention constructs a complete closed loop of prediction-monitoring-feedback, using quantified behavioral similarity as the reward value, and synchronously iterating the discrimination weights and synthesis parameters. This mechanism enables the system to adapt to specific individual pets and home environments, achieving dynamic alignment between understanding accuracy and induction strategies, and possessing extremely strong self-evolution capabilities. Attached Figure Description

[0016] Figure 1 This is a flowchart of a human-pet cross-species bidirectional translation method based on multimodal data proposed in this invention; Figure 2 This is a diagram illustrating the overall execution framework of a human-pet cross-species bidirectional translation method based on multimodal data proposed in this invention. Figure 3 This is a structural diagram of a human-pet cross-species bidirectional translation system based on multimodal data proposed in this invention. Detailed Implementation

[0017] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1 In the first embodiment of the present invention, the present invention provides a method and system for bidirectional human-pet cross-species translation based on multimodal data, such as... Figure 1 As shown, it includes the following steps: S1. Construct a pre-trained discriminant model to classify the collected multimodal signals from the pet side into behaviors, and use a natural language processing model to convert the classification results into human language output. Preferably, in step S1, the behavioral classification of the collected pet-side multimodal signals includes: The collected real-time sound frequencies of the pets are matched with the simultaneously filmed pet body movements in a time sequence to form a sound-image correlation data pair; Obtain the spatial orientation information of the pet's current environment, and superimpose the spatial orientation information as a background parameter into the sound-image association data pair; The superimposed data is input into the discrimination model, which determines the pet's specific behavioral intentions by recognizing the pet's sounds, body movements, and interactions with environmental objects.

[0019] Specifically, the framework of the present invention is as follows: Figure 2As shown, the discriminant model can be an existing multimodal network model, as long as it has the ability to extract temporal features, extract dynamic behavior features from video frames, and fuse them with audio features, it can meet the requirements of this solution, such as VideoMAE, TimeSformer, or CLIP-Video. During the pre-training phase, the system pre-trains the model using existing public datasets such as CatMeows, DogInAction, and AVA-Kinetics.

[0020] The system simultaneously captures vocal signals and video information from pets via a microphone array and a depth camera. It uses a unified clock reference to timestamp and align the audio and video streams, ensuring a one-to-one correspondence between each frame of a limb image and its corresponding spectral characteristics. For example, when the system captures continuous meows between 1000Hz and 1500Hz, it simultaneously extracts key skeletal features from the video stream, such as the height of the pet's back arch and the frequency of its tail wagging, and encapsulates these features into temporally relevant audio-visual data pairs.

[0021] The system uses SLAM or object recognition technology to delineate the pet's activity area and defines specific objects within the space, such as food bowls, doors, and litter boxes, as spatial semantic coordinates. When the pet is within a 0.5-meter radius of the food bowl, the system uses the spatial orientation information "location: next to the food bowl" as a background parameter and overlays it into the aforementioned audio-visual association data pair in the form of metadata.

[0022] The data is input into a discrimination model, which not only recognizes the rhythm of sounds and body movements, but also combines the interaction with environmental objects to make a comprehensive judgment. For example, if multimodal features match the behavioral templates of approaching a food bowl, high-frequency vocalizations, and eye contact, the discrimination model will label the output behavioral intent as hunger / seeking food. Subsequently, a natural language processing model will convert this label into human language such as "I am hungry, please give me some food" for output.

[0023] Through the above implementation methods, the present invention can transform discrete pet biological signals into structured intent tags that contain physical environmental background and behavioral logic, effectively improving the accuracy and semantic richness of understanding pet behavior in cross-species translation.

[0024] S2. Simultaneously collect human-side action and speech signals, perform sentiment analysis through the natural language processing model, and extract sentiment feature vectors representing human interaction intentions; Preferably, in step S2, the extraction of the emotional feature vector representing the human interaction intention includes: Collect human speech and action signals, and use the natural language processing model to identify the meaning of human language in the input information and the corresponding body movements; The parsed language meaning and body movements are encapsulated into an emotional feature vector.

[0025] Specifically, the system utilizes microphone and vision sensors to acquire real-time speech waveform signals and 3D skeleton sequence signals from the human side. The speech signals are converted into text by a speech recognition module and input into natural language processing models such as BERT or RoBERTa to extract semantic features. Simultaneously, a visual model is used to calculate the rate of displacement change of skeleton joints per unit time to quantify the acceleration and amplitude features of limb movements, thereby characterizing the limb actions. Subsequently, using a linear projection layer or vector concatenation, the extracted semantic features and limb movement features are fused and dimensionality reduced to generate a fixed-dimensional emotion feature vector.

[0026] When a human receives a signal from a pet requesting food, if they turn sideways towards the food storage area, wave their hand, and speak slowly and calmly (e.g., "Coming right now"), the system fuses the low-acceleration waving motion captured by the visual model with the "commitment / soothing" semantics extracted by BERT to generate an emotional feature vector representing "positive, soothing, and in action." If the human stands still, waves their hand dramatically, and speaks rapidly (e.g., "Stop barking"), the system generates an emotional feature vector representing "negative, stopping, and rejection."

[0027] Through the above implementation methods, the system can digitally deconstruct and integrate the meaning of human language and body auxiliary information in different interactive contexts, thereby accurately locking the emotional tone of subsequent induction signal synthesis.

[0028] S3. Input the emotional feature vector into the generation model, combine it with the bioacoustic carrier of the target species, and synthesize and output the induced speech with emotional features; Preferably, in step S3, the bioacoustic carrier of the target species is extracted from an existing dataset, including: Annotated audio samples of the target species are retrieved from the existing dataset, and signal processing is used to separate the individual timbre features from the environmental noise features in the audio samples. The audio samples after feature stripping are fitted with a minimum residual, and the fitted baseline acoustic curve is used as the bioacoustic carrier.

[0029] Specifically, the system retrieves audio samples of the target species labeled with emotion tags from publicly available datasets such as AudioSet or CatMeows. It uses short-time Fourier transform to convert the time-domain signal into a frequency-domain distribution and employs spectral subtraction to remove environmental noise features. Simultaneously, it uses linear predictive coding to analyze the spectral envelope of the signal, extracting and removing parameters characterizing individual formants, thereby stripping away individual timbre features. Subsequently, for multiple sets of denoised frequencies under the same emotion tag, it performs polynomial regression using the least squares method, performs residual fitting to calculate the average energy trend of this type of signal on the frequency-time axis, and uses this fitted baseline acoustic curve as the bioacoustic carrier.

[0030] For example, when extracting carriers for seeking interaction categories, the system processes dozens of groups of calls from different individuals, filters out background noise and the sharp or deep timbre unique to each pet, and finally fits a frequency fluctuation curve that is universal in the species.

[0031] Through the above implementation methods, the present invention can extract common acoustic features across individuals from chaotic biological audio, providing a standardized physical carrier for the injection of emotional information.

[0032] Preferably, in step S3, the synthesis and output of the induced speech with emotional features includes: The emotional feature vector is parsed into the envelope, fundamental frequency, and harmonic components of the synthesized waveform; The generative model is invoked to modulate the waveform of a preset bioacoustic carrier, and an induction signal that retains emotional characteristics and conforms to the auditory frequency band of the target species is output.

[0033] Specifically, the generation model receives the emotion feature vector output from step S2 and parses it into a set of scalars that control acoustic features through a parameter mapping layer, including the envelope parameter that adjusts the overall energy over time, the fundamental frequency offset that determines the pitch, and the harmonic components that characterize the fullness of the sound quality.

[0034] In one specific implementation, the generative model can be implemented using a generative adversarial network (GAN) architecture. The generator takes an emotional feature vector as conditional input and upsamples and modulates a preset bioacoustic carrier through a transposed convolutional layer. A discriminator ensures that the output signal's acoustic distribution conforms to the biological characteristics of the target species. In another specific implementation, a diffusion model can also be used as the generative model. This involves shifting the latent representation of the bioacoustic carrier in the latent variable space to reflect emotional bias, and then reconstructing a speech waveform that retains the original carrier skeleton and possesses the target emotional features via a decoder.

[0035] For example, when the emotional feature vector output by S2 represents "positive, soothing, in action", the generative model will lower the fundamental frequency component to obtain a deeper and softer tone, and extend the decay time of the waveform envelope to produce a smooth listening experience, thereby modulating the biocarrier into an inducing signal with soothing characteristics; conversely, when the emotional feature vector represents "negative, stopping, rejection", the generative model will increase the fundamental frequency component and shorten the rise time of the waveform envelope, and increase the energy intensity of the harmonic components, thereby synthesizing an inducing signal with a sense of warning and oppression.

[0036] Through the above implementation methods, the present invention achieves a precise mapping from abstract emotional vectors to specific acoustic physical parameters, ensuring the bio-realism and emotional expressiveness of the output induced signal.

[0037] S4. Predict the predicted behavioral response of the pet after receiving the prompting voice, and simultaneously monitor the actual behavioral response of the pet after receiving the prompting voice. Preferably, in step S4, the predicted behavioral response of the pet after receiving the induced voice includes: The original human semantic labels corresponding to the induced speech output by the generation model are synchronized in real time. Based on the semantic tags, a matching baseline behavior template is retrieved from a pre-defined cross-species behavior association database to serve as the predicted behavioral response.

[0038] Specifically, while outputting the induced speech, the system extracts the original human semantic tags parsed in step S2 in real time. In one specific implementation, the system has a built-in cross-species behavior association library based on a key-value pair structure. This library pre-stores the mapping relationship between human semantic commands and the expected response actions of pets. The system uses the semantic tags as search keys to retrieve the corresponding baseline behavior templates from the association library.

[0039] For example, when the semantic tag is "recall", the retrieved behavior template is "displacement occurs towards the center coordinate of the device, and the angle between the displacement vector direction and the device is less than 30°"; when the semantic tag is "stop", the retrieved behavior template is "the current motion state terminates, or displacement occurs away from the center coordinate of the device".

[0040] Through the above implementation methods, the present invention transforms the abstract translation intent into a quantifiable expectation of physical behavior, providing a clear logical reference benchmark for determining whether the interaction is successful.

[0041] S5. Calculate the error between the predicted behavioral response and the actual behavioral response, and use a reinforcement learning algorithm to synchronously iteratively optimize the classification weights of the discriminant model and the synthesis parameters of the generative model.

[0042] Preferably, in step S5, calculating the error between the predicted behavioral response and the actual behavioral response includes: The predicted behavioral response and the actual behavioral response are mapped to the same high-dimensional behavioral representation space to generate predicted behavioral vectors and actual behavioral vectors. Calculate the similarity between the predicted behavior vector and the actual behavior vector in the representation space to obtain a quantitative score representing the convergence of behaviors.

[0043] Specifically, the system utilizes a pre-trained multimodal encoder, such as a multilayer perceptron or a fully connected network, as a feature transformation model. First, the baseline behavior template retrieved in S4 is transformed into a predicted behavior vector; simultaneously, the real-time limb dynamics, displacement vectors, and posture features of the pet monitored in S1 are extracted as actual behavior vectors. In one feasible implementation, the system projects the features from these two different sources onto the same 128-dimensional high-dimensional behavior representation space using a multilayer perceptron with shared weights. Within this space, the system uses a cosine similarity algorithm to calculate the cosine of the angle between the predicted behavior vector and the actual behavior vector. The calculated quantization score ranges from [-1, 1]. The closer the score is to 1, the more consistent the pet's actual response is with human interaction expectations. For example, if the human intention is "calling," the predicted vector points to "centripetal displacement," while the pet is actually "accelerating closer," then the two vectors are highly consistent in direction within the representation space, and the similarity quantization score is close to 1. If the pet remains stationary or runs away, the vector angle increases, and the quantization score decreases significantly.

[0044] Through the above implementation methods, the present invention transforms expected instructions and physical actions into mathematical metrics within a unified space, eliminating obstacles to cross-modal data comparison and providing feasible objective function inputs for reinforcement learning.

[0045] Preferably, in step S5, the synchronous iterative optimization of the classification weights of the discriminant model and the synthesis parameters of the generative model using a reinforcement learning algorithm includes: The quantized score is used as the reward value for reinforcement learning, and the quantized score is set to be positively correlated with the reward value; With the goal of maximizing the reward value, the recognition feature weights of the discrimination model and the acoustic synthesis parameters of the generation model are adjusted simultaneously to make the system's judgment criteria for the pet's intentions and the generation model's inducement strategy more coordinated.

[0046] Specifically, the system maps the quantized score calculated in step S5 to the reward function of reinforcement learning algorithms such as PPO or SAC. It is important to note that the specific parameter update logic of the PPO, SAC, or other reinforcement learning algorithms follows the standard reinforcement learning framework disclosed in the art. Subsequently, the higher the quantized score, the greater the reward value obtained by the system. In one feasible implementation, the reward function... It can be designed as: ; in, The quantified score calculated in the current step. This represents the change in score compared to the previous interaction period. and These are the preset weighting coefficients.

[0047] The system aims to maximize the cumulative reward value and uses a backpropagation mechanism to synchronously propagate the error gradient back to both modules: For the discrimination model, the weight parameters of the feature extraction layer are adjusted. If the actual behavior deviates from the expectation, the system will fine-tune the sampling bias of the pet's action features to correct its recognition benchmark of the pet's original intention.

[0048] For the generative model, the control strategy of the acoustic parameter mapping layer is adjusted. If the induced speech fails to elicit the expected response, the system iteratively changes the weighting of the fundamental frequency shift, envelope duration, and harmonic intensity to optimize the induction strategy and find a combination of acoustic parameters that is more likely to resonate with the pet.

[0049] For example, if the discrimination model identifies the cat's behavior as "begging for food," but the cat does not approach the food bowl after the prompting voice is given, the reinforcement learning system will simultaneously reflect on whether the discrimination model misjudged the cat's initial intention or whether the generated prompting frequency was not accurate enough. Through multiple iterations, the system will automatically find the optimal acoustic modulation solution that best encourages the pet to "approach the food bowl" in that environmental context.

[0050] Through the above implementation methods, the present invention achieves the co-evolution of the discrimination end and the generation end, enabling the system to automatically complete the personalized alignment of translation logic and induction strategy for different individual pets and their specific interaction habits.

[0051] Example 2 This invention provides a human-pet cross-species bidirectional translation system based on multimodal data, the structure of which is as follows: Figure 3 As shown, it includes: The pet behavior classification module is used to build a pre-trained discriminant model to classify the collected multimodal signals from pets and to convert the classification results into human language output through a natural language processing model. The emotion feature extraction module is used to simultaneously collect human action and speech signals, perform emotion analysis through the natural language processing model, and extract emotion feature vectors that represent human interaction intentions. The induced speech synthesis module is used to input the emotional feature vector into the generation model, combine it with the bioacoustic carrier of the target species, and synthesize and output induced speech with emotional features. The pet behavior monitoring module is used to predict the pet's behavioral response after receiving the prompting voice, and simultaneously monitor the pet's actual behavioral response after receiving the prompting voice. The model synchronous iteration module is used to calculate the error between the predicted behavioral response and the actual behavioral response, and to synchronously iteratively optimize the classification weights of the discriminant model and the synthesis parameters of the generative model using a reinforcement learning algorithm.

[0052] To further verify the effectiveness of this solution in real-world interaction scenarios, we selected Traini, a leading end-to-end translation framework, as the core comparison. The experiment focused on two dimensions: translation accuracy and guidance success rate, recording the performance evolution over 60 days. This experiment was conducted in a simulated home environment. The experimental sample included 20 pairs of human-pet pairs. The canine samples covered 15 common breeds, including Labradors, Shiba Inus, and Border Collies, with ages ranging from 1 to 8 years. The human samples included volunteers of different genders and age groups.

[0053] The experimental results are shown in Table 1. As can be seen from the table, in the initial stage of the experiment, Trainini's behavioral translation accuracy reached 81.50%, which was better than the 75.30% of this approach. As the testing period progressed, the accuracy of this approach began to climb significantly, reaching 93.50% in the 45-60 day phase. In terms of guidance success rate, Trainini's performance remained around 60.00%, while this approach showed a continuous upward trend, eventually reaching 73.60%.

[0054] Experimental results fully demonstrate that this invention, by introducing a reinforcement learning feedback loop and an emotion feature induction mechanism, overcomes the technical bottlenecks of traditional translation systems, such as the upper limit of recognition accuracy and the single interactive feedback. The system possesses strong self-iterative capabilities, enabling it to complete deep adaptation to specific individual pets in a short time, significantly improving the intelligence level and practical value of cross-species interaction.

[0055] Table 1. Performance comparison of the control group and this scheme at different test stages. ; Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A human-pet cross-species bidirectional translation method based on multimodal data, characterized in that, include: S1. Construct a pre-trained discriminant model to classify the collected multimodal signals from the pet side, and convert the classification results into human language output through a natural language processing model. S2. Simultaneously collect human-side action and speech signals, perform sentiment analysis through the natural language processing model, and extract sentiment feature vectors representing human interaction intentions; S3. Input the emotional feature vector into the generation model, combine it with the bioacoustic carrier of the target species, and synthesize and output the induced speech with emotional features; S4. Predict the predicted behavioral response of the pet after receiving the prompting voice, and simultaneously monitor the actual behavioral response of the pet after receiving the prompting voice. S5. Calculate the error between the predicted behavioral response and the actual behavioral response, and use a reinforcement learning algorithm to synchronously iteratively optimize the classification weights of the discriminant model and the synthesis parameters of the generative model.

2. The human-pet cross-species bidirectional translation method based on multimodal data according to claim 1, characterized in that, In step S1, the behavioral classification of the collected pet-side multimodal signals includes: The collected real-time sound frequencies of the pets are matched with the simultaneously captured pet body movements in a time sequence to form a sound-image correlation data pair; Obtain the spatial orientation information of the pet's current environment, and superimpose the spatial orientation information as a background parameter into the sound-image association data pair; The superimposed data is input into the discrimination model, which determines the pet's specific behavioral intentions by recognizing the pet's sounds, body movements, and interactions with environmental objects.

3. The human-pet cross-species bidirectional translation method based on multimodal data according to claim 1, characterized in that, In step S2, the extraction of the emotional feature vector representing the human interaction intention includes: Collect human speech and action signals, and use the natural language processing model to identify the meaning of human language in the input information and the corresponding body movements; The parsed language meaning and body movements are encapsulated into an emotional feature vector.

4. The human-pet cross-species bidirectional translation method based on multimodal data according to claim 1, characterized in that, In step S3, the bioacoustic carrier of the target species is extracted from an existing dataset, including: Annotated audio samples of the target species are retrieved from the existing dataset, and signal processing is used to separate the individual timbre features from the environmental noise features in the audio samples. The audio samples after feature stripping are fitted with a minimum residual, and the fitted baseline acoustic curve is used as the bioacoustic carrier.

5. The human-pet cross-species bidirectional translation method based on multimodal data according to claim 1, characterized in that, In step S3, the synthesis and output of the induced speech with emotional features includes: The emotional feature vector is parsed into the envelope, fundamental frequency, and harmonic components of the synthesized waveform; The generative model is invoked to modulate the waveform of a preset bioacoustic carrier, and an induction signal that retains emotional characteristics and conforms to the auditory frequency band of the target species is output.

6. The human-pet cross-species bidirectional translation method based on multimodal data according to claim 1, characterized in that, In step S4, the predicted behavioral response of the pet after receiving the induced voice includes: The original human semantic labels corresponding to the induced speech output by the generation model are synchronized in real time. Based on the semantic tags, a matching baseline behavior template is retrieved from a pre-defined cross-species behavior association database to serve as the predicted behavioral response.

7. The human-pet cross-species bidirectional translation method based on multimodal data according to claim 1, characterized in that, In step S5, the error between the predicted behavioral response and the actual behavioral response is calculated, including: The predicted behavioral response and the actual behavioral response are mapped to the same high-dimensional behavioral representation space to generate predicted behavioral vectors and actual behavioral vectors. Calculate the similarity between the predicted behavior vector and the actual behavior vector in the representation space to obtain a quantitative score representing the convergence of behaviors.

8. A human-pet cross-species bidirectional translation method based on multimodal data according to claim 1, characterized in that, In step S5, the synchronous iterative optimization of the classification weights of the discriminant model and the synthesis parameters of the generative model using a reinforcement learning algorithm includes: The quantized score is used as the reward value for reinforcement learning, and the quantized score is set to be positively correlated with the reward value; With the goal of maximizing the reward value, the recognition feature weights of the discrimination model and the acoustic synthesis parameters of the generation model are adjusted simultaneously to make the system's judgment criteria for the pet's intentions and the generation model's inducement strategy more coordinated.

9. A human-pet cross-species bidirectional translation system based on multimodal data, characterized in that, The system for implementing the human-pet cross-species bidirectional translation method according to any one of claims 1-8 comprises: The pet behavior classification module is used to build a pre-trained discriminant model to classify the collected multimodal signals from pets and to convert the classification results into human language output through a natural language processing model. The emotion feature extraction module is used to simultaneously collect human action and speech signals, perform emotion analysis through the natural language processing model, and extract emotion feature vectors that represent human interaction intentions. The induced speech synthesis module is used to input the emotional feature vector into the generation model, combine it with the bioacoustic carrier of the target species, and synthesize and output induced speech with emotional features. The pet behavior monitoring module is used to predict the pet's behavioral response after receiving the prompting voice, and simultaneously monitor the pet's actual behavioral response after receiving the prompting voice. The model synchronous iteration module is used to calculate the error between the predicted behavioral response and the actual behavioral response, and to synchronously iteratively optimize the classification weights of the discriminant model and the synthesis parameters of the generative model using a reinforcement learning algorithm.

Images