Pet interaction intelligent translation method based on deep neural network

Abstract

This invention relates to the fields of artificial intelligence and deep neural network technology, and discloses a pet interaction intelligent translation method based on deep neural networks. The method acquires pet audio and video streams, converts the audio stream into a Mel spectrogram, and the video stream into an optical flow map containing key points of the head and limbs. Temporal acoustic features and spatiotemporal motion features are extracted using a dual-branch encoder of the deep neural network. Using the temporal acoustic features as the query vector and the spatiotemporal motion features as the key and value vectors, feature mapping is performed through a cross-attention layer to output a multimodal joint representation vector. Gating coefficients are calculated based on the signal-to-noise ratio of the audio stream, and the audio and video components in the multimodal joint representation vector are weighted and fused using these gating coefficients to generate a target fusion vector. This invention eliminates the asynchronous offset error between vocalization and limb movements within the feature space, suppresses invalid feature activation, and blocks environmental background noise interference.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and deep neural network technology, and discloses a pet interaction intelligent translation method based on deep neural networks. Background Technology

[0002] Current deep neural network-based pet interaction translation solutions employ a dual-modal decision layer fusion architecture. The conventional approach involves extracting spectral features from pet audio through an independently running acoustic processing branch, and extracting motion features from pet video through a visual processing branch. After completing their respective feature extractions, existing technologies directly concatenate the two feature vectors at the classification decision layer, then input the concatenated features into the classifier to output the translation result. In this process, acoustic and visual features are extracted in their respective independent network channels, without any cross-computation during the feature extraction stage.

[0003] The aforementioned conventional approach has limitations in handling pet interaction signals in real-world home environments. Pets often exhibit physical movements such as opening their mouths or arching their backs at the moment they emit specific vocalizations, resulting in an inherent asynchronous shift between the vocalization and the sound signal in terms of physical timing. Conventional approaches extract audio and video features separately in independent network channels and then directly concatenate them at the decision layer. This lack of spatiotemporal alignment calculations for the misaligned audio and video signals within the network feature space leads to misaligned temporal correspondences within the concatenated feature vectors, triggering invalid feature activation and causing the translation results to deviate from the pet's true intended expression. Summary of the Invention

[0004] The purpose of this invention is to provide a pet interaction intelligent translation method based on deep neural networks, which can effectively solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A pet interaction intelligent translation method based on deep neural networks includes: acquiring synchronously collected pet audio streams and pet video streams, converting the pet audio streams into Mel spectrograms, and converting the pet video streams into optical flow maps containing head key points and limb key points; The Mel spectrogram is input into the audio branch encoder of the deep neural network, and temporal acoustic features are extracted through one-dimensional convolution. The optical flow map is input into the video branch encoder of the deep neural network, and spatiotemporal motion features are extracted through three-dimensional convolution. The temporal acoustic features and the spatiotemporal motion features are input into the cross-attention layer of the deep neural network. The temporal acoustic features are used as the query vector, and the spatiotemporal motion features are used as the key vector and value vector to perform feature mapping calculation, and output a multimodal joint representation vector. The multimodal joint representation vector is input into the spatiotemporal gated fusion layer in the deep neural network. The gate coefficient is calculated based on the signal-to-noise ratio of the pet audio stream. The audio component and video component in the multimodal joint representation vector are weighted and fused using the gate coefficient to generate the target fusion vector. The target fusion vector is input into the decoder in the deep neural network to output the pet intention text.

[0006] Preferably, the step of converting the pet video stream into an optical flow map containing head key points and limb key points includes: extracting the skeleton from the video frames in the pet video stream to obtain an initial skeleton topology map; In the initial skeleton topology map, the head region and multiple limb regions are divided, and the set of joint coordinates of the head region and multiple limb regions is extracted; Based on the displacement change of the key point coordinate set between adjacent video frames, the horizontal and vertical optical flow components of the head key points are generated, as well as the horizontal and vertical optical flow components of the limb key points are generated. The horizontal and vertical optical flow components of the key points of the head are concatenated to generate the head optical flow tensor. The horizontal and vertical optical flow components of the key points of the limbs are concatenated to generate the limb optical flow tensor. The head optical flow tensor and the limb optical flow tensor are fused to generate the optical flow map.

[0007] Preferably, the step of extracting temporal acoustic features by one-dimensional convolution includes: performing frame processing on the Mel spectrogram along the time dimension to obtain a spectral frame sequence; The spectral frame sequence is input into a one-dimensional convolutional network containing a first convolutional layer, a second convolutional layer, and a first pooling layer. Local frequency domain features are extracted in the first convolutional layer using a first-size convolutional kernel, and cross-frame frequency domain features are extracted in the second convolutional layer using a second-size convolutional kernel. The temporal acoustic features are obtained by downsampling the cross-frame frequency domain features through the first pooling layer. The steps for extracting spatiotemporal motion features through 3D convolution include: dividing the optical flow map into a sequence of time windows; inputting the sequence of time windows into a 3D convolutional network containing a 3D convolutional kernel and a second pooling layer; performing sliding convolution calculations on the sequence of time windows in both the time and spatial dimensions using the 3D convolutional kernel; and reducing the dimensionality of the sliding convolution calculation results using the second pooling layer to obtain the spatiotemporal motion features.

[0008] Preferably, the step of performing feature mapping calculation using the temporal acoustic features as query vectors and the spatiotemporal motion features as key and value vectors includes: performing linear mapping on the temporal acoustic features to generate a query matrix, and performing linear mapping on the spatiotemporal motion features to generate a key matrix and a value matrix. The query matrix is ​​divided into multiple query sub-matrices, the key matrix is ​​divided into multiple key sub-matrices, and the value matrix is ​​divided into multiple value sub-matrices; For each query submatrix, the product of the query submatrix and the transpose of the corresponding key submatrix is ​​calculated to obtain the attention score matrix. The attention score matrix is ​​normalized, and the normalized attention score matrix is ​​multiplied by the corresponding value submatrix to obtain multiple attention output submatrixes. The multiple attention output submatrixes are concatenated to obtain the multimodal joint representation vector.

[0009] Preferably, the step of calculating the gating coefficient based on the signal-to-noise ratio of the pet audio stream includes: calculating the frame signal power of each frame of audio signal in the pet audio stream, and extracting the background noise power of the silent segment in the pet audio stream; The initial signal-to-noise ratio of each frame audio signal is calculated based on the difference between the frame signal power and the background noise power. The initial signal-to-noise ratio is input into a fully connected layer containing an activation function, and the initial signal-to-noise ratio is mapped to a preset numerical range to generate the gating coefficients; The step of weighted fusion of audio and video components in the multimodal joint representation vector using the gating coefficient includes: multiplying the gating coefficient by the audio component in the multimodal joint representation vector, multiplying the difference between the value 1 and the gating coefficient by the video component in the multimodal joint representation vector, and adding the two multiplication results into a vector sum to generate the target fusion vector.

[0010] Preferably, the step of inputting the target fusion vector into the decoder in the deep neural network includes: inputting the target fusion vector as an initial hidden state into the first self-attention layer of the decoder, performing self-attention calculation on the initial hidden state, and generating decoder self-attention features; The decoder self-attention features and the target fusion vector are input into the cross-attention layer of the decoder. The decoder self-attention features are used as the decoding query vector, and the target fusion vector is used as the decoding key vector and the decoding value vector to perform cross-attention calculation to generate decoder joint features. The decoder's joint features are input into a feedforward neural network layer for nonlinear transformation. The output vector after nonlinear transformation is input into a softmax classification layer. The output vector after nonlinear transformation is mapped to a word probability distribution. The word with the highest probability value is output as the pet intent text.

[0011] Preferably, the process of extracting the set of joint coordinates of the head region and the multiple limb regions includes: when a target joint in the head region or the multiple limb regions is detected to be occluded in the current video frame, extracting the historical coordinate trajectory of the target joint in the preceding multiple consecutive video frames of the current video frame; The historical coordinate trajectory is input into a Kalman filter, and the predicted coordinates of the target key point in the current video frame are calculated using the state prediction equation of the Kalman filter. Determine whether the distance between the predicted coordinates and the coordinates of the visible adjacent joints of the target joint in the current video frame is within a preset distance threshold range. If it is within the preset distance threshold range, then use the predicted coordinates as the coordinates of the target joint in the current video frame and update the joint coordinate set.

[0012] Preferably, the step of extracting cross-frame frequency domain features using a second-size convolutional kernel in the second convolutional layer includes: configuring the second convolutional layer as a multi-scale dilated convolutional layer containing three parallel dilated convolutional branches, wherein the three parallel dilated convolutional branches have a first dilation rate, a second dilation rate, and a third dilation rate, respectively. The local frequency domain features are respectively input into the three parallel dilated convolution branches. Cross-frame frequency domain features at the first time scale are extracted through the branch with the first dilation rate, cross-frame frequency domain features at the second time scale are extracted through the branch with the second dilation rate, and cross-frame frequency domain features at the third time scale are extracted through the branch with the third dilation rate. The cross-frame frequency domain features of the first time scale, the cross-frame frequency domain features of the second time scale, and the cross-frame frequency domain features of the third time scale are concatenated in the channel dimension to generate multi-scale concatenated features, and the multi-scale concatenated features are input into the first pooling layer.

[0013] Preferably, the step of dividing the query matrix into multiple query sub-matrices includes: dividing the Mel spectrogram into frequency bands to obtain high-frequency sub-bands, mid-frequency sub-bands, and low-frequency sub-bands; Calculate the average energy value of the high-frequency sub-band, the average energy value of the mid-frequency sub-band, and the average energy value of the low-frequency sub-band; The target number of partitions for the multiple query sub-matrices is determined based on the proportional relationship between the average energy value of the high-frequency sub-band, the average energy value of the mid-frequency sub-band, and the average energy value of the low-frequency sub-band. The query matrix is ​​uniformly divided according to the target number of partitions to obtain the plurality of query submatrices; the key matrix is ​​uniformly divided according to the target number of partitions to obtain the plurality of key submatrices; and the value matrix is ​​uniformly divided according to the target number of partitions to obtain the plurality of value submatrices.

[0014] Preferably, the step of extracting the background noise power of the silent segment in the pet audio stream includes: performing voice endpoint detection on the pet audio stream and dividing the pet audio stream into a sound segment and a silent segment; Extract multiple frames of silent audio signals within the initial time window of the silent segment, calculate the initial average power of the multiple frames of silent audio signals, and use the initial average power as the initial background noise power; During the continuous acquisition of the pet audio stream, multiple frames of updated silence audio signals within subsequent time windows are extracted, and the average update power of the multiple frames of updated silence audio signals is calculated. Calculate the moving average of the updated average power and the initial background noise power, use the moving average as the updated background noise power, replace the initial background noise power with the updated background noise power, and calculate the initial signal-to-noise ratio of subsequent audio frames.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention solves the technical problem of temporal misalignment of multimodal features by constructing a cross-attention layer and a spatiotemporal gating fusion layer within a deep neural network. Using temporal acoustic features extracted from the audio branch as the query vector and spatiotemporal motion features extracted from the video branch as the key and value vectors for feature mapping calculation, the acoustic features can retrieve action features at corresponding time points in the visual feature sequence, eliminating the asynchronous offset error between vocal signals and limb movements within the feature space. Furthermore, by combining gating coefficients calculated based on the audio signal-to-noise ratio with adaptive weighting of the audio and video components in the joint representation vector, invalid feature activation corresponding to misalignment within the feature space is suppressed, ensuring that the acoustic and visual features in the target fusion vector input to the decoder are under the same temporal reference.

[0016] 2. This invention converts the video stream into an optical flow map that distinguishes between head and limb keypoints. When a keypoint is occluded, a Kalman filter is used to perform prediction updates based on the coordinate distance between adjacent keypoints, maintaining the continuity of the video branch input data. By adaptively determining the number of query matrix partitions based on the proportional relationship of the average energy value of the frequency band, and by updating the background noise power of silent segments in real time through the moving average, the network can dynamically adjust the calculation path and fusion weights according to the frequency domain physical properties of the current input signal and changes in background noise, thus blocking the interference of environmental background noise in the feature extraction process. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the overall method of the present invention; Figure 2 This is a flowchart of the optical flow map generation process for key points of the head and limbs in this invention; Figure 3 This is a flowchart of the audio branch multi-scale one-dimensional convolution feature extraction process of the present invention; Figure 4 This is a flowchart of the spatiotemporal alignment of multimodal features in the cross-attention layer of this invention; Figure 5 This is a flowchart of the adaptive weighted fusion process of the spatiotemporal gating fusion layer of the present invention; Figure 6 This is a flowchart of the autoregressive generation of intent text by the Transformer decoder of the present invention. Detailed Implementation

[0018] Please refer to the attached document. Figure 1 This embodiment provides a pet interaction intelligent translation method based on deep neural networks, which acquires synchronously collected pet audio and video streams. The audio and video streams are acquired using the same sampling clock to ensure that the timestamp corresponding to each video frame is strictly aligned with the timestamp of the audio sampling points within that time period, maintaining consistency of the time reference. The acquired pet audio stream is preprocessed, converting it into a Mel spectrogram. The preprocessing process includes pre-emphasis, framing, and windowing operations. Pre-emphasis is used to compensate for the attenuation of high-frequency components during pet vocalization, improving the high-frequency detail recognition of the spectrum. Framing divides the continuous audio stream into audio frames of equal length, with overlapping sampling points between adjacent audio frames to avoid spectral leakage. Windowing uses a Hamming window to weight the single-frame audio signal, suppressing the spectral tailing effect during the Fourier transform process.

[0019] A Fast Fourier Transform (FFT) is performed on the preprocessed single-frame audio signal to convert the time-domain audio signal into a power spectrum in the frequency domain. The power spectrum is then input into a preset Mel filter bank. The power spectrum is weighted and integrated by the Mel filter bank to obtain the Mel spectral features of the single-frame audio signal. The Mel spectral features of all audio frames are stitched together along the time dimension to generate a two-dimensional Mel spectrogram. The horizontal axis of the Mel spectrogram represents the time dimension, corresponding to the temporal progression of the audio stream, while the vertical axis represents the Mel frequency dimension, corresponding to the energy distribution of different frequency bands. The value of each pixel is the signal energy value at the corresponding time and frequency band position.

[0020] The conversion relationship between Mel scale and linear frequency is defined by the following formula:

[0021] Where m is the Mel scale value and f is the linear frequency value in Hz. This formula maps linear frequencies to Mel frequencies that conform to the auditory perception characteristics of both human ears and pets, making the generated Mel spectrogram more closely match the auditory perception characteristics of pet vocal signals.

[0022] Table 1. Explanation of Mel spectrum generation parameters and their physical meanings

[0023] The parameters defined in this table are used to standardize the generation process of the Mel spectrogram, ensuring that the generated Mel spectrogram can fully preserve the temporal and frequency domain characteristics of the pet audio signal, and providing a unified input format for feature extraction of subsequent audio branches.

[0024] The synchronously acquired pet video stream undergoes preprocessing, including frame size normalization, pixel value normalization, and denoising. Frame size normalization adjusts all video frames to a uniform pixel size, and pixel value normalization maps the pixel values ​​of video frames to the 0-1 range. Denoising uses Gaussian filtering to remove image noise from video frames, improving the accuracy of subsequent keypoint detection. The preprocessed pet video stream is then converted into an optical flow map containing head and limb keypoints. Motion detection is performed on adjacent video frames in the stream to extract the displacement information of keypoints in the pet's head and limb regions between adjacent frames, generating horizontal and vertical optical flow components for each keypoint. The optical flow components of all keypoints are then stitched together and aligned in spatial dimensions to generate an optical flow map. The spatial dimension of the optical flow map is consistent with the size of the video frames, and the channel dimension corresponds to the optical flow component of different keypoints. The value of each channel represents the displacement value of the corresponding keypoint at that spatial location.

[0025] The generated Mel spectrogram is input into the audio branch encoder in the deep neural network, where temporal acoustic features are extracted through one-dimensional convolution. The audio branch encoder consists of multiple cascaded one-dimensional convolutional and pooling layers. The convolutional kernels of the one-dimensional convolutional layers slide along the time dimension of the Mel spectrogram to extract the temporal correlation features and frequency distribution features of the audio signal. The output of the one-dimensional convolutional layer is defined by the following formula:

[0026] in, Let k be the convolutional output feature of the audio branch at time step t, and k be the size of the one-dimensional convolutional kernel. Let be the weight parameters at the i-th position of the convolution kernel. Let be the input spectral features at time step ti. represents the bias parameter of the convolutional layer, and ReLU is a linear rectified activation function used to introduce a nonlinear transformation into the convolutional output, improving the network's feature fitting ability. The audio branch encoder performs multi-layer convolution and pooling processing on the input Mel spectrogram, outputting temporal acoustic features with fixed dimensions. These temporal acoustic features fully preserve the temporal variation characteristics and frequency domain energy distribution characteristics of the pet audio signal.

[0027] The generated optical flow map is input into the video branch encoder in the deep neural network, where spatiotemporal motion features are extracted through 3D convolution. The video branch encoder consists of multiple cascaded 3D convolutional layers and pooling layers. The convolutional kernels of the 3D convolutional layers slide simultaneously along the temporal, height, and width dimensions of the optical flow map, extracting both the spatial motion distribution features and temporal motion variation features of the pet's keypoints. The output of the 3D convolutional layer is defined by the following formula:

[0028] in, Let be the convolutional output feature at the t-th time step and spatial coordinates (x, y) of the video branch. The time dimension size of the convolution kernel. , The spatial dimension of the convolution kernel. Let be the weight parameters of the convolution kernel at position (τ,i,j). The input optical flow map features are for the corresponding locations. These are the bias parameters for the convolutional layer. After performing multi-layer 3D convolution and pooling processing on the input optical flow map, the video branch encoder outputs spatiotemporal motion features with fixed dimensions. These features fully preserve the spatial distribution characteristics and temporal variation characteristics of the pet's head and limb movements.

[0029] Please refer to the attached document. Figure 4 The extracted temporal acoustic features and spatiotemporal motion features are input into the cross-attention layer of the deep neural network. The temporal acoustic features are used as the query vector, and the spatiotemporal motion features are used as the key and value vectors for feature mapping calculation, outputting a multimodal joint representation vector. The cross-attention layer first performs a linear mapping on the input temporal acoustic features and spatiotemporal motion features, mapping both features to the same feature dimension to achieve dimension alignment. Then, attention calculation is performed based on the aligned features, enabling the temporal acoustic features to retrieve action features at corresponding time points in the spatiotemporal motion feature sequence, achieving spatiotemporal alignment of audio and video features within the feature space. The attention calculation process of the cross-attention layer is defined by the following formula:

[0030] Where Q is the query matrix, generated by linear mapping of temporal acoustic features; K is the key matrix; and V is the value matrix, both generated by linear mapping of spatiotemporal motion features. The feature dimension of the key matrix, is a scaling factor used to prevent the gradient of the softmax function from vanishing due to an excessively large inner product result. Softmax is a normalized exponential function used to map the attention score to a weight value in the 0-1 range.

[0031] The cross-attention layer employs a multi-head attention mechanism to perform multi-dimensional mapping calculations on features. The query matrix, key matrix, and value matrix are each divided into multiple sub-matrices. Attention calculations are performed on each sub-matrix separately. The results of all sub-matrices are then concatenated, and a final multimodal joint representation vector is generated through a linear mapping. The multi-head attention calculation process is defined by the following formula:

[0032] Where h is the number of attention heads. , , , Let be the linear mapping weight matrix corresponding to the i-th attention head. This is the output linear mapping weight matrix after multi-head concatenation, and Concat is the concatenation operation along the channel dimension.

[0033] Please refer to the attached document. Figure 5 The generated multimodal joint representation vector is input into the spatiotemporal gating fusion layer of the deep neural network. Gating coefficients are calculated based on the signal-to-noise ratio (SNR) of the pet audio stream. These gating coefficients are then used to weight and fuse the audio and video components in the multimodal joint representation vector to generate the target fusion vector. First, the SNR of the pet audio stream is calculated. The SNR is based on the frame power of the audio signal and the background noise power, defined by the following formula:

[0034] in, Let be the signal-to-noise ratio of the audio signal in frame t, in dB. Let be the frame signal power of the audio signal in frame t. The background noise power is denoted as . The calculated signal-to-noise ratio (SNR) is input into a fully connected layer containing an activation function, mapping the SNR to a preset value range of 0-1 to generate gating coefficients. The generation process of the gating coefficients is defined by the following formula:

[0035] Where g is the gating coefficient, and σ is the Sigmoid activation function, used to map the input to the 0-1 interval. These are the weight parameters of the fully connected layer. Here, SNR represents the bias parameter of the fully connected layer, and SNR represents the signal-to-noise ratio characteristic of the audio stream.

[0036] Based on the generated gating coefficients, the audio and video components in the multimodal joint representation vector are weighted and fused to generate the target fusion vector. The weighted fusion process is defined by the following formula:

[0037] in, For the target fusion vector, For the audio component in the multimodal joint representation vector, This represents the video component in the multimodal joint representation vector. The weighted fusion process adaptively adjusts the weights of the audio and video components based on the signal-to-noise ratio (SNR) of the audio signal. When the SNR is high, the weight of the audio component is increased to fully utilize the clear sound signal characteristics; when the SNR is low, the weight of the video component is increased to supplement the noise interference of the audio signal using body movement characteristics.

[0038] The generated target fusion vector is input into the decoder in the deep neural network. The decoder performs sequential decoding on the input target fusion vector and outputs pet intention text. The decoder adopts an autoregressive decoding structure, using the target fusion vector as contextual features. Starting from a preset start character, it generates pet intention text word by word until a preset end character is generated, completing the entire decoding process. The output pet intention text corresponds to the pet's true expressed intention, including but not limited to common pet intentions such as hunger, thirst, seeking companionship, playing, pain, fear, alertness, happiness, and dissatisfaction.

[0039] In this embodiment, Mel spectrograms and keypoint optical flow maps are generated by synchronously acquiring pet audio and video streams, respectively. Temporal acoustic features and spatiotemporal motion features are extracted by a dual-branch encoder. A cross-attention layer is used to achieve spatiotemporal alignment of audio and video features in the feature space, eliminating asynchronous offset errors between vocal signals and limb movements. Then, a spatiotemporal gating fusion layer is used to achieve adaptive weighted fusion of audio and video components based on the audio signal-to-noise ratio. Finally, the decoder outputs the pet's intention text, realizing multimodal joint translation of pet interaction signals. This solves the problems of temporal misalignment and invalid feature activation caused by direct splicing of bimodal features at the decision layer in the prior art.

[0040] Please refer to the attached document. Figure 2In another embodiment, skeleton extraction is performed on video frames in the pet video stream to obtain an initial skeleton topology map. The skeleton extraction employs a keypoint detection network optimized for companion pets. Forward inference is performed on the input video frames, outputting the two-dimensional pixel coordinates of all joints in the pet's body and their corresponding detection confidence scores. Based on the detected joint coordinates and a preset pet skeleton topology connection relationship, an initial skeleton topology map is constructed. The skeleton topology connection relationship defines the connection rules between adjacent joints, ensuring that the generated skeleton topology map conforms to the pet's physiological structural characteristics. The initial skeleton topology map is divided into a head region and multiple limb regions. Based on the pet's physiological structure, the skeleton topology map is divided into a head region, forelimb region, hindlimb region, trunk region, and tail region. Each region contains a corresponding set of joints. The joint coordinate sets of the head region and multiple limb regions are extracted. The coordinates of each joint are (x, y) two-dimensional pixel coordinates, corresponding to the image coordinate system of the video frame. The origin of the image coordinate system is the upper left corner of the video frame, with the x-axis pointing horizontally to the right and the y-axis pointing vertically downwards.

[0041] Table 2. Topological Region Division and Key Point Definition of Pet Skeleton

[0042] The region divisions and key point sets defined in this table are used to standardize the construction process of the pet skeleton topology map, clarify the boundaries between the head region and the limb region, and ensure that the generated optical flow map can distinguish the independent motion features of head movements and limb movements, providing structured input data for feature extraction of subsequent video branches.

[0043] Based on the displacement changes of the keypoint coordinate set between adjacent video frames, the horizontal and vertical optical flow components of the head keypoints and the limb keypoints are generated. The calculation process of the optical flow components is defined by the following formula:

[0044]

[0045] in, Let i be the horizontal optical flow component of the i-th keypoint in frame t. Let i be the vertical optical flow component of the i-th keypoint in frame t. , Let be the two-dimensional pixel coordinates of the i-th keypoint in frame t. , Let be the two-dimensional pixel coordinates of the i-th keypoint in frame t-1. The horizontal optical flow component corresponds to the horizontal displacement of the keypoint, with positive values ​​indicating movement to the right and negative values ​​indicating movement to the left; the vertical optical flow component corresponds to the vertical displacement of the keypoint, with positive values ​​indicating downward movement and negative values ​​indicating upward movement.

[0046] The horizontal and vertical optical flow components of the head keypoints are concatenated to generate a head optical flow tensor. The number of channels in the head optical flow tensor is twice the number of head keypoints, and each keypoint corresponds to two optical flow component channels (horizontal and vertical). Similarly, the horizontal and vertical optical flow components of the limb keypoints are concatenated to generate a limb optical flow tensor. The number of channels in the limb optical flow tensor is twice the number of limb keypoints. The head and limb optical flow tensors are then concatenated along the channel dimension, and the concatenated tensor is adjusted to match the spatial dimension of the video frame to generate an optical flow graph. Each channel of the optical flow graph corresponds to one optical flow component of a keypoint, and the value at each spatial location corresponds to the motion displacement of that keypoint, thus fully preserving the spatiotemporal motion information of the pet's head and limb movements.

[0047] During the extraction of the coordinate set of joints in the head region and multiple limb regions, when a target joint in the head region or multiple limb regions is detected to be occluded in the current video frame, the historical coordinate trajectory of the target joint in the preceding consecutive video frames is extracted. The occlusion criteria for a target joint are: the detection confidence of the target joint is lower than a preset confidence threshold, or the coordinates of the target joint exceed the effective image range of the video frame. The extracted historical coordinate trajectory is the optimal coordinate sequence of the target joint in the preceding 10 consecutive video frames, which completely records the motion trajectory of the target joint before occlusion.

[0048] The extracted historical coordinate trajectory is input into a Kalman filter, and the predicted coordinates of the target joint in the current video frame are calculated using the Kalman filter's state prediction equation. The Kalman filter employs a uniform motion model, and the state vector contains the two-dimensional coordinates of the target joint and its horizontal and vertical motion velocities. The state prediction equation is defined by the following formula:

[0049]

[0050] in, Let be the state prediction vector for frame t. Let A be the optimal state estimation vector for frame t-1, A be the state transition matrix, and B be the control input matrix. To control the input vector, The covariance matrix of the predicted state, Let be the covariance matrix of the optimal state estimate in frame t-1, and Q be the process noise covariance matrix. State vector Where x and y are the two-dimensional pixel coordinates of the target joint. , Let A be the velocity of the target joint in the horizontal and vertical directions. The state transition matrix A is a 4×4 matrix, corresponding to the state transition rules of the uniform motion model.

[0051] The Kalman filter determines whether the distance between the predicted coordinates and the coordinates of the visible adjacent joints of the target joint in the current video frame is within a preset distance threshold. Visible adjacent joints are those with a skeleton topological connection to the target joint and whose detection confidence is higher than a preset confidence threshold. The coordinate distance is calculated using Euclidean distance, and the preset distance threshold is set based on the physiological structural dimensions of the pet's skeleton. If the distance between the predicted coordinates and the coordinates of the visible adjacent joints is within the preset distance threshold, the predicted coordinates are used as the coordinates of the target joint in the current video frame, and the joint coordinate set is updated. If the distance exceeds the preset distance threshold, the predicted coordinates are corrected based on the coordinates of the visible adjacent joints and the skeleton topological connection, and then the joint coordinate set is updated. The state update process of the Kalman filter is defined by the following formula:

[0052]

[0053]

[0054] in, Let H be the Kalman gain of frame t, H be the observation matrix, and R be the observation noise covariance matrix. Let be the observation vector of the t-th frame. Let I be the optimal state estimation vector for frame t, and let I be the identity matrix. Let be the covariance matrix of the optimal state estimate for frame t.

[0055] In this embodiment, by extracting the skeleton and dividing the regions of the pet video frames, optical flow tensors that distinguish between head key points and limb key points are generated and fused to obtain a structured optical flow map. At the same time, when the key points are occluded, coordinate prediction and updates are performed by combining historical trajectories and constraints of adjacent key points through a Kalman filter, which maintains the integrity and coherence of the video branch input data, avoids the loss of motion features due to occlusion, and provides a stable input for the subsequent extraction of spatiotemporal motion features.

[0056] Please refer to the attached document. Figure 3In one optional embodiment, the Mel spectrogram is segmented along the time dimension to obtain a spectral frame sequence. The frame length and frame shift of the segmented processing are consistent with the framing parameters of the audio stream preprocessing process, ensuring that the time step of the spectral frame sequence is aligned with the temporal progression of the audio stream. Each spectral frame corresponds to the frequency domain feature vector of a time step in the Mel spectrogram. The spectral frame sequence is input into a one-dimensional convolutional network containing a first convolutional layer, a second convolutional layer, and a first pooling layer. In the first convolutional layer, a first-size convolutional kernel is used to extract local frequency domain features. The kernel size of the first convolutional layer is 3, the stride is 1, and the padding is 1. The convolutional kernel slides along the time dimension to extract the local frequency domain correlation features of each time step and its adjacent time steps, capturing the spectral detail features of a single frame or two adjacent frames. In the second convolutional layer, cross-frame frequency domain features are extracted using a second-sized convolutional kernel. The second convolutional layer is configured as a multi-scale dilated convolutional layer containing three parallel dilated convolutional branches. The three parallel dilated convolutional branches have a first dilation rate, a second dilation rate, and a third dilation rate, respectively. The kernel size of all branches is consistent. By setting different dilation rates, the receptive field of the convolutional layer is expanded without increasing the number of kernel parameters, thereby extracting cross-frame frequency domain features at different time scales.

[0057] Table 3. Correspondence between Multi-Scale Dilated Convolution Branch Configuration and Feature Extraction Scale

[0058] The multi-scale dilated convolution branch configuration defined in this table is used to achieve cross-frame frequency domain feature extraction at different time scales. By setting different dilation rates, the receptive field of the convolutional layer can be expanded without increasing the number of convolutional kernel parameters, while preserving the vocal features at different time scales and avoiding the loss of detailed features caused by single-scale convolution.

[0059] The local frequency domain features output from the first convolutional layer are input into three parallel dilated convolutional branches. The branch with the first dilation rate extracts cross-frame frequency domain features at the first time scale, the branch with the second dilation rate extracts cross-frame frequency domain features at the second time scale, and the branch with the third dilation rate extracts cross-frame frequency domain features at the third time scale. These cross-frame frequency domain features at the first, second, and third time scales are then concatenated along the channel dimension to generate multi-scale concatenated features. The generation process of multi-scale concatenated features is defined by the following formula:

[0060] in, This is a multi-scale splicing feature. The output feature of the first void ratio branch, This is the output feature of the second void ratio branch. The output features of the third porosity branch are represented by Concat, which is a concatenation operation along the channel dimension. The generated multi-scale concatenated features are input into the first pooling layer, which is a one-dimensional average pooling layer with a pooling window size of 2 and a stride of 2. The multi-scale concatenated features are downsampled to reduce the temporal dimension length of the features while retaining key temporal acoustic features. The final output is a temporal acoustic feature with a fixed dimension.

[0061] The optical flow map is divided into a sequence of time windows, each 8 frames long, with 4 overlapping frames between adjacent time windows to ensure temporal continuity and prevent loss of motion features. The time window sequence is input into a 3D convolutional network containing a 3D convolutional kernel and a second pooling layer. The 3D convolutional kernel performs sliding convolution calculations on the time window sequence in both the temporal and spatial dimensions. The kernel size is (3,3,3), corresponding to 3 frames in the temporal dimension and 3×3 pixels in the spatial dimension, with a stride of (1,1,1) and padding of (1,1,1), simultaneously extracting motion change features in the temporal dimension and keypoint distribution features in the spatial dimension. The sliding convolution calculation results are then dimensionality-reduced using a second pooling layer. This second pooling layer employs a 3D max pooling layer with a pooling window size of (2,2,2) and a stride of (2,2,2), reducing the dimensionality of the convolution output features, decreasing computational cost, and preserving spatiotemporal motion features. The final output is a fixed-dimensional spatiotemporal motion feature.

[0062] During the feature mapping calculation of the cross-attention layer, the Mel spectrum is divided into frequency bands to obtain high-frequency, mid-frequency, and low-frequency sub-bands. The frequency band division is based on the linear frequency range corresponding to the Mel frequencies: the low-frequency sub-band corresponds to the linear frequency range of 0-1000Hz, the mid-frequency sub-band to the linear frequency range of 1000Hz-4000Hz, and the high-frequency sub-band to the linear frequency range of 4000Hz-8000Hz. These three sub-bands cover the complete effective frequency range of the pet vocalization signal. The average energy values ​​of the high-frequency, mid-frequency, and low-frequency sub-bands are calculated. The average energy value is the average of the spectral energy at all time steps within the corresponding sub-band, reflecting the energy distribution of the pet vocalization signal in different frequency bands.

[0063] Based on the proportional relationship between the average energy values ​​of the high-frequency sub-band, the mid-frequency sub-band, and the low-frequency sub-band, the target number of partitions for multiple query sub-matrices is determined. The target number of partitions corresponds to the number of attention heads in the cross-attention layer. When the average energy value of the low-frequency sub-band accounts for more than 60% of the total average energy value of the three sub-bands, the target number of partitions is set to 8; when the average energy value of the low-frequency sub-band accounts for between 40% and 60%, the target number of partitions is set to 6; and when the average energy value of the low-frequency sub-band accounts for less than 40%, the target number of partitions is set to 4.

[0064] The query matrix is ​​uniformly partitioned according to the target number of partitions to obtain multiple query submatrices. The feature dimension of each query submatric is the total feature dimension of the query matrix divided by the target number of partitions. The key matrix is ​​also uniformly partitioned according to the target number of partitions to obtain multiple key submatrices. Similarly, the value matrix is ​​uniformly partitioned according to the target number of partitions to obtain multiple value submatrices. For each query submatric, the product of the query submatric and the transpose of the corresponding key submatric is calculated to obtain the attention score matrix. The attention score matrix is ​​then normalized, and multiplied by the corresponding value submatric to obtain multiple attention output submatrices. These multiple attention output submatrices are concatenated to obtain the multimodal joint representation vector.

[0065] In this embodiment, multi-scale dilated convolutional layers are used to extract cross-frame frequency domain features at different time scales, fully preserving the multi-scale temporal features of pet vocalization signals. Three-dimensional convolution is used to extract joint features of the spatiotemporal dimensions of optical flow maps. At the same time, the number of query matrix partitions of the cross-attention layer is adaptively determined according to the frequency band energy distribution of the audio signal, enabling the network to dynamically adjust the computation path of feature mapping according to the physical properties of the input signal, thereby improving the accuracy of multimodal feature alignment.

[0066] In another optional embodiment, the frame signal power of each frame of audio signal in the pet audio stream is calculated. The frame signal power is the sum of the squares of all sampling points of a single frame audio signal divided by the number of sampling points in a single frame, reflecting the energy intensity of the single frame audio signal. The background noise power of the silent segments in the pet audio stream is extracted. Voice endpoint detection is performed on the pet audio stream using a dual-threshold endpoint detection algorithm. Based on the short-time energy and zero-crossing rate of the audio signal, the pet audio stream is divided into audible and silent segments. The audible segments correspond to the time periods when the pet vocalizes, and the silent segments correspond to the time periods when there is no pet vocalization and ambient noise. Multiple frames of silent audio signals within an initial time window of 500ms are extracted from the silent segments. This initial time window contains consecutive silent frames. The initial average power of the multiple frames of silent audio signals is calculated and used as the initial background noise power.

[0067] During the continuous acquisition of the pet audio stream, multiple frames of updated silence audio signals within subsequent time windows are extracted. The length of each subsequent time window is 200ms, and updates are performed every 100ms to ensure that the background noise power follows changes in ambient noise in real time. The average updated power of the multiple frames of updated silence audio signals is calculated, and the moving average of the average updated power and the initial background noise power is calculated. This moving average is used as the updated background noise power, replacing the initial background noise power, to calculate the initial signal-to-noise ratio of subsequent audio frames. The moving average update process for background noise power is defined by the following formula:

[0068] in, Let be the background noise power updated in the t-th time window. This is the sliding smoothing coefficient, with a value ranging from 0.8 to 0.95. Let be the background noise power in the (t-1)th time window. Let be the updated average power of the silent audio signal within the t-th time window. This moving average update process can smooth out random fluctuations in ambient noise while tracking slow changes in ambient noise in real time, ensuring the accuracy of background noise power calculation.

[0069] Based on the difference between the frame signal power and the background noise power, the initial signal-to-noise ratio (SNR) of each frame of audio signal is calculated. This initial SNR is then input into a fully connected layer containing an activation function, mapping it to a preset value range of 0-1 to generate gating coefficients. The fully connected layer performs a linear transformation on the input initial SNR, and then uses a sigmoid activation function to map the output of the linear transformation to the 0-1 range, generating the gating coefficients for the corresponding frame. A moving average is then applied to the gating coefficients of all frames in the entire audio stream to obtain the final gating coefficients, thus avoiding abrupt changes in the fusion weights caused by fluctuations in the SNR of a single frame.

[0070] Table 4. Mapping Relationship between Audio Signal-to-Noise Ratio Range and Gating Coefficient

[0071] The mapping relationship between the signal-to-noise ratio range and the gating coefficient defined in this table is used to standardize the weighted fusion logic of the spatiotemporal gating fusion layer. When the audio signal-to-noise ratio is high, the weight of the audio component is increased to make full use of the clear sound signal features; when the audio signal-to-noise ratio is low, the weight of the video component is increased to use body movement features to supplement the noise interference of the audio signal, thereby achieving adaptive weighted fusion of multimodal features.

[0072] The audio and video components in the multimodal joint representation vector are weighted and fused using generated gating coefficients. The gating coefficients are multiplied by the audio component in the multimodal joint representation vector to obtain the weighted audio features. The difference between the value 1 and the gating coefficients is multiplied by the video component in the multimodal joint representation vector to obtain the weighted video features. The weighted audio features and the weighted video features are then vector-summed to generate the target fusion vector. This weighted fusion process adaptively adjusts the weights of the two modal features based on the quality of the audio signal, suppresses environmental noise interference with the audio features, and blocks the activation of invalid features corresponding to misalignments in the feature space, ensuring that the audio and video features in the target fusion vector are at the same time reference.

[0073] Please refer to the attached document. Figure 6The target fusion vector is input into the decoder in the deep neural network. This target fusion vector serves as the initial hidden state input to the first self-attention layer of the decoder, where self-attention calculations are performed to generate decoder self-attention features. The decoder employs a Transformer-based autoregressive decoding structure, containing multiple cascaded decoding layers. Each decoding layer sequentially includes a masked self-attention layer, a cross-attention layer, and a feedforward neural network layer. The masked self-attention layer uses a lower triangular mask matrix to mask the attention score matrix, setting the attention score for future positions to negative infinity. This prevents information leakage at future times during decoding, ensuring the legitimacy of the autoregressive decoding. The query vector, key vector, and value vector of the masked self-attention layer all originate from the output of the previous layer of the decoder, used to capture the temporal dependencies between the generated intent text.

[0074] The decoder's self-attention features and the target fusion vector are input into the decoder's cross-attention layer. Using the decoder's self-attention features as the decoding query vector and the target fusion vector as the decoding key vector, cross-attention calculation is performed to generate the decoder's joint features. This cross-attention calculation process aligns the generated text features with the multimodal fusion features, ensuring that each generated word corresponds to the pet's audio and video features, thus improving the accuracy of the decoding results.

[0075] The decoder's joint features are input into a feedforward neural network layer for nonlinear transformation. The feedforward neural network layer contains two cascaded fully connected layers, with a linear rectified activation function placed between the two fully connected layers. The computation process of the feedforward neural network layer is defined by the following formula:

[0076] in, , These are the weight parameters of the fully connected layer. , These are the bias parameters for the fully connected layer. The output vector after nonlinear transformation is input to the normalization layer for normalization, and then the normalized vector is input to the softmax classification layer. The output vector after nonlinear transformation is mapped to a word probability distribution, which is calculated using the following formula:

[0077] in, Let i be the output probability of the i-th word in the vocabulary. Let be the value at the i-th position in the output vector of the feedforward neural network layer, and N be the total number of words in the preset pet intention vocabulary, which includes common pet intention words and their corresponding combinations. The word with the highest probability value is used as the output word at the current time step, and the output word at the current time step is used as the input for decoding at the next time step. This decoding process is repeated until a preset end symbol is generated. All generated words are then concatenated in order to output the pet intention text.

[0078] In this embodiment, the background noise power is updated in real time using a sliding mean, enabling accurate calculation of the signal-to-noise ratio (SNR). Adaptive gating coefficients are generated based on the SNR, and the audio and video components of the multimodal joint representation vector are weighted and fused to suppress environmental noise interference with the audio features. Simultaneously, a complete Transformer decoder structure is used to accurately map the multimodal fusion features to the pet's intention text, ensuring that the output intention text accurately reflects the pet's true expression. Figure 1 To.

Claims

1. A pet interaction intelligent translation method based on deep neural networks, characterized in that, include: Acquire synchronously collected pet audio stream and pet video stream, convert the pet audio stream into a Mel spectrogram, and convert the pet video stream into an optical flow map containing head key points and limb key points; The Mel spectrogram is input into the audio branch encoder of the deep neural network, and temporal acoustic features are extracted through one-dimensional convolution. The optical flow map is input into the video branch encoder of the deep neural network, and spatiotemporal motion features are extracted through three-dimensional convolution. The temporal acoustic features and the spatiotemporal motion features are input into the cross-attention layer of the deep neural network. The temporal acoustic features are used as the query vector, and the spatiotemporal motion features are used as the key vector and value vector to perform feature mapping calculation, and output a multimodal joint representation vector. The multimodal joint representation vector is input into the spatiotemporal gated fusion layer in the deep neural network. The gate coefficient is calculated based on the signal-to-noise ratio of the pet audio stream. The audio component and video component in the multimodal joint representation vector are weighted and fused using the gate coefficient to generate the target fusion vector. The target fusion vector is input into the decoder in the deep neural network to output the pet intention text.

2. The intelligent translation method for pet interaction based on deep neural networks according to claim 1, characterized in that, The step of converting the pet video stream into an optical flow map containing head key points and limb key points includes: extracting the skeleton from the video frames in the pet video stream to obtain an initial skeleton topology map; In the initial skeleton topology map, the head region and multiple limb regions are divided, and the set of joint coordinates of the head region and multiple limb regions is extracted; Based on the displacement change of the key point coordinate set between adjacent video frames, the horizontal and vertical optical flow components of the head key points are generated, as well as the horizontal and vertical optical flow components of the limb key points are generated. The horizontal and vertical optical flow components of the key points of the head are concatenated to generate the head optical flow tensor. The horizontal and vertical optical flow components of the key points of the limbs are concatenated to generate the limb optical flow tensor. The head optical flow tensor and the limb optical flow tensor are fused to generate the optical flow map.

3. The intelligent translation method for pet interaction based on deep neural networks according to claim 1, characterized in that, The steps for extracting temporal acoustic features through one-dimensional convolution include: performing frame-segmentation processing on the Mel spectrogram along the time dimension to obtain a spectral frame sequence; The spectral frame sequence is input into a one-dimensional convolutional network containing a first convolutional layer, a second convolutional layer, and a first pooling layer. Local frequency domain features are extracted in the first convolutional layer using a first-size convolutional kernel, and cross-frame frequency domain features are extracted in the second convolutional layer using a second-size convolutional kernel. The temporal acoustic features are obtained by downsampling the cross-frame frequency domain features through the first pooling layer. The steps for extracting spatiotemporal motion features through 3D convolution include: dividing the optical flow map into a sequence of time windows; inputting the sequence of time windows into a 3D convolutional network containing a 3D convolutional kernel and a second pooling layer; performing sliding convolution calculations on the sequence of time windows in both the time and spatial dimensions using the 3D convolutional kernel; and reducing the dimensionality of the sliding convolution calculation results using the second pooling layer to obtain the spatiotemporal motion features.

4. The intelligent translation method for pet interaction based on deep neural networks according to claim 1, characterized in that, The step of performing feature mapping calculation using the temporal acoustic features as query vectors and the spatiotemporal motion features as key and value vectors includes: performing linear mapping on the temporal acoustic features to generate a query matrix, and performing linear mapping on the spatiotemporal motion features to generate a key matrix and a value matrix. The query matrix is ​​divided into multiple query sub-matrices, the key matrix is ​​divided into multiple key sub-matrices, and the value matrix is ​​divided into multiple value sub-matrices; For each query submatrix, the product of the query submatrix and the transpose of the corresponding key submatrix is ​​calculated to obtain the attention score matrix. The attention score matrix is ​​normalized, and the normalized attention score matrix is ​​multiplied by the corresponding value submatrix to obtain multiple attention output submatrixes. The multiple attention output submatrixes are concatenated to obtain the multimodal joint representation vector.

5. The intelligent translation method for pet interaction based on deep neural networks according to claim 1, characterized in that, The step of calculating the gating coefficient based on the signal-to-noise ratio of the pet audio stream includes: calculating the frame signal power of each frame of audio signal in the pet audio stream, and extracting the background noise power of the silent segment in the pet audio stream; The initial signal-to-noise ratio of each frame audio signal is calculated based on the difference between the frame signal power and the background noise power. The initial signal-to-noise ratio is input into a fully connected layer containing an activation function, and the initial signal-to-noise ratio is mapped to a preset numerical range to generate the gating coefficients; The step of weighted fusion of audio and video components in the multimodal joint representation vector using the gating coefficient includes: multiplying the gating coefficient by the audio component in the multimodal joint representation vector, multiplying the difference between the value 1 and the gating coefficient by the video component in the multimodal joint representation vector, and adding the two multiplication results into a vector sum to generate the target fusion vector.

6. The intelligent translation method for pet interaction based on deep neural networks according to claim 1, characterized in that, The step of inputting the target fusion vector into the decoder in the deep neural network includes: inputting the target fusion vector as the initial hidden state into the first self-attention layer of the decoder, performing self-attention calculation on the initial hidden state, and generating decoder self-attention features; The decoder self-attention features and the target fusion vector are input into the cross-attention layer of the decoder. The decoder self-attention features are used as the decoding query vector, and the target fusion vector is used as the decoding key vector and the decoding value vector to perform cross-attention calculation to generate decoder joint features. The decoder's joint features are input into a feedforward neural network layer for nonlinear transformation. The output vector after nonlinear transformation is input into a softmax classification layer. The output vector after nonlinear transformation is mapped to a word probability distribution. The word with the highest probability value is output as the pet intent text.

7. The intelligent translation method for pet interaction based on deep neural networks according to claim 2, characterized in that, In the process of extracting the set of joint coordinates of the head region and the multiple limb regions, the following is included: when the target joint in the head region or the multiple limb regions is detected to be occluded in the current video frame, the historical coordinate trajectory of the target joint in the preceding multiple consecutive video frames of the current video frame is extracted. The historical coordinate trajectory is input into a Kalman filter, and the predicted coordinates of the target key point in the current video frame are calculated using the state prediction equation of the Kalman filter. Determine whether the distance between the predicted coordinates and the coordinates of the visible adjacent joints of the target joint in the current video frame is within a preset distance threshold range. If it is within the preset distance threshold range, then use the predicted coordinates as the coordinates of the target joint in the current video frame and update the joint coordinate set.

8. The intelligent translation method for pet interaction based on deep neural networks according to claim 3, characterized in that, The step of extracting cross-frame frequency domain features using a second-size convolutional kernel in the second convolutional layer includes: configuring the second convolutional layer as a multi-scale dilated convolutional layer containing three parallel dilated convolutional branches, wherein the three parallel dilated convolutional branches have a first dilation rate, a second dilation rate and a third dilation rate, respectively; The local frequency domain features are respectively input into the three parallel dilated convolution branches. Cross-frame frequency domain features at the first time scale are extracted through the branch with the first dilation rate, cross-frame frequency domain features at the second time scale are extracted through the branch with the second dilation rate, and cross-frame frequency domain features at the third time scale are extracted through the branch with the third dilation rate. The cross-frame frequency domain features of the first time scale, the cross-frame frequency domain features of the second time scale, and the cross-frame frequency domain features of the third time scale are concatenated in the channel dimension to generate multi-scale concatenated features, and the multi-scale concatenated features are input into the first pooling layer.

9. The intelligent translation method for pet interaction based on deep neural networks according to claim 4, characterized in that, The step of dividing the query matrix into multiple query sub-matrices includes: dividing the Mel spectrogram into frequency bands to obtain high-frequency sub-bands, mid-frequency sub-bands and low-frequency sub-bands; Calculate the average energy value of the high-frequency sub-band, the average energy value of the mid-frequency sub-band, and the average energy value of the low-frequency sub-band; The target number of partitions for the multiple query sub-matrices is determined based on the proportional relationship between the average energy value of the high-frequency sub-band, the average energy value of the mid-frequency sub-band, and the average energy value of the low-frequency sub-band. The query matrix is ​​uniformly divided according to the target number of partitions to obtain the plurality of query submatrices; the key matrix is ​​uniformly divided according to the target number of partitions to obtain the plurality of key submatrices; and the value matrix is ​​uniformly divided according to the target number of partitions to obtain the plurality of value submatrices.

10. The intelligent translation method for pet interaction based on deep neural networks according to claim 5, characterized in that, The step of extracting the background noise power of the silent segment in the pet audio stream includes: performing voice endpoint detection on the pet audio stream and dividing the pet audio stream into a sound segment and the silent segment; Extract multiple frames of silent audio signals within the initial time window of the silent segment, calculate the initial average power of the multiple frames of silent audio signals, and use the initial average power as the initial background noise power; During the continuous acquisition of the pet audio stream, multiple frames of updated silence audio signals within subsequent time windows are extracted, and the average update power of the multiple frames of updated silence audio signals is calculated. Calculate the moving average of the updated average power and the initial background noise power, use the moving average as the updated background noise power, replace the initial background noise power with the updated background noise power, and calculate the initial signal-to-noise ratio of subsequent audio frames.

Images