Multimodal audio-visual media remote interaction system

Abstract

The present invention discloses a multimodal audio-visual media remote interaction system, which relates to the field of interactive technology. The present invention breaks away from the way of realizing interaction by preset rules, but instead realizes multi-dimensional quantitative analysis of user interaction behavior by integrating visual data, audio data and body movement data, and generates interaction effect evaluation values, thereby providing a scientific basis for the formulation of optimal interaction strategies. The fusion of multimodal data can fully capture the user's interaction status, avoid information loss or deviation that may be caused by a single data source, and significantly improve the accuracy and reliability of the evaluation results; secondly, the complex behavior data is converted into quantifiable interaction effect evaluation values, which facilitates the system to quickly respond to and optimize user needs, thereby improving the intelligence level of the interaction process.

Description

Technical Field

[0001] The present invention relates to the field of interactive technology, and in particular to a multimodal audio-visual media remote interactive system. Background Art

[0002] With the rapid development of information technology, remote interaction systems have been widely used in fields such as education, healthcare, entertainment, and office work. Traditional remote interaction systems typically rely on a single modality of data (such as audio or video). This limitation makes it difficult for the system's interactive effects to meet the diverse needs of users. For example, in remote meetings, relying solely on voice communication may not accurately convey emotional information; while in remote teaching, the lack of body movement capture can lead to insufficient interactivity.

[0003] In recent years, the rise of multimodal technologies has brought new opportunities for remote interaction systems. By integrating visual data (such as facial expressions and eye direction), audio data (such as voice intonation and speech rate), and body movement data (such as gestures and posture), multimodal audiovisual media remote interaction systems can more comprehensively capture user interaction behaviors, thereby improving the system's intelligence and user experience.

[0004] A single metric cannot fully reflect the user's actual interactive experience, primarily because remote interactions involve multiple complex factors. A single metric can only assess a specific dimension and cannot encompass the user's comprehensive emotional, behavioral, and cognitive needs. Furthermore, the user's interactive experience is often influenced by multiple modal data. For example, in a remote meeting, clear speech may appear monotonous due to a lack of body movement feedback, while high-quality video may lose its appeal due to a bland tone of voice. Therefore, relying solely on a single metric for evaluation leads to a one-sided understanding of the user experience, which in turn affects the optimization direction and ultimate effectiveness of the system. Summary of the Invention

[0005] The purpose of the present invention is to provide a multimodal audio-visual media remote interaction system to solve the problem that relying on a single indicator for evaluation will lead to a one-sided understanding of the user experience, thereby affecting the optimization direction and final effect of the system.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal audio-visual media remote interaction system, comprising the following modules:

[0007] Visual data acquisition module: used to obtain real-time visual data of users;

[0008] Audio data acquisition module: used to obtain real-time audio data of users;

[0009] Body movement data acquisition module: used to obtain the user's real-time body movement data;

[0010] Preprocessing module: connected to the visual data acquisition module, audio data acquisition module and body movement data acquisition module, used to preprocess real-time visual data, real-time audio data and real-time body movement data;

[0011] Initial interaction strategy generation module: This module is connected to the preprocessing module and is used to input the preprocessed real-time visual data, real-time audio data, and real-time body movement data into the pre-training model to generate the initial interaction strategy.

[0012] Optimized interaction strategy generation module: used to obtain the user behavior pattern matrix and the interaction effect evaluation adjustment amount, calculate the comprehensive matrix based on the user behavior pattern matrix and the interaction effect evaluation adjustment amount, optimize the initial interaction strategy based on the comprehensive matrix, and generate the optimized interaction strategy.

[0013] Furthermore, the interaction strategy generation module is optimized, including the following submodules:

[0014] User behavior pattern matrix construction submodule: used to construct the user behavior pattern matrix through the user's historical visual data, historical audio data, and historical body movement data;

[0015] Interaction effect evaluation value prediction submodule: used to build a deep learning model and train the deep learning model to obtain the optimal model parameters; obtain the real-time user behavior pattern matrix and input it into the deep learning model for prediction. Based on the model's optimal parameters and the real-time facial expression feature values, real-time voice intonation feature values, and real-time body movement feature values ​​extracted from the real-time user behavior pattern matrix, the interaction effect evaluation value is calculated;

[0016] Preset submodule: used to preset an ideal evaluation value of the interaction effect and an optimization threshold;

[0017] Interaction effect judgment submodule: Calculates the interaction effect evaluation difference based on the interaction effect evaluation value and the ideal interaction effect evaluation value; compares the interaction effect evaluation difference with the optimization threshold; if the interaction effect evaluation value is less than the optimization threshold, generates an optimized interaction strategy based on the interaction effect evaluation value; if the interaction effect evaluation value is greater than or equal to the optimization threshold, transmits the interaction effect evaluation value to the interaction effect evaluation adjustment amount calculation submodule;

[0018] Interaction effect evaluation adjustment calculation submodule: used to calculate the sensitivity of real-time facial expression feature values, real-time voice intonation feature values, and real-time body movement feature values ​​to the interaction effect evaluation value based on the partial derivative formula, and construct a sensitivity matrix. The interaction effect evaluation adjustment value is calculated based on the sensitivity matrix and the difference between the interaction effect evaluation value;

[0019] Comprehensive matrix construction sub-module: used to obtain a comprehensive matrix based on the real-time user behavior pattern matrix and the interactive effect evaluation adjustment amount, and input it into the deep learning model to predict the interactive effect evaluation value again; transmit the currently predicted interactive effect evaluation value to the interactive effect evaluation adjustment amount calculation sub-module, and stop predicting until the currently predicted interactive effect evaluation value is less than the optimization threshold;

[0020] Optimal interaction strategy generation sub-module: used to receive the optimized interaction strategy and generate candidate interaction strategies, calculate the final scores for the optimized interaction strategy and the candidate interaction strategies, and screen out the optimal interaction strategy based on the final scores.

[0021] Furthermore, the optimal interaction strategy generation sub-module includes the following units:

[0022] Candidate interaction strategy generation unit: used to calculate the cosine similarity between the comprehensive matrix corresponding to the optimized interaction strategy or the real-time user behavior pattern matrix and the user behavior pattern matrix; regard the user behavior pattern matrix with a cosine similarity greater than k as the candidate matrix, where 0 < k < 1; calculate the feature mean based on the features of the candidate matrix, and add the feature mean to the optimized interaction strategy to obtain the candidate interaction strategy;

[0023] Priority weight calculation unit; used to calculate the priority weight of the candidate interaction strategy according to the cosine similarity;

[0024] Final score calculation unit: used to calculate the final scores of the candidate interaction strategy and the optimized interaction strategy according to the cosine similarity and the priority weight;

[0025] Optimal interaction strategy screening unit: used to regard the optimized interaction strategy or the candidate interaction strategy with the highest final score as the optimal interaction strategy.

[0026] Furthermore, the user behavior pattern matrix includes a facial expression feature vector, a voice intonation feature vector, and a body movement feature vector, and the facial expression feature vector, the voice intonation feature vector, and the body movement feature vector form a 3x3 user behavior pattern matrix.

[0027] [[ID=​​​

[0029]

[0030] Among them, E represents the interaction effect evaluation value, f face，i represents the ith facial expression feature value in the real-time user behavior pattern matrix, f voice，i represents the i-th voice intonation feature value in the real-time user behavior pattern matrix, f gesture，i represents the i-th limb movement eigenvalue in the real-time user behavior pattern matrix, a, b, c, and d represent the optimal model parameters, n represents the total number of eigenvalues ​​in the real-time user behavior pattern matrix, log represents the logarithmic function, and e represents the exponential decay function.

[0031] Furthermore, the calculation formula for the interaction effect evaluation adjustment amount is:

[0032] ΔM=γ·ΔE·F;

[0033] Among them, ΔM represents the interaction effect evaluation adjustment amount, γ represents the adjustment coefficient, ΔE represents the interaction effect evaluation difference, and F represents the sensitivity matrix.

[0034] The technical effects and advantages of the present invention are as follows:

[0035] The present invention breaks away from the way of realizing interaction through preset rules. Instead, it integrates visual data, audio data and body movement data to realize multi-dimensional quantitative analysis of user interaction behavior and generate interaction effect evaluation values, thereby providing a scientific basis for the formulation of optimal interaction strategies. The fusion of multimodal data can fully capture the user's interaction status, avoid the information loss or deviation that may be caused by a single data source, and significantly improve the accuracy and reliability of the evaluation results; secondly, the complex behavior data is converted into quantifiable interaction effect evaluation values, which facilitates the system to quickly respond to and optimize user needs, thereby improving the intelligence level of the interaction process; finally, the optimal interaction strategy generated based on the evaluation value can dynamically adapt to the needs of different users and scene changes, while enhancing the user experience, it also provides strong support for the design of personalized services, and ultimately promotes the performance of the human-computer interaction system to a higher level. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of the structure of the multimodal audio-visual media remote interaction system provided in this embodiment. DETAILED DESCRIPTION

[0037] The following will be combined with the accompanying drawings to clarify the technical solutions in the embodiments of the present invention; although it is clear that the embodiments described are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] This embodiment offers high flexibility and scalability, and can be applied across multiple fields. In telemedicine, doctors can use the system to conduct remote video consultations with patients and utilize multimodal information for diagnosis. In education, teachers can use the system to provide remote lectures and real-time interaction for students, improving teaching effectiveness and the learning experience. It can also be used in remote conferencing. In short, multimodal audiovisual media remote interaction systems, with their rich interactive methods, high intelligence, and wide range of applications, are becoming an important development direction in the future of remote interaction.

[0039] This embodiment integrates visual data, audio data, and body movement data to achieve multi-dimensional quantitative analysis of user interaction behavior and generate interaction effect evaluation values, thereby providing a scientific basis for the formulation of optimal interaction strategies. The beneficial effects of this method are: first, the fusion of multimodal data can fully capture the user's interaction status, avoid information loss or bias that may be caused by a single data source, and significantly improve the accuracy and reliability of the evaluation results; second, complex behavioral data is converted into quantifiable interaction effect evaluation values, which facilitates the system to quickly respond to and optimize user needs, thereby improving the intelligence level of the interaction process; finally, the optimal interaction strategy generated based on the evaluation value can dynamically adapt to the needs of different users and changes in scenarios, enhancing the user experience while also providing strong support for the design of personalized services, ultimately pushing the performance of the human-computer interaction system to a higher level.

[0040] like Figure 1 As shown, this embodiment discloses a multimodal audio-visual media remote interaction system, including the following modules:

[0041] Visual data acquisition module: used to obtain real-time visual data of users.

[0042] A camera installed on the user side (such as a webcam or a mobile phone's front camera) is used to capture the user's RGB image. The RGB image includes visual information such as facial expressions, eye contact, and head movements. The camera can be a high-resolution RGB camera or a depth camera (such as an infrared camera) to support more complex visual feature extraction.

[0043] Real-time visual data includes real-time acquired facial key point coordinates, head rotation angle, eye tracking data, facial expression classification labels, etc.

[0044] Among them, the user's facial key points are extracted based on the pre-trained model (facial key points refer to the two-dimensional or three-dimensional coordinates of specific parts of the face (such as eyes, nose, mouth, eyebrows, etc.) in the image) to judge the user's expression state according to the relative position changes of the key points.

[0045] The head rotation angle describes the rotation of the user's head relative to the reference plane, which is used to determine whether the user is looking directly at the system or away from it, thereby judging the user's concentration or emotional state. It usually includes three dimensions:

[0046] Pitch angle: the angle at which the head moves up and down;

[0047] Yaw angle: the angle of the head turning left and right;

[0048] Roll angle: The angle at which the head tilts.

[0049] Eye tracking data records the user's eye movement trajectory and gaze position, which is used to determine the user's attention distribution and points of interest, and to assess the user's fatigue level through blinking frequency and pupil size. It mainly includes:

[0050] Gaze point coordinates: The (x, y) coordinates of the user's gaze on the screen. For example, a gaze point of (300, 200) indicates that the user is focusing on a certain area of ​​the screen.

[0051] Pupil size: reflects the user's concentration or emotional state. Example: Pupil diameter is 4mm.

[0052] Blink frequency: The number of blinks per unit time. Example: Blink frequency is 20 times per minute.

[0053] Audio data acquisition module: used to obtain real-time audio data of users.

[0054] The user's voice signal is captured in real time through the user's microphone, and the existing automatic speech recognition (ASR) technology is used to convert the audio signal into text content. Therefore, the real-time audio data includes the real-time acquired voice text, acoustic features, emotion labels, background noise information, etc.

[0055] Among them, signal processing techniques (such as short-time Fourier transform STFT and Mel-frequency cepstral coefficient MFCC) are used to extract the acoustic features of the speech signal, divide the speech signal into short-time frames, and calculate the speaking rate, volume and pitch characteristics of each frame.

[0056] Use noise detection algorithms (such as Wiener filtering and deep learning noise reduction models) to analyze the type and intensity of background noise.

[0057] Body movement data acquisition module: used to obtain the user's real-time body movement data.

[0058] The user's three-dimensional spatial information is captured using a depth camera (such as Microsoft Kinect or Intel RealSense). Accelerometers and gyroscopes worn on the user's body provide real-time limb motion data. This data includes the coordinates of key skeletal points, motion labels, limb motion trajectories, limb velocity, and acceleration.

[0059] Among them, the human pose estimation algorithm based on deep learning (such as OpenPose, AlphaPose, HRNet) is used to extract the coordinates of skeleton key points.

[0060] Use an action classification model based on time series analysis (such as LSTM, Transformer) to classify the user's body movements and obtain action categories.

[0061] Use Kalman filter or particle filter algorithm to smooth and track the user's limb movement trajectory.

[0062] Compute velocity and acceleration using numerical differentiation methods.

[0063] Preprocessing module: connected to the visual data acquisition module, audio data acquisition module and body movement data acquisition module, used to preprocess real-time visual data, real-time audio data and real-time body movement data.

[0064] Preprocessing of real-time visual data:

[0065] Denoising: Use median filtering or Gaussian filtering to remove random noise in the image.

[0066] Lighting Correction: Adjust image brightness and contrast to ensure clear images under different lighting conditions.

[0067] Resizing: Scale images to a uniform resolution to reduce computational effort.

[0068] Depth data calibration: Calibrate the depth image to eliminate sensor errors.

[0069] Preprocessing process of real-time audio data:

[0070] Noise reduction: Use Wiener filtering or deep learning noise reduction models to remove background noise.

[0071] Volume Normalization: Adjust the volume of the audio signal to a uniform range.

[0072] The preprocessing process of real-time limb motion data is as follows:

[0073] Coordinate data processing: downsampling and denoising of coordinate data.

[0074] Timestamp Alignment: Align acceleration and angular velocity data with the camera's timestamp.

[0075] Motion trajectory smoothing: Use Kalman filtering or particle filtering algorithms to smooth the motion trajectory of limb parts.

[0076] Initial interaction strategy generation module: connected to the preprocessing module, used to input the preprocessed real-time visual data, real-time audio data and real-time body movement data into the pre-training model to generate the initial interaction strategy.

[0077] The pre-trained model is a multi-classification problem. Historical visual data, audio data, and body movement data are used as the training set for the pre-trained model, resulting in a pre-trained model. Predicted labels include: manually calibrated facial expression type and key point coordinates (horizontally and vertically aligned eye gaze), manually calibrated emotion labels, speech rate and volume, manually calibrated action labels, and body movement trajectories. The pre-processed real-time visual data, audio data, and body movement data are fed into the pre-trained model to generate three prediction results, which are then fused to generate the initial interaction strategy.

[0078] Optimized interaction strategy generation module: used to obtain the user behavior pattern matrix and the interaction effect evaluation adjustment amount, calculate the comprehensive matrix based on the user behavior pattern matrix and the interaction effect evaluation adjustment amount, optimize the initial interaction strategy based on the comprehensive matrix, and generate the optimized interaction strategy.

[0079] Optimize the interaction strategy generation module, including the following sub-modules:

[0080] User behavior pattern matrix construction submodule: used to construct a user behavior pattern matrix based on the user's historical visual data, historical audio data, and historical body movement data; the user behavior pattern matrix includes facial expression feature vectors, voice intonation feature vectors, and body movement feature vectors. Facial expression feature vectors, voice intonation feature vectors, and body movement feature vectors form a 3x3 user behavior pattern matrix.

[0081] Interaction effect evaluation value prediction submodule: used to build a deep learning model and train the deep learning model to obtain the optimal model parameters; obtain the real-time user behavior pattern matrix and input it into the deep learning model for prediction. Based on the model's optimal parameters and the real-time facial expression feature values, real-time voice intonation feature values, and real-time body movement feature values ​​extracted from the real-time user behavior pattern matrix, the interaction effect evaluation value is calculated;

[0082] Construct a deep learning model and train the deep learning model to obtain the optimal model parameters, including: using the user behavior pattern matrix as the training set of the deep learning model, manually calibrating the historical interaction effect evaluation values ​​of the user behavior pattern matrix as prediction labels, and the input of the deep learning model is the real-time user behavior pattern matrix; constructing the loss function of the deep learning model based on the absolute error and relative error between the historical interaction effect evaluation values ​​and the predicted interaction effect evaluation values, and using the gradient descent method to minimize the loss function to obtain the optimal model parameters, thereby obtaining a trained deep learning model.

[0083] The loss function constructed based on the number of samples in the real-time user behavior pattern matrix is:

[0084]

[0085] Where L is the loss function, is the absolute error, ω1 is the weight coefficient of the absolute error, Represents the relative error, ω2 is the weight coefficient of the relative error, y j is the predicted value of the j-th sample (i.e., the interaction effect evaluation value predicted by the deep learning model), is the true value of the jth sample (i.e., the predicted label in the training set).

[0086] Absolute error ensures that the deep learning model accounts for the prediction bias of each sample, while relative error further ensures that the model maintains high accuracy even when predicting smaller values. This dual consideration enables the model to optimize prediction performance more evenly during training, especially when processing data with a large dynamic range, effectively avoiding model bias caused by a single error metric.

[0087] Based on this loss function, the gradient descent method is used to minimize the loss function to obtain the optimal model parameters, and the optimal model parameters are applied to the calculation process of the interaction effect evaluation value:

[0088]

[0089] Among them, E represents the interaction effect evaluation value, f face，i represents the ith facial expression feature value in the real-time user behavior pattern matrix, f voice，i represents the i-th voice intonation feature value in the real-time user behavior pattern matrix, f gesture，i represents the i-th limb movement eigenvalue in the real-time user behavior pattern matrix, a, b, c, and d represent the optimal model parameters, n represents the total number of eigenvalues ​​in the real-time user behavior pattern matrix, log represents the logarithmic function, and e represents the exponential decay function.

[0090] Since the data in the above formula contain different units, each data is standardized and normalized before prediction to obtain data of the same dimension, thereby eliminating the influence of the unit.

[0091] By optimizing the deep learning model's parameters and combining facial expression, voice, and body movement eigenvalues ​​to calculate the interaction effect evaluation value, a multi-dimensional quantification and precise description of the user's status can be achieved. The optimal parameters ensure the rationality of the eigenvalue weight distribution, allowing the features of different modalities to exert their due influence in the evaluation. The fusion calculation of multiple eigenvalues ​​fully considers the complexity and diversity of human interactive behavior, resulting in a more representative and comprehensive evaluation result. This method not only effectively reflects subtle changes in the interaction process but also provides a reliable numerical basis for understanding user emotions and behaviors, significantly enhancing the practical application value of the evaluation value.

[0092] Preset submodule: used to preset an ideal evaluation value of the interaction effect and an optimization threshold.

[0093] Collect a historical interaction dataset, which includes several samples. Each sample includes a user behavior pattern matrix and a corresponding interaction effect evaluation value. Arrange the interaction effect evaluation values ​​from low to high, define the samples corresponding to the bottom 5% of interaction effect evaluation values ​​as high-scoring samples, and take the mean of the interaction effect evaluation values ​​of the high-scoring samples as the ideal interaction effect evaluation value. The optimization threshold is calculated based on the ideal interaction effect evaluation value:

[0094] P = ideal × (1-c), where P is the optimization threshold, ideal is the ideal evaluation value of the interaction effect, and c is the tolerance coefficient, preferably 0.1.

[0095] Interaction effect judgment submodule: The interaction effect evaluation difference |E-ideal| is calculated based on the interaction effect evaluation value and the ideal interaction effect evaluation value; the interaction effect evaluation difference is compared with the optimization threshold. If the interaction effect evaluation value is less than the optimization threshold, an optimized interaction strategy is generated based on the interaction effect evaluation value; if the interaction effect evaluation value is greater than or equal to the optimization threshold, the interaction effect evaluation value is transmitted to the interaction effect evaluation adjustment amount calculation submodule.

[0096] Through the interaction effect evaluation value, the ideal interaction effect evaluation value and the optimization threshold, it is dynamically determined whether the current interaction state needs to be optimized, and the corresponding optimization strategy or adjustment amount is generated.

[0097] The optimization strategy is a specific improvement measure generated based on the initial strategy and the optimization threshold:

[0098] (1) Visual-related optimization

[0099] Facial expression adjustment: prompts the speaker to adjust his or her emotional state based on facial expression labels (such as smiling, frowning, etc.).

[0100] Eye gaze guidance: Use eye gaze horizontal and vertical coordinate data to remind participants to keep their eyes focused on the screen or a specific speaker.

[0101] (2) Audio-related optimization

[0102] Speech speed and volume adjustment: Based on speech speed and volume data, the speaker is advised to adjust the expression method.

[0103] Emotional label feedback: Combined with manually calibrated emotional labels, it provides emotional management suggestions for speakers.

[0104] (3) Body movement optimization

[0105] Action label guidance: Based on action labels (such as nodding, shaking head, waving, etc.), participants are encouraged to enhance interaction through body language.

[0106] Motion trajectory analysis: Track the movement trajectory of body parts to ensure that the movements are natural and consistent with the meeting context.

[0107] Interaction effect evaluation adjustment calculation submodule: used to calculate the sensitivity of real-time facial expression feature values, real-time voice intonation feature values, and real-time body movement feature values ​​to the interaction effect evaluation value based on the partial derivative formula, and construct a sensitivity matrix. The interaction effect evaluation adjustment value is calculated based on the sensitivity matrix and the difference between the interaction effect evaluation value;

[0108] The sensitivity values ​​are:

[0109]

[0110] is the sensitivity value of the i-th facial expression feature value to the interaction effect evaluation value in the real-time user behavior pattern matrix, is the sensitivity value of the i-th voice tone feature value to the interaction effect evaluation value in the real-time user behavior pattern matrix, is the sensitivity value of the i-th body movement feature value in the real-time user behavior pattern matrix to the interaction effect evaluation value, represents the partial derivative.

[0111] The sensitivity value of each eigenvalue in the real-time user behavior pattern matrix to the interaction effect evaluation value is combined into a sensitivity matrix.

[0112]

[0113] Among them, ΔM represents the adjustment amount of the interaction effect evaluation, γ represents the adjustment coefficient, ΔE represents the difference of the interaction effect evaluation, F represents the sensitivity matrix, max represents the maximum value operation, and max(F) represents the maximum eigenvalue in the sensitivity matrix.

[0114] By combining the difference of the interaction effect evaluation with the sensitivity matrix and adjusting it using the adjustment coefficient, the system can dynamically determine the optimal adjustment amount. It can not only quickly identify the gap between the current interaction state and the ideal state, but also perform targeted optimization according to the different sensitivities of each eigenvalue to the interaction effect. For example, when a certain eigenvalue has a greater impact on the interaction effect, its corresponding sensitivity value is higher, and the system will focus more on adjusting this feature to achieve a better interaction effect. This adjustment strategy based on sensitivity and difference enables the system to flexibly respond in a complex and changing interaction environment, ensuring that each adjustment is more accurate and effective, thereby significantly improving the overall interaction quality and user satisfaction.

[0115] Comprehensive matrix construction sub-module: It is used to obtain a comprehensive matrix based on the real-time user behavior pattern matrix and the adjustment amount of the interaction effect evaluation, and input it into the deep learning model to predict the interaction effect evaluation value again; transmit the currently predicted interaction effect evaluation value to the interaction effect evaluation adjustment amount calculation sub-module until the currently predicted interaction effect evaluation value is less than the optimization threshold, then stop the prediction.

[0116] Optimal interaction strategy generation sub-module: It is used to receive the optimized interaction strategy and generate candidate interaction strategies, calculate the final scores for the optimized interaction strategy and the candidate interaction strategies, and screen out the optimal interaction strategy based on the final scores.

[0117] The optimal interaction strategy generation sub-module includes the following units:

[0118] Candidate interaction strategy generation unit: It is used to calculate the cosine similarity between the comprehensive matrix corresponding to the optimized interaction strategy or the real-time user behavior pattern matrix and the user behavior pattern matrix; regard the user behavior pattern matrix with a cosine similarity greater than k as the candidate matrix, where 0 < k < 1, preferably, k is greater than or equal to 0.7; calculate the feature mean based on the features of the candidate matrix, and add the feature mean to the optimized interaction strategy to obtain the candidate interaction strategy.

[0119] Among them, the mean value of each dimension feature in the candidate matrix is added to the eigenvalue corresponding to the optimized strategy, thereby expanding the expression ability of the optimized strategy and compensating for potential deficiencies. There are at least two candidate strategies. After the feature mean adjusts the optimized interaction strategy, multiple candidate interaction strategies are obtained.

[0120] Priority weight calculation unit; It is used to calculate the priority weights of the candidate interaction strategies according to the cosine similarity.

[0121] f(s) = s2 ;

[0122] Where f(s) represents the priority weight, and s represents the cosine similarity between the comprehensive matrix corresponding to the optimized interaction strategy or the real-time user behavior pattern matrix and the user behavior pattern matrix.

[0123] Final score calculation unit: used to calculate the final score of the candidate interaction strategy based on cosine similarity and priority weight.

[0124] S=w1×s+w2×f(s), S is the final score, w1 represents the influence coefficient of cosine similarity, w1=0.1s, w2=0.1f(s), and w2 represents the influence coefficient of priority weight.

[0125] Optimal interaction strategy screening unit: used to select the optimized interaction strategy or candidate interaction strategy with the highest final score as the optimal interaction strategy.

[0126] During interaction policy optimization, although the optimized interaction policy is generated based on the predicted evaluation value, its cosine similarity with the target policy is not necessarily high. This may be because the optimized policy does not fully cover or match the feature requirements of the target policy. Therefore, introducing a candidate matrix and its mean to supplement the optimization policy is an effective approach. Specifically, the feature mean of the candidate matrix can be incorporated into the optimization policy through a weighted fusion method. The mean of each dimension of the candidate matrix is ​​proportionally added to the corresponding feature value of the optimization policy, thereby expanding the expressive power of the optimization policy and addressing potential shortcomings. When calculating the score, this fused policy can more comprehensively reflect the importance of multi-dimensional features, thereby improving consistency with the target policy. The significance of ultimately selecting the optimal interaction policy lies in not only integrating the advantages of the optimization policy but also enhancing its adaptability and robustness to the target requirements by introducing the candidate matrix mean. This ensures that the generated policy can achieve higher interaction effect evaluation values ​​in practical applications and better meet user needs and system goals.

[0127] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A multimodal audio-visual media remote interaction system, characterized in that: Includes the following modules: Visual data acquisition module: used to obtain real-time visual data of users; Audio data acquisition module: used to obtain real-time audio data of users; Body movement data acquisition module: used to obtain the user's real-time body movement data; Preprocessing module: connected to the visual data acquisition module, audio data acquisition module and body movement data acquisition module, used to preprocess real-time visual data, real-time audio data and real-time body movement data; Initial interaction strategy generation module: This module is connected to the preprocessing module and is used to input the preprocessed real-time visual data, real-time audio data, and real-time body movement data into the pre-training model to generate the initial interaction strategy. Optimized interaction strategy generation module: used to obtain the user behavior pattern matrix and the interaction effect evaluation adjustment amount, calculate the comprehensive matrix based on the user behavior pattern matrix and the interaction effect evaluation adjustment amount, optimize the initial interaction strategy based on the comprehensive matrix, and generate the optimized interaction strategy; Optimize the interaction strategy generation module, Includes the following submodules: User behavior pattern matrix construction submodule: used to construct the user behavior pattern matrix through the user's historical visual data, historical audio data, and historical body movement data; Interaction effect evaluation value prediction submodule: used to build a deep learning model and train the deep learning model to obtain the optimal model parameters; obtain the real-time user behavior pattern matrix and input it into the deep learning model for prediction. Based on the model's optimal parameters and the real-time facial expression feature values, real-time voice intonation feature values, and real-time body movement feature values ​​extracted from the real-time user behavior pattern matrix, the interaction effect evaluation value is calculated; Preset submodule: used to preset an ideal evaluation value of the interaction effect and an optimization threshold; Interaction effect judgment submodule: Calculates the interaction effect evaluation difference based on the interaction effect evaluation value and the ideal interaction effect evaluation value; compares the interaction effect evaluation difference with the optimization threshold; if the interaction effect evaluation difference is less than the optimization threshold, generates an optimized interaction strategy based on the interaction effect evaluation value; if the interaction effect evaluation difference is greater than or equal to the optimization threshold, transmits the interaction effect evaluation value to the interaction effect evaluation adjustment amount calculation submodule; Interaction effect evaluation adjustment amount calculation submodule: used to calculate the sensitivity values ​​of real-time facial expression feature values, real-time voice intonation feature values ​​and real-time body movement feature values ​​to the interaction effect evaluation value according to the partial derivative formula, and construct a sensitivity matrix, and calculate the interaction effect evaluation adjustment amount based on the sensitivity matrix and the interaction effect evaluation difference.

2. The multimodal audio-visual media remote interaction system according to claim 1, characterized in that: Comprehensive matrix construction submodule: This module is used to obtain a comprehensive matrix based on the real-time user behavior pattern matrix and the interaction effect evaluation adjustment value, and input it into the deep learning model to predict the interaction effect evaluation value again; the currently predicted interaction effect evaluation value is transmitted to the interaction effect evaluation adjustment calculation submodule until the current interaction effect evaluation difference is less than the optimization threshold, at which point the prediction is stopped; Optimal interaction strategy generation submodule: used to receive the optimized interaction strategy and generate candidate interaction strategies, calculate the final scores of the optimized interaction strategy and the candidate interaction strategies, and screen out the optimal interaction strategy based on the final scores.

3. The multimodal audio-visual media remote interaction system according to claim 2, characterized in that: The optimal interaction strategy generation submodule includes the following units: Candidate interaction strategy generation unit: used to calculate the cosine similarity between the comprehensive matrix corresponding to the optimized interaction strategy or the real-time user behavior pattern matrix and the user behavior pattern matrix; The user behavior pattern matrix with cosine similarity greater than k is taken as the candidate matrix, 0 <k<1; Calculate the feature mean based on the features of the candidate matrix, add the feature mean to the optimized interaction strategy, and obtain the candidate interaction strategy; Priority weight calculation unit; Used to calculate the priority weights of candidate interaction strategies based on cosine similarity; Final score calculation unit: used to calculate the final scores of candidate interaction strategies and optimized interaction strategies based on cosine similarity and priority weight; Optimal interaction strategy screening unit: used to select the optimized interaction strategy or candidate interaction strategy with the highest final score as the optimal interaction strategy.

4. The multimodal audio-visual media remote interaction system according to claim 3, characterized in that: The user behavior pattern matrix includes facial expression feature vectors, voice intonation feature vectors and body movement feature vectors. Facial expression feature vectors, voice intonation feature vectors and body movement feature vectors form a 3x3 user behavior pattern matrix.

5. The multimodal audio-visual media remote interaction system according to claim 4, characterized in that: Construct a deep learning model and train the deep learning model to obtain the optimal model parameters, including: using the user behavior pattern matrix as the training set of the deep learning model, using the historical interaction effect evaluation value of the user behavior pattern matrix as the prediction label, and the input of the deep learning model is the real-time user behavior pattern matrix; constructing the loss function of the deep learning model based on the absolute error and relative error between the historical interaction effect evaluation value and the predicted interaction effect evaluation value, and using the gradient descent method to minimize the loss function to obtain the optimal model parameters, thereby obtaining a trained deep learning model.

6. The multimodal audio-visual media remote interaction system according to claim 5, characterized in that: The calculation formula for the interaction effect evaluation value is: ； Among them, E represents the interaction effect evaluation value, represents the ith facial expression eigenvalue in the real-time user behavior pattern matrix, represents the i-th voice intonation feature value in the real-time user behavior pattern matrix, represents the eigenvalue of the i-th body movement in the real-time user behavior pattern matrix, a, b, c, and d represent the optimal parameters of the model, and n represents the total number of eigenvalues ​​in the real-time user behavior pattern matrix. represents the logarithmic function, Represents an exponential decay function.

7. The multimodal audio-visual media remote interaction system according to claim 6, characterized in that: The calculation formula for the interaction effect evaluation adjustment amount is: ； in, represents the interaction effect evaluation adjustment amount, represents the adjustment coefficient, represents the difference in interaction effect evaluation, and F represents the sensitivity matrix.

Images