Multimedia interaction data processing method and system

Abstract

The invention provides a multimedia interaction data processing method and system, relates to the technical field of data fusion, and aims to screen out action interaction data in an action mode and voice interaction data in a voice mode from interaction data so as to determine a dynamic time warping distance between action interaction features and voice interaction features, and obtain a dynamic time warping distance between the action interaction features and the voice interaction features. According to the associated interaction information and the dynamic time warping distance, determining a fusion gating coefficient when modal fusion is performed on the action modal and the voice modal, and performing modal fusion on the action interaction feature and the voice interaction feature through the fusion gating coefficient to obtain a data fusion feature of the multimedia; according to the method and the device, information compensation is performed on the data fusion features based on the information loss amount, and the compensated data fusion features are used to perform classified visual display on the interactive data in the multimedia, so that the data fusion of the voice mode and the action mode can be realized, and the discriminability of interactive data processing can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data fusion, and more specifically, to a method and system for processing multimedia interactive data. Background Art

[0002] Multimedia interactive data refers to a collection of data generated by the interaction between users and systems in a multimodal interactive environment through audio, video, text, touch, gesture and other means. Interactive data is widely used in fields such as virtual reality, augmented reality, human-computer interaction and intelligent education, involving multidimensional information such as time series, space and situational perception. Its characteristics include high dimensionality, unstructured, time-varying and strong semantic association, which makes data processing have high computational complexity. With the development of artificial intelligence, 5G communication and edge computing, the research on multimedia interactive data in the fields of intelligent analysis, user behavior modeling and personalized recommendation has been continuously deepened, which has promoted the optimization of intelligent interactive systems.

[0003] Traditional interactive data processing methods mainly rely on a single modality for interactive analysis, and fail to fully utilize the complementary information between multimodal data, resulting in incomplete interactive information and limitations in semantic understanding. In processing methods based only on voice modality, multimedia can only extract the user's voice information, but cannot perceive its accompanying action behavior, resulting in ambiguity in the understanding of interactive intentions. At the same time, in analysis methods that rely only on motion modality, multimedia can only perceive the user's posture, gesture or motion trajectory, but cannot combine voice information to gain a deeper understanding of the interactive context. For example, in a gesture recognition system, the same user action may represent different meanings in different contexts, and the lack of voice auxiliary information may lead to misclassification or misunderstanding. Therefore, how to achieve data fusion of voice modality and motion modality to improve the discriminability of interactive data processing has become a difficult problem faced by the industry. Summary of the invention

[0004] The present application provides a method and system for processing multimedia interactive data, which can realize data fusion of voice modality and action modality, thereby improving the discriminability of interactive data processing.

[0005] In a first aspect, the present application provides a method for processing multimedia interactive data, the data processing method comprising the following steps: Acquire the interaction data to be processed in the multimedia, filter out the action interaction data in the action mode and the voice interaction data in the voice mode from the interaction data, and then determine the associated interaction information between the action interaction data and the voice interaction data; Extracting action interaction features in the action interaction data and voice interaction features in the voice interaction data, thereby determining a dynamic time warping distance between the action interaction features and the voice interaction features, and obtaining an amount of information loss generated during feature extraction; Determine a fusion gating coefficient when performing modal fusion of the action modality and the voice modality according to the associated interaction information and the dynamic time warping distance, and perform modal fusion of the action interaction feature and the voice interaction feature through the fusion gating coefficient to obtain a multimedia data fusion feature; The data fusion feature is compensated for information based on the amount of information loss, and then the compensated data fusion feature is used to classify and visualize the interactive data in the multimedia.

[0006] In this embodiment, filtering out the action interaction data in the action mode and the voice interaction data in the voice mode from the interaction data specifically includes: Classifying the interaction data according to data sources to obtain interaction records of action modalities and interaction records of voice modalities; Determine the action interaction data under the action mode through the interaction records of all action modes; The voice interaction data under the voice modality is determined through the interaction records of all voice modalities.

[0007] In this embodiment, determining the associated interaction information between the action interaction data and the voice interaction data specifically includes: Divide the multimedia interaction process into multiple time periods based on sliding windows; Extracting overlapping features of voice commands and action events in each time series time period from the action interaction data and the voice interaction data; The associated interaction information between the action interaction data and the voice interaction data is determined through all overlapping features.

[0008] In this embodiment, extracting the action interaction features in the action interaction data and the voice interaction features in the voice interaction data specifically includes: Extracting action interaction features from the action interaction data using a convolutional neural network; The speech interaction features in the speech interaction data are extracted using the Mel spectrum.

[0009] In this embodiment, determining the dynamic time warping distance between the action interaction feature and the voice interaction feature specifically includes: Construct the Euclidean distance matrix between the action modality and the speech modality; The dynamic time warping distance between the action interaction feature and the voice interaction feature is calculated using the Euclidean distance matrix.

[0010] In this embodiment, obtaining the amount of information loss generated during feature extraction specifically includes: Determine the feature information entropy of the action interaction features and the voice interaction features; Acquire data information entropy of the action interaction data and the voice interaction data; The amount of information loss generated during the feature extraction process is determined by the feature information entropy and the data information entropy.

[0011] In this embodiment, determining the fusion gating coefficient when performing modal fusion of the action modality and the speech modality according to the associated interaction information and the dynamic time warping distance specifically includes: Extracting action information entropy in action mode and speech information entropy in speech mode; According to the associated interaction information and the dynamic time warping distance, the action information entropy and the voice information entropy are respectively adjusted to obtain the action adjustment entropy and the voice adjustment entropy; The fusion gating coefficient when the action modality and the speech modality are fused is determined by the action adjustment entropy and the speech adjustment entropy.

[0012] In this embodiment, the action interaction feature and the voice interaction feature are modally fused by the fusion gating coefficient to obtain multimedia data fusion features, which specifically include: Extracting fusion coefficients of the action mode and the speech mode respectively from the fusion gating coefficients; The action interaction feature and the voice interaction feature are weightedly fused using various fusion coefficients to obtain multimedia data fusion features.

[0013] In this embodiment, performing information compensation on the data fusion feature based on the information loss amount specifically includes: Initialize a neural network-based information compensation model; Using the information loss amount as a compensation parameter of the information compensation model; The information compensation model is used to compensate for the information loss in the data fusion feature to obtain the compensated data fusion feature.

[0014] In a second aspect, the present application provides a multimedia interactive data processing system for executing a multimedia interactive data processing method, the multimedia interactive data processing system comprising: A data partitioning module is used to obtain the interaction data to be processed in the multimedia, filter out the action interaction data in the action mode and the voice interaction data in the voice mode from the interaction data, and then determine the associated interaction information between the action interaction data and the voice interaction data; A feature extraction module, used to extract the action interaction features in the action interaction data and the voice interaction features in the voice interaction data, thereby determining the dynamic time warping distance between the action interaction features and the voice interaction features, and obtaining the amount of information loss generated during the feature extraction process; A data fusion module, used to determine a fusion gating coefficient when the action mode and the voice mode are modally fused according to the associated interaction information and the dynamic time warping distance, and to perform modal fusion on the action interaction feature and the voice interaction feature through the fusion gating coefficient to obtain a multimedia data fusion feature; The information compensation module is used to compensate the data fusion feature based on the information loss amount, and then use the compensated data fusion feature to classify and visualize the interactive data in the multimedia.

[0015] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects: First, the interactive data to be processed in the multimedia is obtained, and the action interaction data under the action mode and the voice interaction data under the voice mode are screened out from the interactive data, and then the associated interaction information between the action interaction data and the voice interaction data is determined; secondly, the action interaction features in the action interaction data and the voice interaction features in the voice interaction data are extracted, and then the dynamic time warping distance between the action interaction features and the voice interaction features is determined, and the amount of information loss generated in the feature extraction process is obtained; then, according to the associated interaction information and the dynamic time warping distance, the fusion gating coefficient when the action mode and the voice mode are modally fused is determined, and the action interaction features and the voice interaction features are modally fused through the fusion gating coefficient to obtain the data fusion features of the multimedia; finally, the data fusion features are information compensated based on the information loss, and then the compensated data fusion features are used to classify and visualize the interactive data in the multimedia.

[0016] It can be seen that the present application scheme compensates the data fusion features for information based on the amount of information loss, and then uses the compensated data fusion features to classify and visualize the interactive data in multimedia; first, in the process of multimodal data fusion, due to operations such as modal conversion, feature extraction and data alignment, information may be lost or distorted, thereby affecting the integrity and expression ability of the fusion features. By calculating the amount of information loss in the feature extraction process, the effective information lost in the data conversion process can be quantified, thereby providing a basis for subsequent information compensation. The determination of the amount of information loss can also provide a confidence index for data fusion, so that the fusion model can be used in situations with high data integrity. In the case of high information loss, a higher weight is given, and compensation or weighted processing is performed when the information loss is large, thereby improving the stability and accuracy of data fusion; then, the data fusion feature determines the matching degree of different modal data in the joint representation space. Through a reasonable modal fusion strategy, the complementarity of the data can be enhanced, and the problem of a single modality dominating the fusion result can be avoided. At the same time, the determination of the fusion feature can improve the expressiveness of the interactive data, making the data more discriminative in subsequent classification and visualization. Through the feature vector obtained after fusion, the system can more accurately identify the user's intention and improve the accuracy of interactive data processing, thereby enhancing the system's understanding of multimedia interactive data.

[0017] In summary, the technical solution of the present application can realize the data fusion of speech modality and action modality, thereby improving the discriminability of interactive data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 is a flow chart of a method for processing multimedia interactive data provided by the present application; Figure 2 is an exemplary flow chart for determining action interaction features and voice interaction features provided by the present application; Figure 3 is an exemplary flow chart for determining fusion gating coefficients according to the present application; Figure 4 It is a module structure diagram of the multimedia interactive data processing system provided by this application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] The embodiment of the present application provides a method and system for processing multimedia interactive data, the core of which is to obtain interactive data to be processed in multimedia, screen out action interaction data under action mode and voice interaction data under voice mode from the interactive data, and then determine the associated interaction information between the action interaction data and the voice interaction data; extract action interaction features in the action interaction data and voice interaction features in the voice interaction data, and then determine the dynamic time warping distance between the action interaction features and the voice interaction features, and obtain the amount of information loss generated during the feature extraction process; determine the fusion gating coefficient when the action mode and the voice mode are modally fused according to the associated interaction information and the dynamic time warping distance, and modally fuse the action interaction features and the voice interaction features through the fusion gating coefficient to obtain data fusion features of multimedia; perform information compensation on the data fusion features based on the information loss, and then use the compensated data fusion features to classify and visualize the interactive data in the multimedia. In summary, the present application can realize data fusion of voice mode and action mode, thereby improving the discriminability of interactive data processing.

[0022] Embodiment 1: In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 As shown in FIG. 1 , this figure is an exemplary flow chart of a method for processing multimedia interactive data according to this embodiment of the present application. The waveguide detection method includes the following steps: In step S1, the interactive data to be processed in the multimedia is obtained, and the action interaction data in the action mode and the voice interaction data in the voice mode are screened out from the interactive data, and then the associated interaction information between the action interaction data and the voice interaction data is determined.

[0023] It should be noted that in this application, interaction data represents a collection of interaction records in multimedia to be classified and processed; in specific implementation, the interaction records to be processed are obtained from the multimedia database, so that the collection of all interaction records is used as the interaction data to be processed in the multimedia.

[0024] In this embodiment, filtering out the action interaction data in the action mode and the voice interaction data in the voice mode from the interaction data can be achieved by using the following steps: Classifying the interaction data according to data sources to obtain interaction records of action modalities and interaction records of voice modalities; Determine the action interaction data under the action mode through the interaction records of all action modes; The voice interaction data under the voice modality is determined through the interaction records of all voice modalities.

[0025] It should be noted that in the present application, voice interaction data refers to the communication content and control information triggered by voice commands or voice signals during multimedia interaction; action interaction data refers to the control operations and behavioral feedback triggered by body movements, gestures or motion trajectories during multimedia interaction.

[0026] In the specific implementation, first, the interaction records whose data source is the motion sensor are used as the interaction records of the motion modality, and the interaction records whose data source is the audio sensor are used as the interaction records of the voice modality; then, the set of interaction records of all motion modalities can be used as the motion interaction data under the motion modality; finally, the set of interaction records of all voice modalities can be used as the voice interaction data under the voice modality.

[0027] In this embodiment, determining the associated interaction information between the action interaction data and the voice interaction data may be implemented by using the following steps: Divide the multimedia interaction process into multiple time periods based on sliding windows; Extracting overlapping features of voice commands and action events in each time series time period from the action interaction data and the voice interaction data; The associated interaction information between the action interaction data and the voice interaction data is determined through all overlapping features.

[0028] It should be noted that in the present application, the associated interaction information represents the relationship information between the voice modality and the action modality in the process of interaction, which includes mutual influence, complementarity and dependence; the time period represents a time window segment of a fixed length; and the overlapping feature represents the temporal overlap between the action interaction data and the voice interaction data.

[0029] In the specific implementation, first, in the multimedia interactive data processing, the entire interaction process is divided into multiple fixed-length voice time periods (the default is 500ms) based on the sliding window to ensure that the action interaction data and the voice interaction data are in the same time slice, so that each time slice is used as a time series time period, and multiple time series time periods can be obtained; then, for each time series time period, the cross-correlation analysis can be used to calculate the event overlap of the action interaction data and the voice interaction data in the time series time period as the overlapping features of the voice command and the action event in the time series time period, and the overlapping features in each time series time period can be obtained in the above manner; finally, the set of all overlapping features can be used as the associated interaction information between the action interaction data and the voice interaction data.

[0030] In step S2, the action interaction features in the action interaction data and the voice interaction features in the voice interaction data are extracted, and then the dynamic time warping distance between the action interaction features and the voice interaction features is determined, and the amount of information loss generated in the feature extraction process is obtained.

[0031] In this embodiment, reference Figure 2 As shown in FIG. 1 , this figure is an exemplary flow chart of determining action interaction features and voice interaction features in an embodiment of the present application. In this embodiment, extracting action interaction features in the action interaction data and voice interaction features in the voice interaction data can be achieved by using the following steps: In step S21, a convolutional neural network is used to extract action interaction features in the action interaction data; In step S22, the voice interaction features in the voice interaction data are extracted using the Mel spectrum.

[0032] It should be noted that in the present application, the motion interaction feature is the key information feature that reflects human motion, gestures and behaviors; the convolutional neural network is a deep learning model that is widely used to process data with a grid structure. When extracting the motion interaction features in the motion interaction data, the convolutional neural network performs feature learning on the original data through multiple convolutional layers, and can automatically extract complex patterns and features from low to high levels. Specifically, the convolutional neural network performs convolution calculations on the features of the local area with the filter (convolution kernel) through a sliding window operation to extract local features, such as the motion trajectory, speed changes or acceleration information of the action. Through multi-layer convolution, pooling and nonlinear activation functions, CNN can integrate these local features into global features to help the model identify key features in the motion pattern. In the motion interaction data, CNN can effectively capture the temporal changes and spatial features in the motion signal, extract important information reflecting the characteristics of the motion, thereby improving the accuracy and robustness of feature extraction, and supporting subsequent interaction analysis and modal fusion.

[0033] In addition, it should be noted that in the present application, the speech interaction feature is a numerical feature that reflects information such as speech content, emotion and pronunciation pattern. In the extraction process, Mel spectrum is a commonly used speech feature representation method. Mel spectrum simulates the auditory perception of the human ear by converting the speech signal to the Mel frequency scale. The process includes framing and windowing the speech signal, and then obtaining the spectrum of each frame through fast Fourier transform; then, the spectrum is mapped to the Mel frequency axis to generate a Mel spectrum graph; the Mel frequency scale is closer to the human ear's perception of frequency, emphasizing the low-frequency part and compressing the high-frequency part, thereby effectively retaining the perceptual characteristics of speech; in the Mel spectrum graph, each column represents the frequency distribution at a moment, and each row represents a different frequency bandwidth. This representation method can capture key information such as the dynamic changes, intonation, volume and rhythm of speech, and then serve as a feature of speech interaction data, providing a basis for subsequent speech recognition, emotion analysis or modal fusion.

[0034] In this embodiment, determining the dynamic time warping distance between the action interaction feature and the voice interaction feature may be achieved by using the following steps: Construct the Euclidean distance matrix between the action modality and the speech modality; The dynamic time warping distance between the action interaction feature and the voice interaction feature is calculated using the Euclidean distance matrix.

[0035] In the specific implementation, first, the Euclidean distance formula is used to calculate the Euclidean distance between the features under the action mode (i.e., action interaction features) and the features under the voice mode (i.e., voice interaction features) in each time period, and then all Euclidean distances are used to construct the Euclidean distance matrix between the action mode and the voice mode according to a fixed step size; then, the cumulative cost model is used to calculate the length of the best matching path in the Euclidean distance matrix as the dynamic time warping distance between the action interaction features and the voice interaction features.

[0036] It should be noted that in this application, the dynamic time warping distance is a quantitative value that measures the similarity between two time series data; the Euclidean distance matrix is ​​a matrix used to measure the similarity between data points, where each element of the matrix represents the Euclidean distance between two data points. Euclidean distance is the most commonly used measurement method, which is used to calculate the straight-line distance between two points in space. The cumulative cost model is a model used to calculate the optimal matching path in time series data alignment, ensuring that features at different time steps can find the best matching relationship between variable-length sequences. In dynamic time warping (DTW), the cumulative cost model measures the local similarity between action interaction features and voice interaction features by constructing a Euclidean distance matrix, and uses a dynamic programming method to calculate the minimum cumulative matching cost path, where each element of the cumulative cost matrix is ​​recursively calculated by the current Euclidean distance and the minimum cumulative cost of the previous step. The total length of the final path is the cumulative cost value of the optimal matching path. This path ensures nonlinear alignment in the time dimension, overcomes the time offset between voice and action modalities, and improves the matching degree of cross-modal features, thereby obtaining a more accurate dynamic time warping distance and optimizing subsequent modal fusion and information compensation.

[0037] In this embodiment, the amount of information loss generated during feature extraction can be obtained by using the following steps: Determine the feature information entropy of the action interaction features and the voice interaction features; Acquire data information entropy of the action interaction data and the voice interaction data; The amount of information loss generated during the feature extraction process is determined by the feature information entropy and the data information entropy.

[0038] It should be noted that, in this application, the amount of information loss is a quantitative value that measures the difference between the original data and the processed data; the feature information entropy is a parameter used to measure the uncertainty of the feature, and the higher the feature information entropy, the richer the information contained in the feature; the data information entropy is a parameter used to measure the uncertainty of the data, and the data information entropy indicates that the data contains richer information.

[0039] In the specific implementation, first, the probability density estimation method can be used to calculate the probability distribution of the action interaction features and the voice interaction features, so as to respectively perform probability distribution modeling on the action interaction features and the voice interaction features. For discrete features, the occurrence frequency of each feature value can be directly counted, and for continuous features, the Gaussian mixture model can be used to approximate its distribution, so that the distribution entropy of the action interaction features and the voice interaction features can be used as the feature information entropy of the action interaction features and the voice interaction features; then, the histogram statistical method is used to calculate the probability distribution of the action interaction data and the voice interaction data, so as to respectively perform probability distribution modeling on the action interaction data and the voice interaction data, so that the distribution entropy of the action interaction data and the voice interaction data can be calculated as the feature information entropy of the action interaction features and the voice interaction features; finally, the difference between the feature information entropy of the action interaction features and the data information entropy of the action interaction data is used as the action information loss value, and the difference between the feature information entropy of the voice interaction features and the data information entropy of the voice interaction data is used as the audio information loss value, so that the sum of the action information loss value and the audio information loss value can be used as the amount of information loss generated in the feature extraction process.

[0040] In step S3, the fusion gating coefficient for modal fusion of the action modality and the voice modality is determined according to the associated interaction information and the dynamic time warping distance, and the action interaction features and the voice interaction features are modally fused through the fusion gating coefficient to obtain multimedia data fusion features.

[0041] In this embodiment, reference Figure 3 As shown, this figure is an exemplary flow chart for determining the fusion gating coefficient in an embodiment of the present application. In this embodiment, the fusion gating coefficient for modal fusion of the action mode and the voice mode is determined according to the associated interaction information and the dynamic time warping distance, which can be implemented by the following steps: In step S31, extracting action information entropy in action mode and speech information entropy in speech mode; In step S32, the action information entropy and the speech information entropy are adjusted according to the associated interaction information and the dynamic time warping distance to obtain the action adjustment entropy and the speech adjustment entropy; In step S33, the fusion gating coefficient for modal fusion of the action modality and the speech modality is determined by the action adjustment entropy and the speech adjustment entropy.

[0042] In the specific implementation, first, the probability density of the motion sensor data (for example, acceleration, angular velocity, and trajectory point set) can be estimated, and the entropy value of the data distribution can be calculated using the information entropy algorithm as the motion information entropy under the motion mode. The probability distribution of the speech signal can be statistically analyzed based on its characteristics (for example, Mel spectrum), and the entropy value can be calculated using the information entropy algorithm as the speech information entropy under the speech mode. Then, an adjustment model based on the attention mechanism is initialized, and the associated interaction information is used as the attention weight calculation factor in the adjustment model. The dynamic time warping distance is used as the time alignment constraint in the adjustment model. The motion information entropy and the speech information entropy are used as the adjustment objects. The adjustment model is used to perform entropy adjustment on the motion information entropy and the speech information entropy, and the results of the entropy adjustment are used as the motion adjustment entropy and the speech adjustment entropy. Finally, the set of the motion adjustment entropy and the speech adjustment entropy can be used as the value range of the fusion gating coefficient, so as to obtain the fusion gating coefficient when the motion mode and the speech mode are modally fused.

[0043] It should be noted that, in the present application, the fusion gating coefficient represents the parameter for dynamically adjusting the weights of each modal feature in the modal fusion process; the action information entropy represents the uncertainty and complexity of the information in the action interaction data; the speech information entropy represents the distribution uncertainty of the information in the speech interaction data; the action regulation entropy represents the complexity of the action information entropy after adjustment based on the regulation parameters; the speech regulation entropy represents the complexity of the speech information entropy after adjustment based on the regulation parameters; the regulation model is a mathematical model based on the attention mechanism, which dynamically adjusts the information entropy of the action modality and the speech modality by calculating the weights of the associated interaction information, so that they are more coordinated in the fusion process and the associated interaction information is more accurately expressed. The information is used to calculate the attention distribution to enhance the feature expression of highly correlated areas, and the dynamic time warping distance is used as a time alignment constraint to ensure that information of different modalities is adjusted on the same time scale. If the dynamic time warping distance is small, it means that the two modalities are more synchronized, which can reduce the amplitude of information entropy adjustment. If the dynamic time warping distance is large, a larger amplitude of entropy adjustment is required to enhance the adaptability to asynchronous information. The adjustment model adopts the entropy adjustment method to optimize the complementarity of multimodal data by adjusting the distribution of information entropy and improve the effectiveness of information fusion. Finally, the results of entropy adjustment are used for subsequent data fusion to make the fused features more discriminative and stable.

[0044] In this embodiment, the action interaction feature and the voice interaction feature are modally fused by the fusion gating coefficient to obtain multimedia data fusion features, which can be achieved by the following steps: Extracting fusion coefficients of the action mode and the speech mode respectively from the fusion gating coefficients; The action interaction feature and the voice interaction feature are weightedly fused using various fusion coefficients to obtain multimedia data fusion features.

[0045] It should be noted that in the present application, the data fusion feature is a composite feature obtained by integrating action features and speech features; in specific implementation, first, the action regulation entropy corresponding to the fusion gating coefficient is used as the fusion coefficient of the action modality, and the speech regulation entropy corresponding to the fusion gating coefficient is used as the fusion coefficient of the speech modality, and the fusion coefficient is a parameter used to adjust the weights of each modality feature in the modality fusion process; then, a feature weighting algorithm can be used, and each fusion coefficient can be used as the fusion weight in the feature weighting algorithm, and the feature weighting algorithm can be used to fuse the action interaction feature and the speech interaction feature into a data fusion feature of multimedia.

[0046] In step S4, the data fusion feature is compensated for information based on the amount of information loss, and then the compensated data fusion feature is used to classify and visualize the interactive data in the multimedia.

[0047] In this embodiment, information compensation for the data fusion feature based on the information loss amount can be achieved by using the following steps: Initialize a neural network-based information compensation model; Using the information loss amount as a compensation parameter of the information compensation model; The information compensation model is used to compensate for the information loss in the data fusion feature to obtain the compensated data fusion feature.

[0048] It should be noted that in this application, the information compensation model is a mathematical model based on a neural network, which aims to correct the information loss caused by feature extraction, modal conversion and data alignment, and improve the integrity and expression ability of data fusion features. The information compensation model uses the amount of information loss as a compensation parameter, and dynamically adjusts the compensation strategy by learning the mapping relationship between data features and information loss. Its core technologies include autoencoders and residual networks, which are used to reconstruct lost information and use attention mechanisms to enhance the weight of key information. In addition, the information compensation model can combine adversarial learning and variational reasoning to generate compensation features to reduce the impact of data incompleteness on downstream tasks (such as classification, visualization). Finally, the information compensation model generates compensated data fusion features, making multimodal fusion more accurate and stable, and improving the effectiveness of interactive data processing.

[0049] It can be seen that the present application scheme compensates the data fusion features for information based on the amount of information loss, and then uses the compensated data fusion features to classify and visualize the interactive data in multimedia; first, in the process of multimodal data fusion, due to operations such as modal conversion, feature extraction and data alignment, information may be lost or distorted, thereby affecting the integrity and expression ability of the fusion features. By calculating the amount of information loss in the feature extraction process, the effective information lost in the data conversion process can be quantified, thereby providing a basis for subsequent information compensation. The determination of the amount of information loss can also provide a confidence index for data fusion, so that the fusion model can be used in situations with high data integrity. In the case of high information loss, a higher weight is given, and compensation or weighted processing is performed when the information loss is large, thereby improving the stability and accuracy of data fusion; then, the data fusion feature determines the matching degree of different modal data in the joint representation space. Through a reasonable modal fusion strategy, the complementarity of the data can be enhanced, and the problem of a single modality dominating the fusion result can be avoided. At the same time, the determination of the fusion feature can improve the expressiveness of the interactive data, making the data more discriminative in subsequent classification and visualization. Through the feature vector obtained after fusion, the system can more accurately identify the user's intention and improve the accuracy of interactive data processing, thereby enhancing the system's understanding of multimedia interactive data.

[0050] In summary, the technical solution of the present application can realize the data fusion of speech modality and action modality, thereby improving the discriminability of interactive data processing.

[0051] Embodiment 2: This application provides a system for processing multimedia interactive data. Figure 4 As shown, this figure is a schematic diagram of a multimedia interactive data processing system according to this embodiment of the present application, and the multimedia interactive data processing system includes: The data partitioning module 100 is used to obtain the interaction data to be processed in the multimedia, filter out the action interaction data in the action mode and the voice interaction data in the voice mode from the interaction data, and then determine the associated interaction information between the action interaction data and the voice interaction data; A feature extraction module 200, for extracting motion interaction features in the motion interaction data and voice interaction features in the voice interaction data, thereby determining a dynamic time warping distance between the motion interaction features and the voice interaction features, and obtaining an amount of information loss generated during the feature extraction process; A data fusion module 300 is used to determine a fusion gating coefficient when the action mode and the voice mode are modally fused according to the associated interaction information and the dynamic time warping distance, and to perform modal fusion on the action interaction feature and the voice interaction feature through the fusion gating coefficient to obtain a multimedia data fusion feature; The information compensation module 400 is used to perform information compensation on the data fusion feature based on the information loss amount, and then use the compensated data fusion feature to classify and visualize the interactive data in the multimedia.

[0052] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0053] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically-erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0054] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A method for processing multimedia interactive data, characterized in that: The data processing method comprises the following steps: Acquire interactive data to be processed in multimedia, filter out action interaction data in an action mode and voice interaction data in a voice mode from the interactive data, and then determine associated interaction information between the action interaction data and the voice interaction data; Extracting action interaction features in the action interaction data and voice interaction features in the voice interaction data, thereby determining a dynamic time warping distance between the action interaction features and the voice interaction features, and obtaining an amount of information loss generated during feature extraction; Determine a fusion gating coefficient when performing modal fusion of the action modality and the voice modality according to the associated interaction information and the dynamic time warping distance, and perform modal fusion of the action interaction feature and the voice interaction feature through the fusion gating coefficient to obtain a multimedia data fusion feature; The data fusion feature is compensated for information based on the amount of information loss, and then the compensated data fusion feature is used to classify and visualize the interactive data in the multimedia.

2. A method for processing multimedia interactive data according to claim 1, characterized in that: Filtering out the action interaction data in the action mode and the voice interaction data in the voice mode from the interaction data specifically includes: Classifying the interaction data according to data sources to obtain interaction records of action modalities and interaction records of voice modalities; Determine the action interaction data under the action mode through the interaction records of all action modes; The voice interaction data under the voice modality is determined through the interaction records of all voice modalities.

3. A method for processing multimedia interactive data according to claim 1, characterized in that: Determining the associated interaction information between the action interaction data and the voice interaction data specifically includes: Divide the multimedia interaction process into multiple time periods based on sliding windows; Extracting overlapping features of voice commands and action events in each time series time period from the action interaction data and the voice interaction data; The associated interaction information between the action interaction data and the voice interaction data is determined through all overlapping features.

4. A method for processing multimedia interactive data according to claim 1, characterized in that: Extracting the action interaction features in the action interaction data and the voice interaction features in the voice interaction data specifically includes: Extracting action interaction features from the action interaction data using a convolutional neural network; The speech interaction features in the speech interaction data are extracted using the Mel spectrum.

5. A method for processing multimedia interactive data according to claim 1, characterized in that: Determining the dynamic time warping distance between the action interaction feature and the voice interaction feature specifically includes: Construct the Euclidean distance matrix between the action modality and the speech modality; The dynamic time warping distance between the action interaction feature and the voice interaction feature is calculated using the Euclidean distance matrix.

6. A method for processing multimedia interactive data as claimed in claim 1, characterized in that: The amount of information loss generated during feature extraction specifically includes: Determine the feature information entropy of the action interaction features and the voice interaction features; Acquire data information entropy of the action interaction data and the voice interaction data; The amount of information loss generated during the feature extraction process is determined by the feature information entropy and the data information entropy.

7. A method for processing multimedia interactive data as claimed in claim 1, characterized in that: Determining the fusion gating coefficient when performing modal fusion of the action modality and the speech modality according to the associated interaction information and the dynamic time warping distance specifically includes: Extracting action information entropy in action mode and speech information entropy in speech mode; According to the associated interaction information and the dynamic time warping distance, the action information entropy and the voice information entropy are respectively adjusted to obtain the action adjustment entropy and the voice adjustment entropy; The fusion gating coefficient when the action modality and the speech modality are fused is determined by the action adjustment entropy and the speech adjustment entropy.

8. A method for processing multimedia interactive data as claimed in claim 1, characterized in that: The action interaction feature and the voice interaction feature are modally fused by the fusion gating coefficient to obtain multimedia data fusion features, which specifically include: Extracting fusion coefficients of the action mode and the speech mode respectively from the fusion gating coefficients; The action interaction feature and the voice interaction feature are weightedly fused using various fusion coefficients to obtain multimedia data fusion features.

9. A method for processing multimedia interactive data as claimed in claim 1, characterized in that: The information compensation of the data fusion feature based on the information loss amount specifically includes: Initialize a neural network-based information compensation model; Using the information loss amount as a compensation parameter of the information compensation model; The information compensation model is used to compensate for the information loss in the data fusion feature to obtain the compensated data fusion feature.

10. A multimedia interactive data processing system, used to execute a multimedia interactive data processing method according to any one of claims 1 to 9, characterized in that: The multimedia interactive data processing system comprises: A data partitioning module is used to obtain the interaction data to be processed in the multimedia, filter out the action interaction data in the action mode and the voice interaction data in the voice mode from the interaction data, and then determine the associated interaction information between the action interaction data and the voice interaction data; A feature extraction module, used to extract the action interaction features in the action interaction data and the voice interaction features in the voice interaction data, thereby determining the dynamic time warping distance between the action interaction features and the voice interaction features, and obtaining the amount of information loss generated during the feature extraction process; A data fusion module, used to determine a fusion gating coefficient when the action mode and the voice mode are modally fused according to the associated interaction information and the dynamic time warping distance, and to perform modal fusion on the action interaction feature and the voice interaction feature through the fusion gating coefficient to obtain a multimedia data fusion feature; The information compensation module is used to compensate the data fusion feature based on the information loss amount, and then use the compensated data fusion feature to classify and visualize the interactive data in the multimedia.

Images