Method and system for extracting and scoring anchor live broadcast data features based on deep learning

Abstract

The present invention provides a method and system for extracting and scoring anchor live broadcast data features based on deep learning, which relates to the field of deep learning technology, including collecting data from anchor live broadcasts; extracting an initial feature map containing expression, action and scene information through a three-dimensional convolution structure, assigning attention weights to different feature dimensions of the initial feature map according to the recognized scene type, and generating video features; generating audio features based on audio data; calculating user interaction features for user interaction data according to time windows; based on the correlation matrix of video features, audio features and user interaction features in different time windows, performing feature fusion through inter-modal attention weights and inter-window attention weights, and combining time series alignment and window quality evaluation to obtain multimodal features; using a cascade network structure to process the multimodal features, and outputting a comprehensive score for the anchor live broadcast.

Description

Technical Field

[0001] The present invention relates to deep learning technology, and in particular to a method and system for extracting and scoring anchor live broadcast data features based on deep learning. Background Art

[0002] With the rapid development of the live streaming industry and the surge in the number of live streamers, platforms need to effectively evaluate live streamer performance to improve user experience and content quality. In recent years, some machine learning-based evaluation methods have begun to be applied to live streamer performance analysis, but existing technologies have the following shortcomings:

[0003] There are inconsistencies in the timing of different modal data (video, audio, user interaction), and an effective dynamic calibration mechanism for time windows needs to be established. Most evaluation methods fail to consider the dynamic changes of time windows, making it difficult to balance short-term interaction fluctuations and long-term performance trends. This leads to poor stability in the scoring results and an inability to reflect the host's comprehensive capabilities at different time scales.

[0004] Existing technologies mostly use static feature extraction, ignoring the dynamic impact of scene changes on feature importance during live broadcast, and it is difficult to adaptively adjust feature focus for different scene types. Summary of the Invention

[0005] The embodiments of the present invention provide a method and system for extracting and scoring anchor live broadcast data features based on deep learning, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provides a deep learning-based method for extracting and scoring anchor live broadcast data features, including:

[0008] Collect video data, audio data, and user interaction data from the host's live broadcast;

[0009] Identify the scene type of the video data through a scene recognition subnetwork; construct a feature extraction network including a backbone network and an adaptive enhancement module, wherein the backbone network extracts an initial feature map containing expression, action, and scene information through a three-dimensional convolutional structure, and the adaptive enhancement module assigns attention weights to different feature dimensions of the initial feature map according to the scene type to generate video features;

[0010] Generating audio features based on audio data; calculating user interaction features based on the user interaction data according to a time window;

[0011] Construct a multi-time window dynamic calibration network. Based on the correlation matrix of the video features, audio features, and user interaction features in different time windows, perform feature fusion through inter-modal attention weights and inter-window attention weights, and combine temporal alignment and window quality evaluation to obtain multimodal features.

[0012] The multimodal features are processed using a cascade network structure to output a comprehensive score for the anchor's live broadcast, which includes an interactivity score, a professional score, and an appeal score.

[0013] In an optional embodiment,

[0014] Identifying the scene type of the video data by the scene recognition sub-network includes:

[0015] The YOLOv5 target detection network is used to perform target detection and region segmentation on video data frames, extract the anchor area, background area and object area, and perform size normalization on the regional image blocks; feature maps of the regional image blocks are extracted respectively, and multi-scale features are extracted through spatial pyramid pooling. The multi-scale features are reduced in dimensionality and then spliced ​​to form a spliced ​​scene feature vector; the scene feature vector is input into a multi-layer fully connected network for feature transformation, and the scene category probability distribution is calculated through the softmax activation function to determine the scene type.

[0016] In an optional embodiment,

[0017] Construct a feature extraction network including a backbone network and an adaptive enhancement module. The backbone network extracts an initial feature map containing expression, action, and scene information through a three-dimensional convolutional structure. The adaptive enhancement module assigns attention weights to different feature dimensions of the initial feature map according to the scene type. Generating video features includes:

[0018] Constructing a backbone network and adaptive enhancement module. The backbone network includes three branches: micro-expressions, body movements, and scene context. Three-dimensional convolution kernels of different sizes are used to extract and splice the initial feature map.

[0019] The adaptive enhancement module includes: obtaining a main category probability distribution through a main category classification head, selecting a subdivision category classification head based on the main category probability distribution to obtain a conditional probability distribution, applying the main category probability distribution and the conditional probability distribution to an embedding matrix to obtain a main category embedding vector and a subdivision category embedding vector, and performing bilinear fusion to obtain a hierarchical scene embedding; hierarchically dynamically modulating the initial feature map based on the hierarchical scene embedding, using the main category embedding vector in a shallow layer, using the main category and subdivision category embedding vectors in a middle layer, and using the hierarchical scene embedding in a deep layer;

[0020] Features modulated at different levels are constructed into a lateral information transfer network through conditional gating units, and multi-scale modulation features are obtained by integration through a feature pyramid network. Corresponding channel attention weights and spatial attention weights are generated for the multi-scale modulation features through hierarchical scene embedding and applied, and a normalized feature map is obtained by combining residual connection and layer normalization. Linear transformation parameters are generated for each scene type based on the hierarchical scene embedding, and the transformation results are weighted and fused according to the scene type probability distribution of the normalized feature map to obtain video features.

[0021] In an optional embodiment,

[0022] The backbone network also includes an expression-action collaborative modeling module, which includes:

[0023] The features of the micro-expression branch and the body movement branch are mapped into a unified space. The inter-modal similarity is calculated based on scaled dot-product attention. The inter-modal similarity is adaptively fused with the relative position encoding of the feature points to generate dynamic association weights. Multi-scale collaborative features are extracted through a graph convolutional network.

[0024] Based on the multi-scale collaborative features, local window units and global window units are constructed, wherein the local window unit uses a sliding window convolution with a learnable step size to extract short-term emotional response features, and the learnable step size is dynamically adjusted based on the hidden state at the current moment; the global window unit combines a bidirectional recurrent neural network and an adaptive temporal attention mechanism to extract long-term performance style features, wherein the temporal attention weight is calculated by a nonlinear transformation; after feature alignment of the short-term emotional response features and the long-term performance style features, the emotion-behavior collaborative features are obtained by fusing them through an adaptive weighting method based on a gating mechanism, and the emotion-behavior collaborative features are used to generate the initial feature map.

[0025] In an optional embodiment,

[0026] A multi-time window dynamic calibration network is constructed. Based on the correlation matrix of the video features, audio features, and user interaction features in different time windows, feature fusion is performed through inter-modal attention weights and inter-window attention weights. Combined with temporal alignment and window quality evaluation, multimodal features are obtained, including:

[0027] Build contextual states based on video, audio, and user interaction features. Perform multi-feature change point detection and multi-scale verification on these features to determine the initial time window sequence. Optimize the window structure through threshold control, and dynamically adjust window parameters through autocorrelation analysis, periodogram analysis, and contextual state mapping.

[0028] Calculate the mean, standard deviation, maximum, and temporal trend characteristics of video features, audio features, and user interaction features in each time window, and calculate the correlation matrix between modalities;

[0029] Based on the correlation matrix, inter-modality attention weights and inter-window attention weights are generated, wherein the inter-modality attention weights are calculated based on the correlation matrix, and the inter-window attention weights are obtained by mapping the correlation matrix through a nonlinear transformation function, and a multi-level attention fusion network is constructed to perform feature fusion; the inter-modality time offset is determined through cross-correlation analysis and time series alignment is performed;

[0030] The Frobenius norm of the correlation matrix in each time window is calculated to obtain the window quality score, which is converted into a scale balance factor through the softmax function. The multi-scale feature contribution of the time-aligned features is balanced according to the scale balance factor to obtain multimodal features.

[0031] In an optional embodiment,

[0032] Contextual states are constructed based on video features, audio features, and user interaction features. Multi-feature change point detection and multi-scale verification are performed on these features to determine the initial time window sequence. The window structure is optimized through threshold control. Window parameters are dynamically adjusted through autocorrelation analysis, periodogram analysis, and contextual state mapping. These include:

[0033] Calculate the temporal changes of video features, audio features, and user interaction features to obtain interaction density indicators, content rhythm indicators, and modal collaboration indicators. Construct a multi-dimensional context feature vector and perform dynamic clustering to obtain the context state.

[0034] Perform multi-feature change point detection and multi-scale verification on video features, audio features and user interaction features, and recursively segment based on the segmentation score function of the change point detection value and the sub-window balance to obtain an initial time window sequence; calculate the interaction density score of each window in the initial time window sequence, and subdivide the windows exceeding a first preset threshold and merge the adjacent windows below a second preset threshold according to the interaction density score, and obtain an optimized time window sequence based on the window subdivision and window merging; perform autocorrelation analysis and periodogram analysis on the optimized time window sequence, wherein the autocorrelation analysis obtains periodic features by calculating the autocorrelation coefficient under time delay, and the periodogram analysis obtains spectral features by calculating the power spectral density; fuse the periodic features and spectral features at different time scales to obtain multi-scale periodic features, construct a periodic synergy matrix based on the multi-scale periodic features, and obtain a cross-modal synergy periodic group through spectral clustering;

[0035] A probability mapping relationship between the situation state and the window parameters is established, the window parameters at each time scale are dynamically adjusted in combination with the cross-modal collaborative period group, and an optimal window parameter set is generated based on the current situation state.

[0036] In an optional embodiment,

[0037] The multimodal features are processed using a cascade network structure to output a comprehensive score for the anchor's live broadcast. The comprehensive score includes an interactivity score, a professional score, and an appeal score.

[0038] The multimodal features are decomposed into temporal features to extract the temporal patterns of expressions, actions, voices, and interactions, and a dynamic attention matrix is ​​constructed based on the scene type for feature adaptive enhancement. A cascade structure is constructed, comprising a bottom-level technical scoring subnetwork, a middle-level perception scoring subnetwork, and a top-level comprehensive scoring subnetwork. The bottom-level technical scoring subnetwork is modeled based on the enhanced temporal features and obtains technical indicator scores through feature fusion. The middle-level perception scoring subnetwork dynamically adjusts the weights of technical indicators based on the scene-aware attention unit to construct a correlation matrix between scoring dimensions and generate interactivity scores, professionalism scores, and appeal scores through a graph attention network. The top-level comprehensive scoring subnetwork uses a skip connection mechanism to fuse multiple layers of features.

[0039] An inter-dimensional learning mechanism is established between each scoring sub-network, and a multi-objective loss function including scoring dimension consistency loss, scoring standard deviation loss and knowledge distillation loss is constructed for optimization. The scoring dimension consistency loss is used to constrain the relative relationship between different scoring dimensions, the scoring standard deviation loss adaptively adjusts the scoring scale based on the distribution of historical scoring data, and the knowledge distillation loss guides the information interaction of the features of each scoring dimension. A phased training strategy is adopted to optimize the multi-objective loss function through the process of pre-training-distillation-fine-tuning to obtain a comprehensive score.

[0040] According to a second aspect of the embodiments of the present invention,

[0041] Provides a deep learning-based live broadcast data feature extraction and scoring system, including:

[0042] The first unit is used to collect video data, audio data and user interaction data during the anchor's live broadcast;

[0043] The second unit is configured to identify the scene type of the video data through a scene recognition subnetwork; construct a feature extraction network including a backbone network and an adaptive enhancement module, wherein the backbone network extracts an initial feature map containing expression, action, and scene information through a three-dimensional convolutional structure; and the adaptive enhancement module assigns attention weights to different feature dimensions of the initial feature map according to the scene type to generate video features; generate audio features based on the audio data; and calculate user interaction features based on the user interaction data according to a time window.

[0044] The third unit is used to build a multi-time window dynamic calibration network. Based on the correlation matrix of the video features, audio features and user interaction features in different time windows, it performs feature fusion through inter-modal attention weights and inter-window attention weights, and combines temporal alignment and window quality evaluation to obtain multimodal features;

[0045] The fourth unit is used to process the multimodal features using a cascade network structure and output a comprehensive score of the anchor's live broadcast, where the comprehensive score includes an interactivity score, a professionalism score, and an appeal score.

[0046] According to a third aspect of the embodiments of the present invention,

[0047] An electronic device is provided, comprising:

[0048] processor;

[0049] a memory for storing processor-executable instructions;

[0050] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0051] According to a fourth aspect of the embodiments of the present invention,

[0052] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0053] The present invention uses deep learning technology to extract multiple data features from the anchor's live broadcast, which can comprehensively analyze the anchor's performance and improve the quality of the live broadcast content and the audience experience.

[0054] The present invention adopts a dynamic calibration network and a multimodal feature fusion method to solve the problem of temporal inconsistency among video, audio, and user interaction data, enabling the scoring system to more accurately reflect the host's performance in different scenarios and enhance the reliability and effectiveness of the scoring.

[0055] The present invention constructs a complete scoring system from bottom-level technical indicators to mid-level perception indicators to top-level comprehensive scores. It not only provides an overall score, but also gives detailed scores for dimensions such as interactivity, professionalism, and appeal, making the scoring results more comprehensive and explainable. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Schematic diagram of a method for extracting and scoring anchor live broadcast data features based on deep learning according to an embodiment of the present invention;

[0057] Figure 2 Comparison chart of feature extraction accuracy under different scene types;

[0058] Figure 3 This is a comparison chart of multimodal feature fusion effects;

[0059] Figure 4 This is an analysis diagram of the multimodal feature fusion effect under different situational states. DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0061] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0062] Figure 1 Schematic diagram of the method for extracting and scoring anchor live broadcast data features based on deep learning according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0063] Collect video data, audio data, and user interaction data from the host's live broadcast;

[0064] Identify the scene type of the video data through a scene recognition subnetwork; construct a feature extraction network including a backbone network and an adaptive enhancement module, wherein the backbone network extracts an initial feature map containing expression, action, and scene information through a three-dimensional convolutional structure, and the adaptive enhancement module assigns attention weights to different feature dimensions of the initial feature map according to the scene type to generate video features; generate audio features based on the audio data; and calculate user interaction features based on the user interaction data according to a time window;

[0065] Construct a multi-time window dynamic calibration network. Based on the correlation matrix of the video features, audio features, and user interaction features in different time windows, perform feature fusion through inter-modal attention weights and inter-window attention weights, and combine temporal alignment and window quality evaluation to obtain multimodal features.

[0066] The multimodal features are processed using a cascade network structure to output a comprehensive score for the anchor's live broadcast, which includes an interactivity score, a professionalism score, and an appeal score.

[0067] In an optional implementation, identifying the scene type of the video data by the scene recognition sub-network includes:

[0068] The YOLOv5 target detection network is used to perform target detection and region segmentation on video data frames, extract the anchor area, background area and object area, and perform size normalization on the regional image blocks; feature maps of the regional image blocks are extracted respectively, and multi-scale features are extracted through spatial pyramid pooling. The multi-scale features are reduced in dimensionality and then spliced ​​to form a spliced ​​scene feature vector; the scene feature vector is input into a multi-layer fully connected network for feature transformation, and the scene category probability distribution is calculated through the softmax activation function to determine the scene type.

[0069] For example, a YOLOv5 network performs object detection on a video frame, outputting multiple bounding boxes and their class labels. The detection results are used to divide the video frame into different regions: the person detection box corresponds to the anchor region, the object detection box corresponds to the object region, and the remaining region is the background region. For example, the anchor (coordinates [120, 50, 320, 450]) and the product (coordinates [400, 200, 550, 380]) are detected. The image blocks in each region are cropped and resized to a uniform size of 224×224, and pixel values ​​are normalized. A ResNet50 network, with the convolutional layers after the fully connected layers removed, is used as the feature extraction network, generating a 7×7×2048-dimensional feature map. A spatial pyramid pooling module is used for multi-scale feature extraction, with four pooling scales: 1×1, 2×2, 3×3, and 6×6. Dimensionality reduction is performed on the pooled features, reducing the feature dimension of each branch from 2048 to 256. The feature vectors generated from the above processing for the anchor area, background area, and object area are concatenated sequentially to form the final scene feature vector. This scene feature vector is then fed into a three-layer fully connected network, with each layer outputting dimensions of 1024, 256, and the number of scene categories, N. The softmax function is used to calculate the probability of each scene category, and the category with the highest probability is selected as the scene type.

[0070] Furthermore, the YOLOv5 object detection network and scene recognition network can be fine-tuned to adapt to changes in video content in different scenarios. For example, for e-commerce live streaming, the detection capabilities of elements such as products and price tags can be enhanced; for game live streaming, the recognition capabilities of game interface elements can be strengthened. Through fine-tuning training in specific scenarios, the system's recognition accuracy can be further improved.

[0071] The present invention uses the YOLOv5 network to achieve precise segmentation of the host, objects, and background areas. It combines the deep features extracted by ResNet50 and the multi-scale features captured by spatial pyramid pooling to improve the expressiveness of scene features. It adopts a multi-layer fully connected network for scene classification and supports scene-specific model fine-tuning, which improves the accuracy and adaptability of scene recognition and provides reliable scene context information for subsequent live broadcast quality evaluation.

[0072] In an optional embodiment, a feature extraction network including a backbone network and an adaptive enhancement module is constructed. The backbone network extracts an initial feature map containing expression, action, and scene information through a three-dimensional convolutional structure. The adaptive enhancement module assigns attention weights to different feature dimensions of the initial feature map according to the scene type. Generating video features includes:

[0073] Constructing a backbone network and adaptive enhancement module. The backbone network includes three branches: micro-expressions, body movements, and scene context. Three-dimensional convolution kernels of different sizes are used to extract and splice the initial feature map.

[0074] The adaptive enhancement module includes: obtaining a main category probability distribution through a main category classification head, selecting a subdivision category classification head based on the main category probability distribution to obtain a conditional probability distribution, applying the main category probability distribution and the conditional probability distribution to an embedding matrix to obtain a main category embedding vector and a subdivision category embedding vector, and performing bilinear fusion to obtain a hierarchical scene embedding; hierarchically dynamically modulating the initial feature map based on the hierarchical scene embedding, using the main category embedding vector in a shallow layer, using the main category and subdivision category embedding vectors in a middle layer, and using the hierarchical scene embedding in a deep layer;

[0075] Features modulated at different levels are constructed into a lateral information transfer network through conditional gating units, and multi-scale modulation features are obtained by integration through a feature pyramid network. Corresponding channel attention weights and spatial attention weights are generated for the multi-scale modulation features through hierarchical scene embedding and applied, and a normalized feature map is obtained by combining residual connection and layer normalization. Linear transformation parameters are generated for each scene type based on the hierarchical scene embedding, and the transformation results are weighted and fused according to the scene type probability distribution of the normalized feature map to obtain video features.

[0076] For example, the backbone network consists of three parallel branches: a micro-expression branch, a body movement branch, and a scene context branch. The micro-expression branch uses small-sized 3D convolutional kernels (3×3×3) to construct a four-layer convolutional network. The convolutional layer channels are 64, 128, 192, and 256, respectively, with a stride of 1 and padding of 1. Each convolutional layer is followed by a batch normalization layer and a Reluctant Unit (ReLU) activation function. This branch focuses on capturing subtle facial changes in the host, such as raised eyebrows and pursed lips. The body movement branch uses medium-sized 3D convolutional kernels (5×5×5) to construct a four-layer convolutional network. The channels are 48, 96, 192, and 384, respectively, with a stride of 1 and padding of 2. This branch primarily extracts large-scale movement features such as gestures and posture. The scene context branch constructs a three-layer convolutional network using large-scale 3D convolution kernels (7×7×7), with 32, 128, and 256 channels, respectively, a stride of 1, and padding of 3. This branch captures environmental information such as background and props. The output feature maps of the three branches are resized to 32×32 through upsampling or downsampling, with 256, 384, and 256 channels, respectively. These are then concatenated along the channel dimension to create an initial feature map of 32×32×896.

[0077] The adaptive enhancement module first constructs a main category classification head, consisting of a global average pooling layer and two fully connected layers. The first fully connected layer has an output dimension of 512 and uses the ReLU activation function. The second layer has an output dimension of 6, corresponding to six main scene types: game live streaming, talent show, knowledge sharing, outdoor activities, e-commerce promotion, and casual chat interaction. A softmax function is applied to obtain the main category probability distribution. For example, for a live broadcast clip, the main category probability distribution is [0.08, 0.72, 0.05, 0.04, 0.06, 0.05], indicating that the clip primarily belongs to the talent show category.

[0078] A subcategory classification head is then selected based on the main category probability distribution. Specifically, a dedicated subcategory classification head is designed for each main category. The structure also includes global average pooling and two fully connected layers, but the output dimension of the last layer is equal to the number of subcategories within that main category. For example, the talent show category has four subcategories: singing, dancing, instrument playing, and painting. Based on the main category with the highest probability, the corresponding subcategory classification head is selected to obtain the conditional probability distribution of the subcategory. The conditional probability distribution obtained for the talent show example above is [0.15, 0.75, 0.05, 0.05], indicating that the live broadcast primarily features dance content.

[0079] The main category probability distribution and conditional probability distribution are converted into embedding vectors using an embedding matrix. The main category embedding matrix has a dimension of 6×256, mapping the main category probability distribution to a 256-dimensional main category embedding vector. The embedding matrices for each sub-category have a dimension of the corresponding number of sub-categories × 128, mapping the conditional probability distribution to a 128-dimensional sub-category embedding vector. These two embedding vectors are combined using a bilinear fusion operation to generate a 384-dimensional hierarchical scene embedding vector. Bilinear fusion is implemented by multiplying the main category embedding vector with a learnable transformation matrix (dimensions of 256×128×384) to obtain an intermediate result, which is then dot-producted with the sub-category embedding vector to generate the fused embedding vector.

[0080] The initial feature map is dynamically modulated hierarchically based on the generated embedding vectors. This hierarchical modulation adopts a coarse-to-fine strategy: the shallow network (the 1st-2nd convolutional layers) uses only the main category embedding vector for feature enhancement. This is achieved through a channel attention mechanism, which converts the main category embedding vector into a weight vector with the same dimension as the number of feature channels through a fully connected layer, thus weighting the feature channels. The mid-layer network (the 3rd-4th convolutional layers) combines the main category and sub-category embedding vectors for modulation, using channel recalibration techniques to generate scaling factors and offset parameters to adjust the features. The deep network (the 5th and subsequent convolutional layers) uses the full hierarchical scene embedding for feature modulation. A feature rescaling technique is used to generate a feature transformation matrix through a small MLP network, which performs nonlinear transformations on the deep features.

[0081] The modulated features at different levels are passed through conditional gating units to construct a lateral information transfer network. The conditional gating unit is designed with three gating mechanisms: input gate, forget gate, and output gate. Each gate consists of two fully connected layers. Its input is the concatenation of the current layer's features and the hierarchical scene embedding, and its output is a control value between 0 and 1. The input gate controls the degree of new information introduced, the forget gate controls the degree of retention of historical information, and the output gate controls the amount of information transferred. For example, in the talent show scenario, the average activation value of the input gate was 0.72, the forget gate was 0.35, and the output gate was 0.81, indicating that the system prioritizes capturing real-time expressions and movements. Through this lateral connection, combining top-down and bottom-up pathways, a feature pyramid network is used to integrate multi-level features, generating modulated features at three scales: P3 (32×32×256), P4 (16×16×256), and P5 (8×8×256).

[0082] Attention weights are applied to multi-scale modulated features using a hierarchical scene embedding. The channel-wise attention mechanism converts the hierarchical scene embeddings into weight vectors with the same dimensions as the number of feature channels through a two-layer MLP network (with dimensions of 384, 128, and 256, respectively), selectively enhancing feature channels. The spatial attention mechanism reshapes the hierarchical scene embeddings and generates a two-dimensional spatial weight map (with the same dimensions as the feature map) through a 1×1 convolution, emphasizing important regions in the feature map. After applying these attention weights, the original features are summed with the attention-enhanced features through a residual connection, followed by layer-wise normalization to produce a normalized feature map.

[0083] Based on hierarchical scene embedding, linear transformation parameters are generated for each scene type. Specifically, a learnable scene adaptation layer is designed to generate a set of 384×384 transformation matrices for each of the six main categories. The normalized feature map is compressed into a 384-dimensional feature vector using global average pooling. Six scene-specific feature representations are then generated by applying these representations to a weighted fusion based on the scene type probability distribution to obtain the final 384-dimensional video feature.

[0084] Method for Generating Audio and User Interaction Features: When generating audio features based on audio data, the original audio signal is first preprocessed, including resampling to 16kHz and pre-emphasis. A short-time Fourier transform (SFT) is then applied to extract the spectrogram, using a 25ms window size and a 10ms step size. 13-dimensional MFCC features (Mel-Frequency Cepstral Coefficients) and first- and second-order difference coefficients are extracted from the spectrogram to form a 39-dimensional basic feature vector. Supplementary features such as audio energy, zero-crossing rate, spectral entropy, and fundamental frequency are further calculated. Finally, these features are concatenated to form a 256-dimensional audio feature vector. When calculating user interaction features based on a time window, a 15-second sliding window with a 5-second step size is used. Interaction density features within the window are calculated, including the number of comments, likes, and gifts, as well as their distribution. Interaction sentiment features are extracted, analyzing the sentiment of comments using word frequency statistics and simple rules. Interaction concentration features are calculated to reflect the distribution of user attention. Finally, a 128-dimensional user interaction feature vector is generated, capturing the temporal patterns and sentiment of user feedback.

[0085] Figure 2This figure compares the accuracy of feature extraction in different scene types. It shows the accuracy of the feature extraction network proposed in this paper in six different live streaming scenarios, and compares it with two existing mainstream methods: a three-dimensional residual network (3D-ResNet) and an inflated 3D convolutional network (I3D). The figure clearly shows that the feature extraction method proposed in this paper, based on a backbone network and an adaptive enhancement module, demonstrates significant advantages across all scene types. Specifically, it achieves 92.7% accuracy in game live streaming, 89.3% in talent shows, and the highest accuracy of 94.5% in knowledge sharing. It also achieves 88.6% in outdoor activities, 91.4% in e-commerce live streaming, and 93.8% in casual chat interactions. The proposed method demonstrates significant advantages in complex and varied scenarios, such as outdoor activities and talent shows, achieving an average accuracy improvement of 12.7% over existing methods. This is mainly due to the hierarchical scene embedding and adaptive enhancement module adopted in the present invention, which can dynamically adjust the feature extraction process according to different scene types and more accurately capture the host's expressions, movements and scene information.

[0086] Figure 3 This figure compares the performance of our proposed hierarchical scene embedding method with three existing multimodal feature fusion methods (CrossModal Transformer, multi-head attention mechanism, and simple feature concatenation) across five key evaluation metrics. The data in the figure shows that our proposed method significantly outperforms the competing methods in expression recognition accuracy, action recognition accuracy, scene understanding accuracy, and inter-modal consistency, achieving 92.7%, 90.5%, 94.3%, and 89.6%, respectively. While the computational latency (43.5ms) is not the lowest, this additional overhead is entirely acceptable given the magnitude of the performance improvement. Our proposed method effectively integrates information from the main and sub-categories through bilinear fusion, achieving more precise scene adaptability. Furthermore, the horizontal information transfer network constructed through conditional gating units enhances the interaction between features at different levels. This design enables the system to effectively process multimodal information in live broadcasts while maintaining high accuracy, providing a solid foundation for comprehensive evaluation of live broadcast quality.

[0087] The feature extraction network of the present invention captures micro-expressions, body movements and scene information through three dedicated branches, and combines hierarchical scene embedding to achieve accurate scene-adaptive feature modulation; adopts layered dynamic modulation and lateral information transmission to greatly improve the feature representation capability; through the attention mechanism of the adaptive enhancement module, accurate positioning of key information in different scenarios is achieved.

[0088] In an optional embodiment, the backbone network further includes an expression-action collaborative modeling module, and the expression-action collaborative modeling module includes:

[0089] The features of the micro-expression branch and the body movement branch are mapped into a unified space. The inter-modal similarity is calculated based on scaled dot-product attention. The inter-modal similarity is adaptively fused with the relative position encoding of the feature points to generate dynamic association weights. Multi-scale collaborative features are extracted through a graph convolutional network.

[0090] Based on the multi-scale collaborative features, local window units and global window units are constructed, wherein the local window unit uses a sliding window convolution with a learnable step size to extract short-term emotional response features, and the learnable step size is dynamically adjusted based on the hidden state at the current moment; the global window unit combines a bidirectional recurrent neural network and an adaptive temporal attention mechanism to extract long-term performance style features, wherein the temporal attention weight is calculated by a nonlinear transformation; after feature alignment of the short-term emotional response features and the long-term performance style features, the emotion-behavior collaborative features are obtained by fusing them through an adaptive weighting method based on a gating mechanism, and the emotion-behavior collaborative features are used to generate the initial feature map.

[0091] For example, micro-expression features and body movement features are mapped to a feature space of the same dimensionality through a linear projection layer. For example, 256-dimensional micro-expression features and 384-dimensional movement features are each mapped to a 320-dimensional feature vector through a linear transformation with learnable parameters. This allows the features of the two modalities to be compared and fused in the same feature space.

[0092] When calculating inter-modal similarity, the mapped features are normalized by performing a dot product operation and dividing by the square root of the feature dimension. In this implementation, when the feature dimension is 320, the normalization factor is set to 18. This scaling operation avoids the vanishing gradient problem caused by excessively large values. Each element in the calculated similarity matrix represents the correlation between the features at the corresponding position of the two modalities.

[0093] When fusing inter-modal similarity with the relative position encoding of feature points, the relative positions of the feature points are first calculated. For each point on the micro-expression feature map, its relative offset from each point on the action feature map is calculated. These offsets are converted into position embedding vectors using a learnable position encoding layer. The position encoding layer consists of two fully connected layers, taking relative coordinates as input and outputting a 32-dimensional position encoding. The position encoding and similarity are weighted and summed to obtain the final dynamic association weight. The weight reflects the strength of the spatial-semantic association between feature points.

[0094] When using a graph convolutional network to extract multi-scale collaborative features, feature points are used as graph nodes, and dynamic association weights are used as edge weights to construct the graph structure. A three-layer graph convolutional network is used, with 128, 192, and 256 output channels per layer, respectively. Each layer's graph convolution operation aggregates information from neighboring nodes, with neighborhood sizes of 3, 5, and 7, respectively, to achieve multi-scale feature extraction. Residual connections are used to preserve feature information across each layer.

[0095] When constructing local window units, a sliding window convolution with a learnable step size is used. The window size is set to 8 frames, with an initial step size of 4 frames. The step size is adjustable within a range of 2-6 frames and is predicted by the fully connected layer using the current hidden state. This dynamic step size allows the temporal granularity of feature extraction to be adaptively adjusted based on the speed of video content changes.

[0096] The global window unit uses a bidirectional long short-term memory network to extract long-term dependencies. The network consists of two bidirectional LSTM layers with a hidden dimension of 512. Temporal attention weights are calculated by passing the hidden state through two fully connected layers with a ReLU activation function in between. This structure can capture long-term performance style characteristics in videos.

[0097] When processing short-term emotional response features (e.g., facial expression changes within 2-3 seconds or combinations of simple nodding and shaking movements) and long-term performance style features (e.g., statistically analyzing behavioral patterns over 30 seconds, such as speaking speed, frequency of expression switching, and frequency of gesture use), both are first projected into a 384-dimensional feature space via linear projection. L2 normalization is used to align the features and ensure consistent feature distribution. A gating mechanism is used to calculate the fusion weight: two features are input into a fully connected layer to generate a vector, the cross-correlation coefficient is calculated, and a sigmoid function is applied to the vector to obtain a fusion weight between 0 and 1. This weight is used to dynamically adjust the fusion ratio of the two features to generate an emotional-behavioral synergistic feature. Finally, this synergistic feature is resized via a 1×1 convolution and then element-wise added to the original initial feature map to output the enhanced initial feature map.

[0098] The present invention realizes the intelligent fusion of micro-expression and body movement features through the expression-action collaborative modeling module, and adopts dynamic step size and adaptive weight mechanism to process features of different time scales. While improving the accuracy of emotion recognition and behavior recognition, it maintains low computational overhead and significantly improves the effect and efficiency of feature extraction.

[0099] In an optional embodiment, a multi-time window dynamic calibration network is constructed. Based on the correlation matrix of the video features, audio features, and user interaction features in different time windows, feature fusion is performed through inter-modal attention weights and inter-window attention weights. The multimodal features obtained by combining temporal alignment and window quality assessment include:

[0100] Build contextual states based on video, audio, and user interaction features. Perform multi-feature change point detection and multi-scale verification on these features to determine the initial time window sequence. Optimize the window structure through threshold control, and dynamically adjust window parameters through autocorrelation analysis, periodogram analysis, and contextual state mapping.

[0101] Calculate the mean, standard deviation, maximum, and temporal trend characteristics of video features, audio features, and user interaction features in each time window, and calculate the correlation matrix between modalities;

[0102] Based on the correlation matrix, inter-modality attention weights and inter-window attention weights are generated, wherein the inter-modality attention weights are calculated based on the correlation matrix, and the inter-window attention weights are obtained by mapping the correlation matrix through a nonlinear transformation function, and a multi-level attention fusion network is constructed to perform feature fusion; the inter-modality time offset is determined through cross-correlation analysis and time series alignment is performed;

[0103] The Frobenius norm of the correlation matrix in each time window is calculated to obtain the window quality score, which is converted into a scale balance factor through the softmax function. The multi-scale feature contribution of the time-aligned features is balanced according to the scale balance factor to obtain multimodal features.

[0104] For example, the temporal changes of video, audio, and interactive features are calculated to obtain interaction density, content rhythm, and modal coordination indicators, which form a contextual feature vector. The main contextual state is obtained through dynamic clustering. A change point detection algorithm is applied to the three features, and the distance between the mean features of the front and back sub-windows is calculated. The effective change points are used to divide the initial window sequence. Windows with high interaction density are subdivided, and adjacent windows with low interaction density are merged. The autocorrelation coefficient and power spectral density are calculated, and the periodic features of the three time scales of short-term, medium-term, and long-term are extracted to form a multi-scale periodic feature vector. Dynamic adjustment is performed based on the mapping relationship between contextual state and window parameters to generate adaptive window parameters for different contextual states.

[0105] Calculate the statistical features of multimodal features within each time window. For each modal feature, calculate the mean, standard deviation, maximum, and temporal trend within each time window. The mean is obtained by averaging the feature vectors within the window, the standard deviation reflects the degree of feature fluctuation, the maximum captures the peak performance within the window, and the temporal trend is estimated by linear regression slope estimation. These statistical features are concatenated to form a window-level feature representation. Calculate the inter-modal correlation matrix, where the matrix elements are the Pearson correlation coefficients between the features of different modalities.

[0106] Attention weights are generated based on the correlation matrix. Inter-modal attention weights are calculated by applying the Softmax function to the correlation matrix. For example, the correlation coefficient between video and audio features is 0.78, the correlation coefficient between video and user interaction features is 0.65, and the correlation coefficient between audio and user interaction features is 0.52. The normalized inter-modal attention weights are 0.42, 0.35, and 0.23, respectively. Inter-window attention weights are obtained by applying a nonlinear transformation to the correlation matrix using the ReLU activation function, highlighting windows with high correlation. A network with a three-layer attention fusion mechanism is constructed for feature fusion: the first layer focuses on intra-modal feature fusion, the second layer processes inter-modal feature interactions, and the third layer integrates multi-window information. The temporal offset between modalities is determined by calculating the cross-correlation function. For example, the average delay between audio features and video features is 0.3 seconds, and the average delay between user interaction features and video features is 1.2 seconds. Temporal alignment is performed based on these offsets.

[0107] Window quality assessment and feature contribution balance are performed. The Frobenius norm of the correlation matrix within each time window is calculated. Windows with larger norm values ​​indicate stronger correlation between modalities and higher quality. The quality scores of all windows are converted into scale balancing factors using the Softmax function. For example, if the quality scores for five time windows are [0.85, 0.65, 0.92, 0.78, 0.56], the converted scale balancing factors are [0.25, 0.16, 0.28, 0.22, 0.09]. These balancing factors are used to balance the multi-scale feature contributions of the time-aligned features to obtain the final multimodal feature representation.

[0108] Live streaming scenarios are complex and ever-changing, with uneven content density and large fluctuations in interaction frequency. Fixed windows cannot adapt to this dynamic nature. This paper introduces contextual state to adaptively adjust the time window size and step size to prevent important information from being fragmented by window boundaries. It also designs multi-feature change point detection to automatically determine the optimal segmentation point, ensuring that window boundaries and content changes are synchronized. It also achieves cross-modal collaboration through periodic analysis to address inter-modal delay issues. Finally, it establishes a multi-level attention mechanism to dynamically balance the contributions of each modality and enhance feature expression capabilities.

[0109] In an optional embodiment, constructing a context state based on video features, audio features, and user interaction features, performing multi-feature change point detection and multi-scale verification on the video features, audio features, and user interaction features to determine an initial time window sequence, optimizing the window structure through threshold control, and dynamically adjusting the window parameters through autocorrelation analysis, periodogram analysis, and context state mapping include:

[0110] Calculate the temporal changes of video features, audio features, and user interaction features to obtain interaction density indicators, content rhythm indicators, and modal collaboration indicators. Construct a multi-dimensional context feature vector and perform dynamic clustering to obtain the context state.

[0111] Perform multi-feature change point detection and multi-scale verification on video features, audio features and user interaction features, and recursively segment based on the segmentation score function of the change point detection value and the sub-window balance to obtain an initial time window sequence; calculate the interaction density score of each window in the initial time window sequence, and subdivide the windows exceeding a first preset threshold and merge the adjacent windows below a second preset threshold according to the interaction density score, and obtain an optimized time window sequence based on the window subdivision and window merging; perform autocorrelation analysis and periodogram analysis on the optimized time window sequence, wherein the autocorrelation analysis obtains periodic features by calculating the autocorrelation coefficient under time delay, and the periodogram analysis obtains spectral features by calculating the power spectral density; fuse the periodic features and spectral features at different time scales to obtain multi-scale periodic features, construct a periodic synergy matrix based on the multi-scale periodic features, and obtain a cross-modal synergy periodic group through spectral clustering;

[0112] A probability mapping relationship between the situation state and the window parameters is established, the window parameters at each time scale are dynamically adjusted in combination with the cross-modal collaborative period group, and an optimal window parameter set is generated based on the current situation state.

[0113] Exemplarily, the temporal change calculation of multimodal features is performed, and the rate of change of video features, audio features, and user interaction features over continuous time segments is calculated. The rate of change of video features is calculated by the cosine distance between feature vectors of adjacent frames, the rate of change of audio features is calculated by the difference in energy spectra, and the rate of change of user interaction features is calculated by a weighted combination of barrage density, like frequency, and gift giving frequency. Based on these change rates, three types of indicators are calculated: interaction density index, content rhythm index, and modal collaboration index.

[0114] The interaction density index is calculated using user interaction data within a sliding window (window size of 15 seconds, step size of 5 seconds). The index value is the weighted sum of the number of bullet comments, number of likes, and number of gifts, with weights set to 0.4, 0.35, and 0.25, respectively. For example, if there are 87 bullet comments, 215 likes, and 8 gifts in a 15-second window, the calculated interaction density index value is (87 × 0.4) + (215 × 0.35) + (8 × 0.25) = 112.8.

[0115] The content cadence metric reflects the rate of content change. It is calculated by weighting the change rates of video and audio features, with weights of 0.6 and 0.4, respectively. The average feature change rate within a 10-second window is calculated. For example, if the average change rate of video features within a window is 0.28 and the average change rate of audio features is 0.35, the resulting content cadence metric is 0.28 × 0.6 + 0.35 × 0.4 = 0.308.

[0116] The modal collaboration index measures the synchronization between different modalities. It is calculated by calculating the mutual information between video and audio features, video and user interaction features, and audio and user interaction features. The weights of these three mutual information sets are 0.4, 0.35, and 0.25, respectively. For example, if the three mutual information sets are 0.65, 0.48, and 0.52, respectively, the calculated modal collaboration index is 0.65 × 0.4 + 0.48 × 0.35 + 0.52 × 0.25 = 0.562.

[0117] These three categories of indicators were combined into a 24-dimensional contextual feature vector (each category contains multiple statistics). Dynamic clustering was performed using the K-means algorithm, with a K value of 6. The clustering results corresponded to six contextual states: high interaction period, content climax period, transition period, calm period, interaction start period, and interaction subsidence period. The center vector of each category served as a representative representation of that contextual state. For example, the mean value of the category center interaction density index for the high interaction period was 158.6, the mean value of the content rhythm index was 0.48, and the mean value of the modal synergy index was 0.72.

[0118] Multi-feature change point detection and multi-scale validation were performed. The PELT algorithm was applied to each of the three modal features for change point detection, with a penalty parameter and minimum interval length set. For each candidate change point, the average feature vector of the preceding and following windows was calculated, and the Euclidean distance between them was used as the change point score. The ratio of the lengths of the preceding and following sub-windows was also calculated as the balance score. The change point score was combined with the balance score to calculate a composite score. Points with a composite score exceeding the set threshold were considered valid change points.

[0119] A segmentation score function is constructed based on the detected change points, and a top-down recursive segmentation strategy is adopted. Starting from the entire time series, the comprehensive score of each possible segmentation point is calculated, and the point with the highest score is selected for segmentation. This process is recursively performed until the length of all subsequences is less than the minimum window size (5 seconds) or there are no segmentation points with scores exceeding the threshold. For example, for a 120-second live broadcast clip, the valid segmentation points detected are 72 seconds, 35 seconds, 95 seconds, and 108 seconds, respectively. The video is finally divided into 5 initial time windows: [0, 35), [35, 72), [72, 95), [95, 108), and [108, 120).

[0120] Then the window structure is optimized. The interaction density score of each initial window is calculated and obtained by weighting the barrage density, like frequency and gift frequency. Assume that the first preset threshold is 75.0 and the second preset threshold is 20.0. The windows with an interaction density score exceeding 75.0 are subdivided and repeatedly subdivided using the bisection method until the interaction density score of the sub-window is lower than the first preset threshold or the window length is less than the minimum allowed length (3 seconds). Adjacent windows with an interaction density score lower than 20.0 are merged, and the interaction density score of the merged window is the average of the original window weighted by duration. For example, the interaction density scores of the above five initial windows are 65.2, 82.5, 48.3, 22.4 and 16.8 respectively. The score of the second window exceeds the first preset threshold, and it is subdivided into two sub-windows [35, 53) and [53, 72); the scores of the fourth and fifth windows are both lower than the second preset threshold, and they are merged into one window [95, 120). The optimized window sequences are [0, 35), [35, 53), [53, 72), [72, 95), and [95, 120).

[0121] Autocorrelation analysis was performed on the optimized window sequence. For each modal feature, the autocorrelation coefficient sequence with a maximum time delay of 60 seconds was calculated. Autocorrelation peaks were extracted in 10-second increments, and the time delays corresponding to these peaks were the potential period lengths. A periodogram analysis was also performed, using a fast Fourier transform to calculate the power spectral density. The first three frequencies with the highest power were extracted, and their reciprocals were used as period estimates. For example, autocorrelation analysis of the video features revealed significant peaks at delays of 13 and 26 seconds; periodogram analysis revealed that the main periods were 12.5, 25, and 38 seconds.

[0122] The periodic features obtained from autocorrelation and periodogram analysis are divided into three time scales: short-term (3-10 seconds), medium-term (10-30 seconds), and long-term (30-60 seconds). For each scale, the weighted average of the autocorrelation peak and the spectral peak is calculated as the period estimate for that scale. The weight is determined by the significance of the peak, which is calculated as the ratio of the peak to the surrounding average. The periodic features of the three modes at the three time scales are combined into a 9-dimensional multi-scale periodic feature vector. For example, the multi-scale periodic features of a live broadcast segment are [8.5, 24.2, 45.8, 7.2, 18.5, 38.2, 5.6, 15.4, 32.7], where the first three values ​​represent the three scale periods of video features, the middle three represent the three scale periods of audio features, and the last three represent the three scale periods of user interaction features.

[0123] A periodicity synergy matrix is ​​constructed based on multi-scale periodicity features. The matrix elements represent the similarity between periods of different modalities and scales, with similarity calculated by the proximity of the periodicity ratios. A spectral clustering algorithm is applied to the periodicity synergy matrix, with the number of clusters set to 4, to obtain groups of cross-modal synergistic periods. For example, the clustering results may reveal that the short-term periods of video features are highly synergistic with the short-term periods of user interaction features, and that the medium-term periods of audio features are highly synergistic with the medium-term periods of video features.

[0124] A probability mapping relationship between the context state and the window parameters is established. The window parameters include window size, sliding step size, and overlap ratio. Through statistical analysis, the probability distribution of the optimal window parameters in each context state is calculated. For example, in the high-interaction context state, the window size has a mean of 7.5 seconds and a standard deviation of 1.2 seconds; the sliding step size has a mean of 2.5 seconds and a standard deviation of 0.8 seconds; the overlap ratio has a mean of 0.65 and a standard deviation of 0.08. The optimal window parameter set is generated by combining the currently detected context state and the cross-modal collaborative cycle group. For example, when the current context state is detected to be the content climax period and the collaborative cycle group shows that the mid-term cycle is dominant, the optimal window parameters generated are: window size of 18 seconds, sliding step size of 5 seconds, and overlap ratio of 0.72.

[0125] Figure 4 The following figure analyzes the multimodal feature fusion performance under different contextual states. It demonstrates the multimodal feature fusion performance of the multi-time window dynamic calibration network of the present invention under six different contextual states and compares it with three existing methods. It can be seen that the dynamic calibration method employed by the present invention significantly outperforms other feature fusion methods in all contextual states. Specifically, during periods of high interaction, the present invention achieved a fusion accuracy of 94.5%, which is 18.3 percentage points higher than the simple feature splicing method, 8.0 percentage points higher than the attention mechanism, and 10.8 percentage points higher than the multi-layer perceptron. Particularly noteworthy is the stable performance of the present invention across different contextual states. Even in relatively calm scenes (such as quiet periods), the fusion accuracy remains high at 88.7%. In contrast, the performance of other methods fluctuates significantly between contextual states, with the simple feature splicing method achieving an accuracy of only 70.8% during periods of subsiding interaction. This result fully demonstrates that the present invention, through the inter-modal attention weights and inter-window attention weights generated based on the correlation matrix, combined with the multi-level attention fusion network approach, effectively captures the inter-modal relationships under different contextual states and dynamically adjusts the feature fusion strategy based on real-time content changes. Especially during the high interaction period and content climax period when the content changes drastically, the advantages of the present invention are more obvious, reaching 94.5% and 93.8% fusion accuracy respectively.

[0126] The multi-feature change point detection and window parameter dynamic adjustment method of the present invention achieves accurate segmentation and intelligent time window division of live broadcast content by integrating video, audio and user interaction features. The situational state is constructed based on three indicators: interaction density, content rhythm and modal collaboration. Combined with PELT change point detection and adaptive window optimization, the system can automatically adjust the analysis granularity according to content changes. Multi-scale periodic features are extracted through autocorrelation analysis and periodogram analysis, which realizes the identification of cross-modal collaborative periodic groups, greatly improving the accuracy and timeliness of live broadcast quality assessment while reducing computing resource consumption.

[0127] In an optional embodiment, the multimodal features are processed using a cascade network structure to output a comprehensive score of the anchor's live broadcast. The comprehensive score includes an interactivity score, a professional score, and an appeal score.

[0128] The multimodal features are decomposed into temporal features to extract the temporal patterns of expressions, actions, voices, and interactions, and a dynamic attention matrix is ​​constructed based on the scene type for feature adaptive enhancement. A cascade structure is constructed, comprising a bottom-level technical scoring subnetwork, a middle-level perception scoring subnetwork, and a top-level comprehensive scoring subnetwork. The bottom-level technical scoring subnetwork is modeled based on the enhanced temporal features and obtains technical indicator scores through feature fusion. The middle-level perception scoring subnetwork dynamically adjusts the weights of technical indicators based on the scene-aware attention unit to construct a correlation matrix between scoring dimensions and generate interactivity scores, professionalism scores, and appeal scores through a graph attention network. The top-level comprehensive scoring subnetwork uses a skip connection mechanism to fuse multiple layers of features.

[0129] An inter-dimensional learning mechanism is established between each scoring sub-network, and a multi-objective loss function including scoring dimension consistency loss, scoring standard deviation loss and knowledge distillation loss is constructed for optimization. The scoring dimension consistency loss is used to constrain the relative relationship between different scoring dimensions, the scoring standard deviation loss adaptively adjusts the scoring scale based on the distribution of historical scoring data, and the knowledge distillation loss guides the information interaction of the features of each scoring dimension. A phased training strategy is adopted to optimize the multi-objective loss function through the process of pre-training-distillation-fine-tuning to obtain a comprehensive score.

[0130] For example, multimodal features are decomposed into temporal features to extract four types of temporal patterns: expression, movement, speech, and interaction. For expression features, the frequency of micro-expression changes, expression duration, and expression switching fluency are extracted. For example, for singing live broadcasts, the average expression change frequency is 12.5 times per minute, the average expression duration is 4.8 seconds, and the expression fluency score is 0.82. For movement features, movement amplitude, movement frequency, and movement stability are extracted. For example, for dance live broadcasts, the average movement amplitude is 0.78, the average movement frequency is 35 times per minute, and the movement stability score is 0.91. For speech features, speech rate, volume variation, and intonation are extracted. For example, the average speech rate of commentary live broadcasts is 180 words per minute, the volume variation variance is 6.2 decibels, and the intonation score is 0.73. For interaction features, the interaction response time, interaction frequency, and interaction depth are extracted. For example, the average interaction response time of e-commerce live broadcasts is 1.8 seconds, the interaction frequency is 4.2 times per minute, and the interaction depth score is 0.85.

[0131] A dynamic attention matrix is ​​constructed based on the scene type for adaptive feature enhancement. First, a feature importance template is designed for each scene type. For example, the game live broadcast scene emphasizes action accuracy and reaction speed, with weights of 0.32 and 0.28 respectively; the talent show scene emphasizes rich expression and action coordination, with weights of 0.35 and 0.38 respectively; the knowledge sharing scene emphasizes language clarity and content organization, with weights of 0.40 and 0.35 respectively. Then, based on the currently identified scene type and confidence level, the corresponding feature importance template is selected to generate attention weights. For mixed scenes, a weighted fusion method is adopted. For example, for a scene identified as 70% talent show and 30% interactive chat, its attention weight is a weighted combination of the two scene templates. Finally, the attention weights are applied to the temporal features to enhance the influence of key features.

[0132] A scoring network with a three-layer cascade structure was constructed. The underlying technical scoring subnetwork was modeled based on the enhanced temporal features and was designed to include five technical indicator scoring modules: naturalness of expression, coordination of movements, voice expressiveness, timeliness of interaction, and richness of content. Each module used a three-layer fully connected network with hidden layer dimensions of 512 and 256, and a ReLU activation function. For example, the naturalness of expression module inputs the temporal features of expression and outputs a score between 0 and 100; the coordination of movements module inputs the temporal features of movements and outputs a score between 0 and 100. The scoring results of the five technical indicators were combined through a feature fusion network consisting of a fully connected layer and an attention pooling layer, which outputs 10 fine-grained technical indicator scores, such as 89 points for vividness of expression, 92 points for naturalness of expression, 95 points for smoothness of movement, and 88 points for accuracy of movement.

[0133] The mid-level perceptual scoring subnetwork dynamically adjusts the weights of technical indicators based on a scene-aware attention unit. This attention unit consists of a scene feature extractor and a weight generator. The scene feature extractor is a modified version of a pretrained ResNet50 network, with the final fully connected layer removed and a global average pooling layer added. It outputs a 2048-dimensional scene feature vector. The weight generator is a two-layer fully connected network that takes the scene feature vector as input and outputs weights for ten technical indicators. For example, for a dance performance, movement smoothness is weighted 0.18, movement accuracy is weighted 0.15, and expression vividness is weighted 0.14. These weights are applied to the technical indicator scores, and a correlation matrix is ​​constructed between the scoring dimensions. The matrix elements represent the degree of correlation between different technical indicators. This correlation matrix is ​​processed using a graph attention network, which consists of two layers of graph convolution and one layer of attention pooling. The network ultimately generates scores for three dimensions: interactivity, professionalism, and appeal. For example, an e-commerce live stream received a score of 92 for interactivity, 87 for professionalism, and 94 for appeal.

[0134] The top-level comprehensive scoring subnetwork uses a skip connection mechanism to fuse multiple layers of features. Features are extracted from the bottom-level technical scoring subnetwork and the middle-level perceptual scoring subnetwork, respectively, and then aggregated to the top-level network via skip connections. The top-level network consists of a feature fusion layer and a score generation layer. The feature fusion layer integrates features from different levels using a gating mechanism, with the gating values ​​calculated using a single-layer neural network. The score generation layer is a three-layer fully connected network with hidden layer dimensions of 256 and 128, outputting a comprehensive score between 0 and 100. For example, combining the three-dimensional scores of the e-commerce live broadcast above, the final comprehensive score is 91 points.

[0135] A cross-dimensional learning mechanism is established between each scoring sub-network. A three-part multi-objective loss function is constructed for optimization. The scoring dimension consistency loss is used to constrain the relative relationship between different scoring dimensions, ensuring the consistency and interpretability of the scoring system. This is achieved by calculating the relative difference between the predicted score and the standard score. For example, the professional score should be 5 points higher than the interactivity score. If the predicted result does not meet this relationship, a corresponding loss is incurred. The scoring standard deviation loss adaptively adjusts the scoring scale based on the distribution of historical scoring data. This is achieved by calculating the difference between the distribution of the current predicted score and the distribution of historical scores. For example, the mean of the historical e-commerce live broadcast interactivity scores is 80 points, with a standard deviation of 8 points. If the current scores are too concentrated or dispersed, a large loss is incurred. The knowledge distillation loss guides the information interaction of the features of each scoring dimension and is calculated by the output difference between the teacher network and the student network. The teacher network is a pre-trained single-dimensional scoring expert model, and the student network is the current multi-dimensional scoring model.

[0136] A phased training strategy is adopted. In the pre-training phase, five technical indicator scoring modules and three dimensional scoring networks are trained separately, using labeled datasets and corresponding single-objective loss functions. For example, the expression naturalness module is trained for 20 rounds using 5,000 labeled samples, with a learning rate of 0.001 and a batch size of 64. In the distillation phase, the pre-trained model is frozen as the teacher network, and the student network containing knowledge distillation loss is trained. For example, the knowledge of the interactive scoring teacher network is transferred to the multi-dimensional scoring network, and 3,000 samples are used for training for 10 rounds with a learning rate of 0.0005. In the fine-tuning phase, the entire network is jointly optimized using the complete multi-objective loss function. For example, the weights of the scoring dimension consistency loss, scoring standard deviation loss, and knowledge distillation loss are set to 0.4, 0.3, and 0.3 respectively, and 2,000 samples are used for training for 5 rounds with a learning rate of 0.0001.

[0137] Through the evaluation of the cascade scoring network, different types of live streaming have different scoring characteristics. For example, game live streaming generally scores higher for professionalism than for interactivity, with an average difference of 7.5 points; talent show live streaming generally scores higher for appeal than professionalism, with an average difference of 5.2 points; and e-commerce live streaming generally scores higher for interactivity, with an average of 85.6 points and a standard deviation of 6.8 points. The system can provide customized scoring criteria for different scenarios to meet diverse evaluation needs.

[0138] This invention achieves accurate assessment of live broadcast quality through a cascaded network structure. Time series feature decomposition extracts multimodal information features, and a dynamic attention matrix adaptively enhances key features based on the scenario type. A three-tiered cascaded scoring structure progresses from technical indicators to perceptual dimensions and then to comprehensive evaluation. Interactive learning mechanisms between dimensions ensure the consistency and interpretability of the scoring system.

[0139] According to a second aspect of the embodiments of the present invention,

[0140] Provides a deep learning-based live broadcast data feature extraction and scoring system, including:

[0141] The first unit is used to collect video data, audio data and user interaction data during the anchor's live broadcast;

[0142] The second unit is configured to identify the scene type of the video data through a scene recognition subnetwork; construct a feature extraction network including a backbone network and an adaptive enhancement module, wherein the backbone network extracts an initial feature map containing expression, action, and scene information through a three-dimensional convolutional structure; and the adaptive enhancement module assigns attention weights to different feature dimensions of the initial feature map according to the scene type to generate video features; generate audio features based on the audio data; and calculate user interaction features based on the user interaction data according to a time window.

[0143] The third unit is used to build a multi-time window dynamic calibration network. Based on the correlation matrix of the video features, audio features and user interaction features in different time windows, it performs feature fusion through inter-modal attention weights and inter-window attention weights, and combines temporal alignment and window quality evaluation to obtain multimodal features;

[0144] The fourth unit is used to process the multimodal features using a cascade network structure and output a comprehensive score of the anchor's live broadcast, where the comprehensive score includes an interactivity score, a professionalism score, and an appeal score.

[0145] According to a third aspect of the embodiments of the present invention,

[0146] An electronic device is provided, comprising:

[0147] processor;

[0148] a memory for storing processor-executable instructions;

[0149] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0150] According to a fourth aspect of the embodiments of the present invention,

[0151] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0152] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting and scoring anchor live broadcast data features based on deep learning, characterized in that: include: Collect video data, audio data, and user interaction data from the host's live broadcast; Identifying the scene type of the video data by a scene recognition sub-network; Constructing a feature extraction network comprising a backbone network and an adaptive enhancement module. The backbone network extracts an initial feature map containing expression, action, and scene information through a three-dimensional convolutional structure. The adaptive enhancement module assigns attention weights to different feature dimensions of the initial feature map according to the scene type to generate video features. Generating audio features based on the audio data. Calculating user interaction features for the user interaction data according to a time window; Construct a multi-time window dynamic calibration network. Based on the correlation matrix of the video features, audio features, and user interaction features in different time windows, perform feature fusion through inter-modal attention weights and inter-window attention weights, and combine temporal alignment and window quality evaluation to obtain multimodal features. The multimodal features are processed using a cascade network structure to output a comprehensive score for the anchor's live broadcast, which includes an interactivity score, a professional score, and an appeal score.

2. The method according to claim 1, characterized in that Identifying the scene type of the video data by the scene recognition sub-network includes: The YOLOv5 target detection network is used to perform target detection and region segmentation on video data frames, extract the anchor area, background area and object area, and perform size normalization on the regional image blocks; feature maps of the regional image blocks are extracted respectively, and multi-scale features are extracted through spatial pyramid pooling. The multi-scale features are reduced in dimensionality and then spliced ​​to form a spliced ​​scene feature vector; the scene feature vector is input into a multi-layer fully connected network for feature transformation, and the scene category probability distribution is calculated through the softmax activation function to determine the scene type.

3. The method according to claim 1, characterized in that Construct a feature extraction network including a backbone network and an adaptive enhancement module. The backbone network extracts an initial feature map containing expression, action, and scene information through a three-dimensional convolutional structure. The adaptive enhancement module assigns attention weights to different feature dimensions of the initial feature map according to the scene type. Generating video features includes: Constructing a backbone network and adaptive enhancement module. The backbone network includes three branches: micro-expressions, body movements, and scene context. Three-dimensional convolution kernels of different sizes are used to extract and splice the initial feature map. The adaptive enhancement module includes: obtaining a main category probability distribution through a main category classification head, selecting a subdivision category classification head based on the main category probability distribution to obtain a conditional probability distribution, applying the main category probability distribution and the conditional probability distribution to an embedding matrix to obtain a main category embedding vector and a subdivision category embedding vector, and performing bilinear fusion to obtain a hierarchical scene embedding; hierarchically dynamically modulating the initial feature map based on the hierarchical scene embedding, using the main category embedding vector in a shallow layer, using the main category and subdivision category embedding vectors in a middle layer, and using the hierarchical scene embedding in a deep layer; Features modulated at different levels are constructed into a lateral information transfer network through conditional gating units, and multi-scale modulation features are obtained by integration through a feature pyramid network. Corresponding channel attention weights and spatial attention weights are generated for the multi-scale modulation features through hierarchical scene embedding and applied, and a normalized feature map is obtained by combining residual connection and layer normalization. Linear transformation parameters are generated for each scene type based on the hierarchical scene embedding, and the transformation results are weighted and fused according to the scene type probability distribution of the normalized feature map to obtain video features.

4. The method according to claim 3, characterized in that The backbone network also includes an expression-action collaborative modeling module, which includes: The features of the micro-expression branch and the body movement branch are mapped into a unified space. The inter-modal similarity is calculated based on scaled dot-product attention. The inter-modal similarity is adaptively fused with the relative position encoding of the feature points to generate dynamic association weights. Multi-scale collaborative features are extracted through a graph convolutional network. Based on the multi-scale collaborative features, local window units and global window units are constructed, wherein the local window unit uses a sliding window convolution with a learnable step size to extract short-term emotional response features, and the learnable step size is dynamically adjusted based on the hidden state at the current moment; the global window unit combines a bidirectional recurrent neural network and an adaptive temporal attention mechanism to extract long-term performance style features, wherein the temporal attention weight is calculated by a nonlinear transformation; after feature alignment of the short-term emotional response features and the long-term performance style features, the emotion-behavior collaborative features are obtained by fusing them through an adaptive weighting method based on a gating mechanism, and the emotion-behavior collaborative features are used to generate the initial feature map.

5. The method according to claim 1, wherein A multi-time window dynamic calibration network is constructed. Based on the correlation matrix of the video features, audio features, and user interaction features in different time windows, feature fusion is performed through inter-modal attention weights and inter-window attention weights. Combined with temporal alignment and window quality evaluation, multimodal features are obtained, including: Build contextual states based on video, audio, and user interaction features. Perform multi-feature change point detection and multi-scale verification on these features to determine the initial time window sequence. Optimize the window structure through threshold control, and dynamically adjust window parameters through autocorrelation analysis, periodogram analysis, and contextual state mapping. Calculate the mean, standard deviation, maximum, and temporal trend characteristics of video features, audio features, and user interaction features in each time window, and calculate the correlation matrix between modalities; Based on the correlation matrix, inter-modality attention weights and inter-window attention weights are generated, wherein the inter-modality attention weights are calculated based on the correlation matrix, and the inter-window attention weights are obtained by mapping the correlation matrix through a nonlinear transformation function, and a multi-level attention fusion network is constructed to perform feature fusion; the inter-modality time offset is determined through cross-correlation analysis and time series alignment is performed; The Frobenius norm of the correlation matrix in each time window is calculated to obtain the window quality score, which is converted into a scale balance factor through the softmax function. The multi-scale feature contribution of the time-aligned features is balanced according to the scale balance factor to obtain multimodal features.

6. The method according to claim 5, characterized in that Contextual states are constructed based on video features, audio features, and user interaction features. Multi-feature change point detection and multi-scale verification are performed on these features to determine the initial time window sequence. The window structure is optimized through threshold control. Window parameters are dynamically adjusted through autocorrelation analysis, periodogram analysis, and contextual state mapping. These include: Calculate the temporal changes of video features, audio features, and user interaction features to obtain interaction density indicators, content rhythm indicators, and modal collaboration indicators. Construct a multi-dimensional context feature vector and perform dynamic clustering to obtain the context state. Perform multi-feature change point detection and multi-scale verification on video features, audio features and user interaction features, and recursively segment based on the segmentation score function of the change point detection value and the sub-window balance to obtain an initial time window sequence; calculate the interaction density score of each window in the initial time window sequence, and subdivide the windows exceeding a first preset threshold and merge the adjacent windows below a second preset threshold according to the interaction density score, and obtain an optimized time window sequence based on the window subdivision and window merging; perform autocorrelation analysis and periodogram analysis on the optimized time window sequence, wherein the autocorrelation analysis obtains periodic features by calculating the autocorrelation coefficient under time delay, and the periodogram analysis obtains spectral features by calculating the power spectral density; fuse the periodic features and spectral features at different time scales to obtain multi-scale periodic features, construct a periodic synergy matrix based on the multi-scale periodic features, and obtain a cross-modal synergy periodic group through spectral clustering; A probability mapping relationship between the situation state and the window parameters is established, the window parameters at each time scale are dynamically adjusted in combination with the cross-modal collaborative period group, and an optimal window parameter set is generated based on the current situation state.

7. The method according to claim 1, characterized in that The multimodal features are processed using a cascade network structure to output a comprehensive score for the anchor's live broadcast. The comprehensive score includes an interactivity score, a professional score, and an appeal score. The multimodal features are decomposed into temporal features to extract the temporal patterns of expressions, actions, voices, and interactions, and a dynamic attention matrix is ​​constructed based on the scene type for feature adaptive enhancement. A cascade structure is constructed, comprising a bottom-level technical scoring subnetwork, a middle-level perception scoring subnetwork, and a top-level comprehensive scoring subnetwork. The bottom-level technical scoring subnetwork is modeled based on the enhanced temporal features and obtains technical indicator scores through feature fusion. The middle-level perception scoring subnetwork dynamically adjusts the weights of technical indicators based on the scene-aware attention unit to construct a correlation matrix between scoring dimensions and generate interactivity scores, professionalism scores, and appeal scores through a graph attention network. The top-level comprehensive scoring subnetwork uses a skip connection mechanism to fuse multiple layers of features. An inter-dimensional learning mechanism is established between each scoring sub-network, and a multi-objective loss function including scoring dimension consistency loss, scoring standard deviation loss and knowledge distillation loss is constructed for optimization. The scoring dimension consistency loss is used to constrain the relative relationship between different scoring dimensions, the scoring standard deviation loss adaptively adjusts the scoring scale based on the distribution of historical scoring data, and the knowledge distillation loss guides the information interaction of the features of each scoring dimension. A phased training strategy is adopted to optimize the multi-objective loss function through the process of pre-training-distillation-fine-tuning to obtain a comprehensive score.

8. A deep learning-based anchor live broadcast data feature extraction and scoring system for implementing the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to collect video data, audio data and user interaction data during the anchor's live broadcast; The second unit is configured to identify the scene type of the video data through a scene recognition subnetwork; construct a feature extraction network including a backbone network and an adaptive enhancement module, wherein the backbone network extracts an initial feature map containing expression, action, and scene information through a three-dimensional convolutional structure, and the adaptive enhancement module assigns attention weights to different feature dimensions of the initial feature map according to the scene type to generate video features; and generate audio features based on the audio data; Calculating user interaction features for the user interaction data according to a time window; The third unit is used to build a multi-time window dynamic calibration network. Based on the correlation matrix of the video features, audio features and user interaction features in different time windows, it performs feature fusion through inter-modal attention weights and inter-window attention weights, and combines temporal alignment and window quality evaluation to obtain multimodal features; The fourth unit is used to process the multimodal features using a cascade network structure and output a comprehensive score of the anchor's live broadcast, where the comprehensive score includes an interactivity score, a professionalism score, and an appeal score.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images