Live broadcast room content recognition and intelligent distribution method and system based on multimodal fusion

Abstract

The present invention provides a method and system for content recognition and intelligent distribution in a live broadcast room based on multimodal fusion, which relates to the field of machine learning technology, including collecting live broadcast room video, audio and barrage data, and extracting features respectively; using a multimodal feature fusion model to perform cross-modal interaction, feature alignment and feature reconstruction on multimodal features to generate cross-modal fusion features; and performing content recognition based on a pre-trained content classification model to obtain category labels and time series features. Further, a content portrait and an associated network of the live broadcast room are constructed, and the user's historical viewing behavior data and a deep reinforcement learning model are combined to generate a real-time rating for the live broadcast room, and dynamically adjust the display weight and push strategy of the live broadcast room. Through multimodal data fusion and deep reinforcement learning, the present invention realizes accurate recognition and personalized recommendation of live broadcast room content, effectively improving user experience and platform operation efficiency.

Description

Technical Field

[0001] The present invention relates to machine learning technology, and in particular to a method and system for identifying and intelligently distributing live broadcast room content based on multimodal fusion. Background Art

[0002] Traditional methods for identifying live broadcast content mainly rely on manual review or simple keyword matching, which is inefficient and inaccurate. Traditional methods for distributing live broadcast content are usually based on simple rules or collaborative filtering algorithms, which are difficult to capture users' complex and changing interests and preferences, resulting in inaccurate recommendation results. The existing technologies mainly have the following defects and deficiencies:

[0003] Insufficient fusion of multimodal information: Traditional live broadcast content analysis methods usually only focus on single modal information, such as analyzing only video images or audio content, ignoring the complementarity and correlation between different modal information, resulting in incomplete and inaccurate content understanding.

[0004] Limited content recognition accuracy: Most existing content recognition methods are based on simple feature extraction and classification models, which are difficult to handle complex live broadcast scenarios and content, resulting in limited recognition accuracy, especially when dealing with live broadcast rooms with multiple themes and rapidly changing content.

[0005] Insufficient distribution efficiency and personalization: Traditional recommendation algorithms are difficult to effectively capture users' real-time interests and behavioral changes, and are unable to make personalized recommendations based on dynamic data such as user viewing history, interactive behavior, and length of stay, resulting in insufficient distribution efficiency and user satisfaction. Summary of the invention

[0006] The embodiments of the present invention provide a method and system for identifying and intelligently distributing live broadcast content based on multimodal fusion, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention,

[0008] Provides a live broadcast content recognition and intelligent distribution method based on multimodal fusion, including:

[0009] Collect the video stream, audio stream and bullet comment data of the live broadcast room; monitor the resolution and frame rate of the video stream in real time. When abnormal video quality is detected, the backup encoder is triggered to repair the image quality and then the video feature vector is obtained by key frame processing through the convolutional neural network; detect the signal-to-noise ratio of the audio stream. When abnormal audio quality is detected, the noise reduction process is triggered and the audio feature vector is obtained through the acoustic feature extraction network; filter the bullet comment data for sensitive words and obtain the text feature vector by word segmentation and semantic understanding through the natural language processing model;

[0010] Inputting the multimodal initial feature data including the video feature vector, the audio feature vector and the text feature vector into a preset multimodal feature fusion model, the multimodal feature fusion model adopts the attention mechanism to perform cross-modal interaction and feature alignment on the video feature vector, the audio feature vector and the text feature vector. In the feature fusion process, when it is detected that a certain modality data is abnormally missing, the feature is reconstructed through the feature compensation mechanism of other modality data to generate cross-modal fusion features; performing multi-dimensional content recognition on the cross-modal fusion features through the pre-trained content classification model to obtain the category label and time series features of the live broadcast room content;

[0011] A content portrait of the live broadcast room is constructed based on the category labels and temporal features, and an association network between the contents of the live broadcast room is constructed based on the knowledge graph. The live broadcast rooms with similar subject features are clustered and grouped using a graph neural network to obtain content group information. The content portrait, the association network and the content group information are input into a deep reinforcement learning model in combination with the user's historical viewing behavior data. The deep reinforcement learning model is updated online based on the user's real-time interactive behavior and stay time data to generate a real-time scoring result for the live broadcast room, and the display weights and push strategies of different types of live broadcast rooms are dynamically adjusted according to the real-time scoring results.

[0012] In an optional embodiment,

[0013] Inputting the multimodal initial feature data including the video feature vector, the audio feature vector and the text feature vector into a preset multimodal feature fusion model, the multimodal feature fusion model uses an attention mechanism to perform cross-modal interaction and feature alignment on the video feature vector, the audio feature vector and the text feature vector. During the feature fusion process, when it is detected that a certain modality data is abnormally missing, feature reconstruction is performed through a feature compensation mechanism of other modality data. The steps of generating cross-modal fusion features include:

[0014] Performing dimensionality unification processing on the multimodal initial feature data, including: mapping the video feature vector through a fully connected layer, the audio feature vector through a two-layer perceptron, and the text feature vector through a mapping layer to a feature space of uniform dimension, and performing regularization processing on the mapped features to obtain preprocessed features;

[0015] In a multimodal feature fusion model, the preprocessed features are subjected to cross-modal interaction, including: the features after unified dimension are subjected to feature transformation through three independent linear projection layers, each linear projection layer comprises a query matrix for generating a cross-modal query vector, a key matrix for generating a key vector, and a value matrix for generating a value vector, and the query matrix, the key matrix, and the value matrix are initialized using a Xavier initialization method that obeys a uniform distribution; the preprocessed features are distributed in parallel to multiple attention heads, each attention head calculates an inter-modal attention score using a corresponding cross-modal query vector and a key vector and obtains a modal fusion attention weight through a softmax function, the modal fusion attention weight is matrix multiplied with the corresponding value vector to obtain an output feature of the attention head, and a random dropout operation is applied to the output feature; the output features of all attention heads are concatenated and processed through a multi-layer stacked attention module to obtain an interactive feature;

[0016] A dynamic programming algorithm is used to perform time-series alignment on the interaction features, the interaction features are organized into a feature sequence according to the time dimension, a distance matrix of the feature sequence is constructed, the minimum alignment cost of the feature sequence is calculated based on the distance matrix, and the interaction features after time-series alignment are adaptively fused through a gating mechanism to obtain cross-modal fusion features;

[0017] During the feature fusion process, the missing modality is detected. When the missing modality is detected, a conditional variational autoencoder is used to reconstruct the missing features. The encoder of the conditional variational autoencoder encodes the existing modal features into latent variables, and the decoder reconstructs the missing features based on the latent variables.

[0018] In an optional embodiment,

[0019] The steps of using a dynamic programming algorithm to perform time-series alignment on the interaction features, organizing the interaction features according to the time dimension to form a feature sequence, constructing a distance matrix of the feature sequence, calculating the minimum alignment cost of the feature sequence based on the distance matrix, and adaptively fusing the time-series aligned interaction features through a gating mechanism to obtain cross-modal fusion features include:

[0020] Calculating the mutual correlation coefficients of the interaction features at adjacent time steps and constructing a time series correlation matrix, wherein each element in the time series correlation matrix represents the degree of correlation between the interaction features of two corresponding time steps calculated by cosine similarity;

[0021] A dynamic time warping algorithm is used to construct a distance matrix of feature sequences. The dynamic time warping algorithm recursively calculates the minimum cumulative distance between two feature sequences and introduces a bandwidth constraint to limit the search range of the alignment path. The range of the bandwidth constraint is determined according to the length of the feature sequence. Each element of the distance matrix represents the Euclidean distance between the feature sequences.

[0022] Based on the distance matrix, a dynamic programming algorithm is used to solve the optimal alignment path. The dynamic programming algorithm calculates the optimal path by constructing a state transfer equation. The state transfer equation considers three state transfer situations: insertion cost, deletion cost, and replacement cost. The alignment sequence subscript pair is obtained by backtracking the state transfer equation;

[0023] Based on the aligned sequence index pairs, feature adaptive fusion is performed, and the feature adaptive fusion adopts a long short-term memory network for a gating mechanism. The long short-term memory network consists of a forget gate, an input gate, and an output gate, wherein the forget gate controls the degree of forgetting of historical information, the input gate controls the degree of updating of current input information, and the output gate controls the degree of output at the current time step. The hidden state of the long short-term memory network is linearly transformed to obtain the final cross-modal fusion feature.

[0024] In an optional embodiment,

[0025] The steps of performing multi-dimensional content recognition on the cross-modal fusion features through a pre-trained content classification model to obtain category labels and time series features of the live broadcast room content include:

[0026] Acquire a knowledge base of a pre-trained content classification model, perform a bidirectional cross-attention calculation on the cross-modal fusion feature and the knowledge base to obtain a first interaction feature and a second interaction feature, and adaptively fuse the first interaction feature and the second interaction feature through a gating mechanism to obtain a semantic representation;

[0027] A hierarchical content classifier is constructed based on the semantic representation, and projection mapping is used to perform feature transformation on the semantic representation to obtain topic features, and the topic features are compared with similar samples and heterogeneous samples to obtain topic classification results; multi-scale convolution is performed on the semantic representation to obtain local features, and global features are obtained through a self-attention mechanism, and the local features and the global features are fused to obtain a sentiment classification result; the semantic representation is encoded through a bidirectional gated recurrent unit to obtain an interaction type classification result, wherein the topic classification result, the sentiment classification result, and the interaction type classification result share the semantic representation;

[0028] Construct a feature memory bank to store historical semantic representations obtained by historical processing, calculate the similarity between each semantic representation in a semantic representation sequence within a time window and the historical semantic representation, perform feature correction on the semantic representation in the semantic representation sequence based on the similarity to obtain a corrected semantic representation, and update the semantic representation in the semantic representation sequence to the feature memory bank when the similarity is less than a preset threshold; use a long short-term memory network to perform temporal encoding on the corrected semantic representation to obtain a hidden state feature, calculate a temporal dynamic attention weight based on the hidden state feature, and weight the corrected semantic representation according to the temporal dynamic attention weight to obtain a dynamic temporal feature;

[0029] A confidence assessment network is constructed based on the content classification model. The confidence assessment network performs confidence assessment on the topic classification results, the sentiment classification results and the interaction type classification results to obtain confidence weights. The topic classification results, the sentiment classification results and the interaction type classification results are weighted based on the confidence weights to obtain category labels. The dynamic time series features are feature spliced ​​with the topic classification results, the sentiment classification results, the interaction type classification results and the confidence weights of the category labels to obtain time series features.

[0030] In an optional embodiment,

[0031] The steps of constructing a content portrait of the live broadcast room based on the category label and the time series feature, constructing an association network between the content of the live broadcast room based on the knowledge graph, and clustering and grouping the live broadcast rooms with similar subject features using a graph neural network to obtain content group information include:

[0032] The topic classification results, sentiment classification results and interaction type classification results in the category label are respectively mapped to the corresponding feature vector space to obtain the topic feature vector, sentiment feature vector and interaction feature vector, the temporal feature is subjected to the temporal attention mechanism to obtain the temporal feature vector, and the topic feature vector, the sentiment feature vector, the interaction feature vector and the temporal feature vector are feature-fused to obtain the content portrait vector of the live broadcast room;

[0033] Extracting semantic feature entities from the content portrait vector, retrieving adjacent entities of the semantic feature entity based on the pre-trained knowledge graph, calculating the semantic association strength between the semantic feature entity and the adjacent entity using knowledge graph entity embedding, constructing the association degree between the live broadcast rooms based on the semantic association strength, and generating an adjacency matrix of the association network of the live broadcast rooms according to the association degree, wherein each element of the adjacency matrix represents the degree of association between the content portrait vectors of the corresponding live broadcast rooms;

[0034] Determine a neighbor node set of the live broadcast room according to the adjacency matrix, perform weighted aggregation on the content portrait vectors in the neighbor node set based on the graph attention mechanism to obtain node messages, perform feature fusion on the node messages and the corresponding content portrait vectors, and perform information propagation on the fused features through a multi-layer graph attention network to obtain node representation features of the association network;

[0035] A similarity matrix of the node representation features of the association network is constructed, a standardized Laplace matrix is ​​calculated based on the similarity matrix, the eigenvalues ​​and eigenvectors of the standardized Laplace matrix are solved, and the solution results are used to cluster the live broadcast rooms with similar theme features to obtain content group information.

[0036] In an optional embodiment,

[0037] The content portrait, the associated network and the content group information are combined with the user's historical viewing behavior data into a deep reinforcement learning model, the deep reinforcement learning model is updated online based on the user's real-time interactive behavior and stay time data, and a real-time scoring result of the live broadcast room is generated. The steps of dynamically adjusting the display weights and push strategies of different types of live broadcast rooms according to the real-time scoring results include:

[0038] The content portrait, association network, content group information and user historical viewing behavior data are feature-fused to obtain a state vector; based on the state vector, the user's behavior sequence in the live broadcast room is obtained, and the behavior sequence is modeled using a hierarchical long short-term memory network, wherein the bottom long short-term memory network encodes the behavior pattern in a single session to obtain a session representation, and the top long short-term memory network integrates multiple session representations to obtain a user's long-term interest representation, and the policy network parameters of the deep reinforcement learning model are updated based on the user's long-term interest representation;

[0039] Construct a hierarchical attention value network, obtain content features, behavior features and group features through the content layer, time layer and group layer attention mechanisms respectively, fuse the content features, behavior features and group features through a cross-layer adaptive gating mechanism to obtain multi-layer attention fusion features, and generate real-time scoring results for the live broadcast room;

[0040] Based on the state vector, the user's long-term interest representation and the multi-layer attention fusion feature, a dynamic heterogeneous graph including live broadcast room nodes and user nodes is constructed, the local subgraph structure of each node is extracted, the local subgraph structure is input into the graph attention network to obtain graph features, the graph features are fused with the real-time scoring results and then input into the gated graph convolution network to obtain graph enhanced node representation, the push probability of the live broadcast room is calculated based on the graph enhanced node representation, and the display weights of different content groups are adaptively adjusted;

[0041] A push strategy is generated based on the push probability and the graph enhanced node representation. The push strategy sorts the live broadcast rooms in real time based on the real-time scoring results, determines the personalized push timing based on the user's long-term interest representation, and dynamically adjusts the push ratio of different types of live broadcast rooms according to the display weight.

[0042] In an optional embodiment,

[0043] Constructing a hierarchical attention value network, obtaining content features, behavior features and group features through the content layer, time layer and group layer attention mechanisms respectively, fusing the content features, behavior features and group features through a cross-layer adaptive gating mechanism to obtain multi-layer attention fusion features, and generating real-time scoring results for the live broadcast room includes the following steps:

[0044] The live broadcast room state information is encoded to obtain a live broadcast room feature vector; a nonlinear transformation is performed on the state vector to obtain a content-aware query vector, the correlation between the content-aware query vector and the live broadcast room feature vector is calculated to obtain an attention correlation, the attention correlation is normalized to obtain a content layer attention weight, and a weighted sum is performed on the live broadcast room feature vector based on the content layer attention weight to obtain a content feature;

[0045] The behavior sequence and the session representation are concatenated to obtain a behavior time series vector, the behavior time series vector is nonlinearly transformed to obtain a behavior importance score, the behavior importance score is normalized to obtain a behavior attention distribution, and the behavior sequence is weightedly fused based on the behavior attention distribution to obtain a behavior feature;

[0046] The content feature is concatenated with the user's long-term interest representation to obtain a group input vector, the group input vector is nonlinearly transformed to obtain a group attention score, the group attention score is normalized to obtain a group attention weight, and the group input vector is weightedly fused based on the group attention weight to obtain a group feature;

[0047] The content feature, the behavior feature and the group feature are concatenated, the importance score of each feature is calculated by a sigmoid function to obtain a feature weight, and the content feature, the behavior feature and the group feature are weightedly fused based on the feature weight to obtain a multi-layer attention fusion feature;

[0048] Perform multi-layer nonlinear transformation on the multi-layer attention fusion features to obtain initial features for score prediction; based on the initial features for score prediction and the state at the previous moment, apply a gated recurrent unit to calculate a reset gating vector and an update gating vector, fuse the reset gating vector, the update gating vector, the state at the previous moment and the initial features for score prediction to obtain a current state, and convert the current state into a real-time scoring result for the live broadcast room through linear mapping.

[0049] According to a second aspect of the embodiments of the present invention,

[0050] Provides a live broadcast content recognition and intelligent distribution system based on multimodal fusion, including:

[0051] The first unit is used to collect the video stream, audio stream and bullet screen data of the live broadcast room; the resolution and frame rate of the video stream are monitored in real time. When abnormal video quality is detected, the backup encoder is triggered to repair the image quality, and then the key frame processing is performed through the convolutional neural network to obtain the video feature vector; the signal-to-noise ratio of the audio stream is detected. When abnormal audio quality is detected, the noise reduction processing is triggered and the audio feature vector is obtained through the acoustic feature extraction network; the bullet screen data is filtered for sensitive words, and then the word segmentation and semantic understanding are performed through the natural language processing model to obtain the text feature vector;

[0052] The second unit is used to input the multimodal initial feature data including the video feature vector, the audio feature vector and the text feature vector into a preset multimodal feature fusion model, wherein the multimodal feature fusion model uses an attention mechanism to perform cross-modal interaction and feature alignment on the video feature vector, the audio feature vector and the text feature vector. During the feature fusion process, when it is detected that a certain modality data is abnormally missing, feature reconstruction is performed through a feature compensation mechanism of other modality data to generate cross-modal fusion features; multi-dimensional content recognition is performed on the cross-modal fusion features through a pre-trained content classification model to obtain category labels and time series features of the live broadcast room content;

[0053] The third unit is used to construct a content portrait of the live broadcast room based on the category label and time series features, and to construct an association network between the contents of the live broadcast room based on the knowledge graph, and to cluster and group the live broadcast rooms with similar subject features using a graph neural network to obtain content group information; the content portrait, the association network and the content group information are input into a deep reinforcement learning model in combination with the user's historical viewing behavior data, and the deep reinforcement learning model is updated online based on the user's real-time interactive behavior and stay time data to generate a real-time scoring result for the live broadcast room, and the display weights and push strategies of different types of live broadcast rooms are dynamically adjusted according to the real-time scoring results.

[0054] According to a third aspect of the embodiments of the present invention,

[0055] An electronic device is provided, comprising:

[0056] processor;

[0057] a memory for storing processor-executable instructions;

[0058] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0059] A fourth aspect of the embodiments of the present invention is:

[0060] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0061] The present invention uses multimodal fusion technology to combine video, audio and barrage data, and utilizes the attention mechanism for cross-modal interaction and feature alignment, so as to understand the content of the live broadcast room more comprehensively and deeply. Even when a certain modal data is abnormally missing, feature compensation and reconstruction can be performed through other modal data, thereby improving the accuracy and robustness of content recognition.

[0062] The present invention is based on the category labels, temporal features and association networks constructed by knowledge graphs of the live broadcast room content, combined with the user's historical viewing behavior and real-time interaction data, and uses a deep reinforcement learning model to dynamically adjust the display weight and push strategy of the live broadcast room, thereby achieving personalized recommendations for thousands of people, improving the user's viewing experience and the platform's traffic distribution efficiency.

[0063] By constructing content portraits and associated networks of live broadcast rooms, the present invention can cluster and group live broadcast rooms and conduct thematic analysis, helping the platform to better understand the ecological distribution and development trends of live broadcast room content, thereby achieving more refined content management and operation and improving the overall content quality of the platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A flowchart of a method for identifying and intelligently distributing live broadcast room content based on multimodal fusion according to an embodiment of the present invention;

[0065] Figure 2 It is a structural diagram of a live broadcast room content recognition and intelligent distribution system based on multimodal fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0068] Figure 1 FIG. 1 is a flow chart of a method for identifying and intelligently distributing content in a live broadcast room based on multimodal fusion according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0069] S1. Collect the video stream, audio stream and bullet screen data of the live broadcast room; monitor the resolution and frame rate of the video stream in real time. When abnormal video quality is detected, the backup encoder is triggered to repair the image quality and then the key frame is processed through the convolutional neural network to obtain the video feature vector; detect the signal-to-noise ratio of the audio stream. When abnormal audio quality is detected, the noise reduction process is triggered and the audio feature vector is obtained through the acoustic feature extraction network; the bullet screen data is filtered for sensitive words and the word segmentation and semantic understanding are performed through the natural language processing model to obtain the text feature vector;

[0070] S2. Input the multimodal initial feature data including the video feature vector, the audio feature vector and the text feature vector into a preset multimodal feature fusion model, wherein the multimodal feature fusion model uses an attention mechanism to perform cross-modal interaction and feature alignment on the video feature vector, the audio feature vector and the text feature vector. During the feature fusion process, when it is detected that a certain modality data is abnormally missing, feature reconstruction is performed through the feature compensation mechanism of other modality data to generate cross-modal fusion features; multi-dimensional content recognition is performed on the cross-modal fusion features through a pre-trained content classification model to obtain the category label and time series features of the live broadcast room content;

[0071] S3. Construct a content portrait of the live broadcast room based on the category labels and time series features, and construct an association network between the contents of the live broadcast room based on the knowledge graph, and use a graph neural network to cluster and group the live broadcast rooms with similar subject features to obtain content group information; input the content portrait, the association network and the content group information into a deep reinforcement learning model in combination with the user's historical viewing behavior data, and the deep reinforcement learning model is updated online based on the user's real-time interactive behavior and stay time data to generate a real-time scoring result for the live broadcast room, and dynamically adjust the display weights and push strategies of different types of live broadcast rooms according to the real-time scoring results.

[0072] In an optional embodiment,

[0073] Inputting the multimodal initial feature data including the video feature vector, the audio feature vector and the text feature vector into a preset multimodal feature fusion model, the multimodal feature fusion model uses an attention mechanism to perform cross-modal interaction and feature alignment on the video feature vector, the audio feature vector and the text feature vector. During the feature fusion process, when it is detected that a certain modality data is abnormally missing, feature reconstruction is performed through a feature compensation mechanism of other modality data. The steps of generating cross-modal fusion features include:

[0074] Performing dimensionality unification processing on the multimodal initial feature data, including: mapping the video feature vector through a fully connected layer, the audio feature vector through a two-layer perceptron, and the text feature vector through a mapping layer to a feature space of uniform dimension, and performing regularization processing on the mapped features to obtain preprocessed features;

[0075] In a multimodal feature fusion model, the preprocessed features are subjected to cross-modal interaction, including: the features after unified dimension are subjected to feature transformation through three independent linear projection layers, each linear projection layer comprises a query matrix for generating a cross-modal query vector, a key matrix for generating a key vector, and a value matrix for generating a value vector, and the query matrix, the key matrix, and the value matrix are initialized using a Xavier initialization method that obeys a uniform distribution; the preprocessed features are distributed in parallel to multiple attention heads, each attention head calculates an inter-modal attention score using a corresponding cross-modal query vector and a key vector and obtains a modal fusion attention weight through a softmax function, the modal fusion attention weight is matrix multiplied with the corresponding value vector to obtain an output feature of the attention head, and a random dropout operation is applied to the output feature; the output features of all attention heads are concatenated and processed through a multi-layer stacked attention module to obtain an interactive feature;

[0076] A dynamic programming algorithm is used to perform time-series alignment on the interaction features, the interaction features are organized into a feature sequence according to the time dimension, a distance matrix of the feature sequence is constructed, the minimum alignment cost of the feature sequence is calculated based on the distance matrix, and the interaction features after time-series alignment are adaptively fused through a gating mechanism to obtain cross-modal fusion features;

[0077] During the feature fusion process, the missing modality is detected. When the missing modality is detected, a conditional variational autoencoder is used to reconstruct the missing features. The encoder of the conditional variational autoencoder encodes the existing modal features into latent variables, and the decoder reconstructs the missing features based on the latent variables.

[0078] Exemplarily, first, the initial feature data of video, audio and text are dimensionalized. The video feature vector is transformed through a fully connected layer; the audio feature vector is transformed through a perceptron containing two layers; the text feature vector is transformed through a mapping layer. The goal of these transformations is to map features of different modalities to feature spaces of the same dimension. For example, the initial dimension of the video feature is 512, the initial dimension of the audio feature is 256, and the initial dimension of the text feature is 768, which are unified to 1024 dimensions after transformation. Afterwards, the mapped features are regularized, such as using L2 regularization, to prevent overfitting, and obtain preprocessed features.

[0079] Next, the preprocessed features are cross-modally interacted in the multimodal feature fusion model. The video, audio, and text features after the unified dimension are transformed through three independent linear projection layers. Each linear projection layer contains three matrices: query matrix, key matrix, and value matrix. These matrices are used to generate cross-modal query vectors, key vectors, and value vectors. The Xavier initialization method is used to initialize these matrices, which makes the values ​​of the matrix elements obey uniform distribution, which helps the training stability of the model. For example, for 1024-dimensional features, the dimensions of the query matrix, key matrix, and value matrix can be set to 1024x1024. The preprocessed features are distributed to multiple attention heads in parallel, assuming that 8 attention heads are used. Each attention head uses the corresponding query vector and key vector to calculate the attention score between the modalities. The attention score represents the correlation between the features of different modalities. Then, the attention score is converted into the modal fusion attention weight through the softmax function, so that the weight value falls between 0 and 1, and the sum of all weights is 1. The modal fusion attention weight is matrix multiplied with the corresponding value vector to obtain the output feature of the attention head. A random dropout operation is applied to the output features, for example, the dropout rate is set to 0.1 to enhance the generalization ability of the model and prevent overfitting. After the output features of all attention heads are concatenated, they are processed through a multi-layer stack of attention modules to obtain the final interaction features.

[0080] Then, a dynamic programming algorithm is used to align the interactive features in time. The interactive features are organized into a feature sequence according to the time dimension. For example, for a 10-second video, one feature is extracted per second, forming a feature sequence of length 10. A distance matrix of the feature sequence is constructed to measure the similarity between features at different time points. For example, the distance can be calculated using cosine similarity. Based on the distance matrix, the dynamic time warping (DTW) algorithm is used to calculate the minimum alignment cost of the feature sequence and find the best matching path. The interactive features after time alignment are adaptively fused through a gating mechanism. The gating mechanism dynamically adjusts the fusion weights of different modal features according to their importance to obtain cross-modal fusion features.

[0081] During the feature fusion process, it is necessary to detect the missing modality. For example, if a video has no audio information, the audio modality is considered missing. When the missing modality is detected, the conditional variational autoencoder is used to reconstruct the missing features. The encoder of the conditional variational autoencoder encodes the existing modal features into latent variables. For example, if the audio is missing, the video and text features are encoded as latent variables. The decoder reconstructs the missing audio features based on the latent variables.

[0082] Through cross-modal interaction and feature alignment, the present invention can effectively fuse information from different modalities, thereby enhancing the representation capability of the final features and improving the performance of downstream tasks. It can effectively handle the situation of missing modalities, compensate for missing information through a feature reconstruction mechanism, improve the robustness of the model, use a dynamic programming algorithm for timing alignment, and a gating mechanism for adaptive fusion, which can better adapt to multimodal data of different lengths and different feature distributions.

[0083] In an optional embodiment,

[0084] The steps of using a dynamic programming algorithm to perform time-series alignment on the interaction features, organizing the interaction features according to the time dimension to form a feature sequence, constructing a distance matrix of the feature sequence, calculating the minimum alignment cost of the feature sequence based on the distance matrix, and adaptively fusing the time-series aligned interaction features through a gating mechanism to obtain cross-modal fusion features include:

[0085] Calculating the mutual correlation coefficients of the interaction features at adjacent time steps and constructing a time series correlation matrix, wherein each element in the time series correlation matrix represents the degree of correlation between the interaction features of two corresponding time steps calculated by cosine similarity;

[0086] A dynamic time warping algorithm is used to construct a distance matrix of feature sequences. The dynamic time warping algorithm recursively calculates the minimum cumulative distance between two feature sequences and introduces a bandwidth constraint to limit the search range of the alignment path. The range of the bandwidth constraint is determined according to the length of the feature sequence. Each element of the distance matrix represents the Euclidean distance between the feature sequences.

[0087] Based on the distance matrix, a dynamic programming algorithm is used to solve the optimal alignment path. The dynamic programming algorithm calculates the optimal path by constructing a state transfer equation. The state transfer equation considers three state transfer situations: insertion cost, deletion cost, and replacement cost. The alignment sequence subscript pair is obtained by backtracking the state transfer equation;

[0088] Based on the aligned sequence index pairs, feature adaptive fusion is performed, and the feature adaptive fusion adopts a long short-term memory network for a gating mechanism. The long short-term memory network consists of a forget gate, an input gate, and an output gate, wherein the forget gate controls the degree of forgetting of historical information, the input gate controls the degree of updating of current input information, and the output gate controls the degree of output at the current time step. The hidden state of the long short-term memory network is linearly transformed to obtain the final cross-modal fusion feature.

[0089] Exemplarily, first, the time series correlation matrix of the interaction features is calculated. Assume that there are feature sequences of two modes, denoted as mode A and mode B, and the feature sequence of each mode contains feature vectors of multiple time steps. For the feature vectors of any two time steps of mode A and mode B, such as the feature vector of the i-th time step of mode A and the feature vector of the j-th time step of mode B, calculate the cosine similarity between the two feature vectors. The cosine similarity values ​​of the feature vector pairs of all time steps are organized into a matrix, namely the time series correlation matrix. Each element of the matrix represents the degree of association between the interaction features of two corresponding time steps. For example, assuming that mode A has 3 time steps and mode B has 4 time steps, the time series correlation matrix is ​​a 3x4 matrix.

[0090] Next, the distance matrix of the feature sequence is constructed based on the dynamic time warping algorithm. Using the time series correlation matrix calculated in the previous step, the dynamic time warping algorithm is used to calculate the minimum cumulative distance between the two feature sequences. The dynamic time warping algorithm calculates the distance between the two sequences in a recursive manner and allows a certain degree of time distortion between the sequences. In order to limit the search range of the alignment path, a bandwidth constraint is introduced. The range of the bandwidth constraint is determined according to the length of the feature sequence, for example, it can be set to 1 / 4 of the length of the feature sequence. Each element of the distance matrix represents the Euclidean distance between the feature sequences. For example, if mode A has 3 time steps and mode B has 4 time steps, the distance matrix is ​​a 3x4 matrix, and each element represents the Euclidean distance between the feature vector of a time step in mode A and the feature vector of a time step in mode B.

[0091] Then, based on the distance matrix, a dynamic programming algorithm is used to solve the optimal alignment path. The dynamic programming algorithm calculates the optimal path by constructing a state transfer equation. The state transfer equation considers three state transfer situations: insertion cost, deletion cost, and replacement cost. The insertion cost and deletion cost can be set to the same value, such as 1, and the replacement cost can be set to the value of the corresponding element in the distance matrix. By backtracking the state transfer equation, the alignment sequence subscript pair can be obtained. For example, if the optimal alignment path is (1, 1), (2, 2), (3, 3), it means that the first time step of mode A is aligned with the first time step of mode B, the second time step of mode A is aligned with the second time step of mode B, and so on.

[0092] Finally, adaptive feature fusion is performed based on the aligned sequence index pairs. The long short-term memory network is used for the gating mechanism, which consists of a forget gate, an input gate, and an output gate. The forget gate controls the degree of forgetting of historical information, the input gate controls the degree of updating of the current input information, and the output gate controls the degree of output at the current time step. The aligned feature sequence is input into the long short-term memory network, and the hidden state of the long short-term memory network is linearly transformed to obtain the final cross-modal fusion feature. For example, the aligned feature vectors of modality A and modality B are spliced ​​together as the input of the long short-term memory network.

[0093] Through timing alignment and adaptive fusion, the present invention can better capture the correlation information between different modalities, thereby improving the characterization capability of cross-modal features. The dynamic time warping algorithm can effectively handle the timing differences between different modalities, thereby enhancing the robustness of the model to timing changes. By introducing bandwidth constraints, the search range of the alignment path can be effectively limited, thereby simplifying the model training process.

[0094] In an optional embodiment,

[0095] The steps of performing multi-dimensional content recognition on the cross-modal fusion features through a pre-trained content classification model to obtain category labels and time series features of the live broadcast room content include:

[0096] Acquire a knowledge base of a pre-trained content classification model, perform a bidirectional cross-attention calculation on the cross-modal fusion feature and the knowledge base to obtain a first interaction feature and a second interaction feature, and adaptively fuse the first interaction feature and the second interaction feature through a gating mechanism to obtain a semantic representation;

[0097] A hierarchical content classifier is constructed based on the semantic representation, and projection mapping is used to perform feature transformation on the semantic representation to obtain topic features, and the topic features are compared with similar samples and heterogeneous samples to obtain topic classification results; multi-scale convolution is performed on the semantic representation to obtain local features, and global features are obtained through a self-attention mechanism, and the local features and the global features are fused to obtain a sentiment classification result; the semantic representation is encoded through a bidirectional gated recurrent unit to obtain an interaction type classification result, wherein the topic classification result, the sentiment classification result, and the interaction type classification result share the semantic representation;

[0098] Construct a feature memory bank to store historical semantic representations obtained by historical processing, calculate the similarity between each semantic representation in a semantic representation sequence within a time window and the historical semantic representation, perform feature correction on the semantic representation in the semantic representation sequence based on the similarity to obtain a corrected semantic representation, and update the semantic representation in the semantic representation sequence to the feature memory bank when the similarity is less than a preset threshold; use a long short-term memory network to perform temporal encoding on the corrected semantic representation to obtain a hidden state feature, calculate a temporal dynamic attention weight based on the hidden state feature, and weight the corrected semantic representation according to the temporal dynamic attention weight to obtain a dynamic temporal feature;

[0099] A confidence assessment network is constructed based on the content classification model. The confidence assessment network performs confidence assessment on the topic classification results, the sentiment classification results and the interaction type classification results to obtain confidence weights. The topic classification results, the sentiment classification results and the interaction type classification results are weighted based on the confidence weights to obtain category labels. The dynamic time series features are feature spliced ​​with the topic classification results, the sentiment classification results, the interaction type classification results and the confidence weights of the category labels to obtain time series features.

[0100] Exemplarily, obtaining multimodal data of the live broadcast room: first obtain multimodal data of the live broadcast room, including video frames, audio signals, text bullets, etc. For example, a video frame is collected every 1 second, and the corresponding audio clips and text bullets are recorded at the same time.

[0101] Extract cross-modal fusion features: Use pre-trained models to extract feature representations of multimodal data. For example, use ResNet50 to extract visual features of video frames, use Whisper (whisper model) to extract speech features of audio signals, and use BERT (bidirectional encoder representation transformation model) to extract text features of text comments. These features are concatenated into a cross-modal fusion feature vector, for example, 512-dimensional visual features, 256-dimensional speech features, and 768-dimensional text features are concatenated into a 1536-dimensional vector.

[0102] Get the knowledge base of the pre-trained content classification model: Use a public dataset containing a large amount of text and labeled data to train a text classification model. Use the last layer parameters of the model as the knowledge base of the content classification model. For example, the knowledge base can be a matrix of dimension 10000x768, where each row represents a category and each column represents a feature dimension.

[0103] Calculate interactive features: Perform bidirectional cross-attention calculation on the cross-modal fusion features and the knowledge base. For example, perform attention calculation on the 1536-dimensional cross-modal fusion features and the 10000x768 knowledge base to obtain the first interactive feature (1536 dimensions) and the second interactive feature (768 dimensions). These two interactive features represent the mutual influence between the cross-modal features and the knowledge base.

[0104] Fusion semantic representation: The first interaction feature and the second interaction feature are adaptively fused through a gating mechanism to obtain the final semantic representation. For example, a sigmoid function is used to control the fusion ratio of the two features to obtain a 1536-dimensional semantic representation vector.

[0105] Construct a hierarchical content classifier: Construct a hierarchical content classifier based on semantic representation, which includes three parts: topic classification, sentiment classification, and interaction type classification.

[0106] Topic classification: Projection mapping is used to transform the semantic representation to obtain topic features. For example, the 1536-dimensional semantic representation is projected into a 256-dimensional topic feature space. The topic features are compared with similar samples and heterogeneous samples to obtain topic classification results, such as "games", "singing", "food", etc. Assuming that the topic features of a sample are closer to samples in the "game" category, it is classified as the "game" topic.

[0107] Sentiment classification: Perform multi-scale convolution on semantic representation to obtain local features, such as using convolution kernels of different sizes to extract local features of different scales. Obtain global features through the self-attention mechanism, such as using the Transformer model to extract global features. Fuse local features with global features to obtain sentiment classification results, such as "positive", "negative", "neutral", etc. Assuming that both the local features and global features of a sample show a positive tendency, it is classified as a "positive" sentiment.

[0108] Interaction type classification: The semantic representation is encoded through a bidirectional gated recurrent unit to obtain the interaction type classification result, such as "single-person live broadcast", "multi-person live broadcast", "audience interaction", etc. Assuming that the semantic representation of a sample contains information about multi-person conversations, it is classified as the "multi-person live broadcast" interaction type.

[0109] Build a feature memory: Build a feature memory to store the semantic representations obtained from historical processing. For example, use a queue to store the semantic representations of the last 10 minutes.

[0110] Feature correction: Calculate the similarity between each semantic representation in the semantic representation sequence within the time window and the historical semantic representation, for example, using cosine similarity. Perform feature correction on the semantic representations in the semantic representation sequence based on the similarity. For example, if the current semantic representation has a high similarity with the historical semantic representation, the current semantic representation is considered reliable and is not modified; if the similarity is low, the historical semantic representation is used to correct the current semantic representation.

[0111] Update feature memory: When the similarity is less than a preset threshold, the semantic representation in the semantic representation sequence is updated to the feature memory. For example, if the similarity between the current semantic representation and all historical semantic representations is less than 0.8, the current semantic representation is added to the feature memory.

[0112] Temporal encoding: Long short-term memory network is used to perform temporal encoding on the corrected semantic representation to obtain hidden state features. For example, a semantic representation sequence within 10 seconds is input into the LSTM network to obtain the hidden state features of each time step.

[0113] Calculate the temporal dynamic attention weights: Calculate the temporal dynamic attention weights based on the hidden state features. For example, use the softmax function to calculate the attention weights for each time step.

[0114] Weighted dynamic temporal features: The corrected semantic representation is weighted according to the temporal dynamic attention weight to obtain the dynamic temporal features. For example, the semantic representation of each time step is multiplied by the corresponding attention weight, and then the sum is obtained to obtain the final dynamic temporal features.

[0115] Confidence assessment: Build a confidence assessment network to conduct confidence assessment on the topic classification results, sentiment classification results, and interaction type classification results to obtain confidence weights. For example, use a small neural network to evaluate the classification results and output a confidence value between 0 and 1.

[0116] Weighted category labels: The topic classification results, sentiment classification results, and interaction type classification results are weighted based on the confidence weights to obtain the final category labels. For example, each classification result is multiplied by the corresponding confidence weight.

[0117] Splicing to obtain time series features: Splice the dynamic time series features with the topic classification results, sentiment classification results, interaction type classification results, and confidence weights of the category labels to obtain the final time series features.

[0118] Through cross-modal fusion features and pre-trained models, the present invention can more comprehensively understand the content of the live broadcast room and improve the accuracy of content recognition. The hierarchical content classifier can classify the content of the live broadcast room in a more fine-grained manner, such as themes, emotions, and interaction types, and provide richer content information. Through temporal feature encoding and dynamic attention mechanism, the temporal dynamic changes of the content of the live broadcast room, such as topic switching and emotional fluctuations, can be captured.

[0119] In an optional embodiment,

[0120] The steps of constructing a content portrait of the live broadcast room based on the category label and the time series feature, constructing an association network between the content of the live broadcast room based on the knowledge graph, and clustering and grouping the live broadcast rooms with similar subject features using a graph neural network to obtain content group information include:

[0121] The topic classification results, sentiment classification results and interaction type classification results in the category label are respectively mapped to the corresponding feature vector space to obtain the topic feature vector, sentiment feature vector and interaction feature vector, the temporal feature is subjected to the temporal attention mechanism to obtain the temporal feature vector, and the topic feature vector, the sentiment feature vector, the interaction feature vector and the temporal feature vector are feature-fused to obtain the content portrait vector of the live broadcast room;

[0122] Extracting semantic feature entities from the content portrait vector, retrieving adjacent entities of the semantic feature entity based on the pre-trained knowledge graph, calculating the semantic association strength between the semantic feature entity and the adjacent entity using knowledge graph entity embedding, constructing the association degree between the live broadcast rooms based on the semantic association strength, and generating an adjacency matrix of the association network of the live broadcast rooms according to the association degree, wherein each element of the adjacency matrix represents the degree of association between the content portrait vectors of the corresponding live broadcast rooms;

[0123] Determine a neighbor node set of the live broadcast room according to the adjacency matrix, perform weighted aggregation on the content portrait vectors in the neighbor node set based on the graph attention mechanism to obtain node messages, perform feature fusion on the node messages and the corresponding content portrait vectors, and perform information propagation on the fused features through a multi-layer graph attention network to obtain node representation features of the association network;

[0124] A similarity matrix of the node representation features of the association network is constructed, a standardized Laplace matrix is ​​calculated based on the similarity matrix, the eigenvalues ​​and eigenvectors of the standardized Laplace matrix are solved, and the solution results are used to cluster the live broadcast rooms with similar theme features to obtain content group information.

[0125] Exemplarily, first, build a content portrait of the live broadcast room. Obtain live broadcast room data from the live broadcast platform, including category labels and time series features. Category labels include topic classification results (such as "beauty", "game"), sentiment classification results (such as "positive", "negative"), and interaction type classification results (such as "comment", "like"). Time series features include the start time of the live broadcast room, duration, changes in the number of viewers, etc.

[0126] The results of topic classification, sentiment classification, and interaction type classification are converted into corresponding feature vectors respectively. For example, the topic classification result "beauty" can be converted into a vector [1, 0, 0], and "game" can be converted into [0, 1, 0]. Similarly, the sentiment classification results and interaction type classification results can also be converted into corresponding feature vectors. For time series features, a time series attention mechanism is used to obtain a time series feature vector. For example, the sequence of changes in the number of viewers is input into an attention network to obtain a vector of a fixed length to represent the time series characteristics of the live broadcast room. The topic feature vector, sentiment feature vector, interaction feature vector, and time series feature vector are concatenated to obtain the content portrait vector of the live broadcast room. For example, a beauty live broadcast with positive emotions, frequent interactions, and a steady increase in the number of viewers can have a final content portrait vector of [1, 0, 0, 1, 0, 1, 0, 0.8, 0.9, 0.7].

[0127] Next, the association network between the live broadcast rooms is constructed. Semantic feature entities are extracted from the content portrait vector. For example, entities such as "lipstick", "eye shadow", and "makeup skills" can be extracted from the content portrait vector of beauty live broadcast. The adjacent entities of these entities are retrieved based on the pre-trained knowledge graph. For example, the adjacent entities of "lipstick" can be "lipstick", "beauty brand", "makeup steps", etc. The semantic association strength between the semantic feature entity and the adjacent entity is calculated using the knowledge graph entity embedding. For example, the semantic similarity between "lipstick" and "lipstick" is calculated. The association between the live broadcast rooms is constructed based on the semantic association strength. If there is a strong association between the semantic feature entities of the two live broadcast rooms, the contents of the two live broadcast rooms are considered to be related. The adjacency matrix of the association network of the live broadcast rooms is generated based on the association degree. Each element of the adjacency matrix represents the degree of association between the corresponding live broadcast room content portrait vectors. For example, if the association between live broadcast room A and live broadcast room B is high, the element values ​​corresponding to A and B in the adjacency matrix are large. Assume that there are three live broadcast rooms, and the correlation between them is 0.8, 0.5, and 0.2 respectively, then the adjacency matrix can be expressed as [[1, 0.8, 0.5], [0.8, 1, 0.2], [0.5, 0.2, 1]].

[0128] Then, the graph neural network is used to cluster and group the live broadcast rooms. The neighbor node set of the live broadcast room is determined according to the adjacency matrix. For example, in the above adjacency matrix, the neighbor nodes of live broadcast room A are live broadcast room B and live broadcast room C. Based on the graph attention mechanism, the content portrait vectors in the neighbor node set are weighted aggregated to obtain the node message. For example, according to the correlation between live broadcast room A and live broadcast room B and live broadcast room C, the content portrait vectors of B and C are weighted averaged to obtain the node message of live broadcast room A. The node message is feature fused with the corresponding content portrait vector. The fused features are propagated through the multi-layer graph attention network to obtain the node representation features of the association network.

[0129] Finally, construct a similarity matrix of the node representation features of the association network. Calculate the standardized Laplace matrix based on the similarity matrix. Solve the eigenvalues ​​and eigenvectors of the standardized Laplace matrix. Use the solution results to cluster the live broadcast rooms with similar theme features to obtain content group information. For example, use the eigenvector as a new feature representation and use the K-means algorithm to cluster the live broadcast rooms to obtain different content groups.

[0130] By constructing content portraits and associated networks of live broadcast rooms, the present invention can more accurately understand the content of live broadcast rooms and user interests, thereby improving the accuracy of content recommendations, clustering and grouping live broadcast rooms with similar themes, making it easier for users to discover content of interest and improving user experience. By effectively organizing and managing the content of live broadcast rooms, the healthy development of the platform content ecology can be promoted.

[0131] In an optional embodiment,

[0132] The content portrait, the associated network and the content group information are combined with the user's historical viewing behavior data into a deep reinforcement learning model, the deep reinforcement learning model is updated online based on the user's real-time interactive behavior and stay time data, and a real-time scoring result of the live broadcast room is generated. The steps of dynamically adjusting the display weights and push strategies of different types of live broadcast rooms according to the real-time scoring results include:

[0133] The content portrait, association network, content group information and user historical viewing behavior data are feature-fused to obtain a state vector; based on the state vector, the user's behavior sequence in the live broadcast room is obtained, and the behavior sequence is modeled using a hierarchical long short-term memory network, wherein the bottom long short-term memory network encodes the behavior pattern in a single session to obtain a session representation, and the top long short-term memory network integrates multiple session representations to obtain a user's long-term interest representation, and the policy network parameters of the deep reinforcement learning model are updated based on the user's long-term interest representation;

[0134] Construct a hierarchical attention value network, obtain content features, behavior features and group features through the content layer, time layer and group layer attention mechanisms respectively, fuse the content features, behavior features and group features through a cross-layer adaptive gating mechanism to obtain multi-layer attention fusion features, and generate real-time scoring results for the live broadcast room;

[0135] Based on the state vector, the user's long-term interest representation and the multi-layer attention fusion feature, a dynamic heterogeneous graph including live broadcast room nodes and user nodes is constructed, the local subgraph structure of each node is extracted, the local subgraph structure is input into the graph attention network to obtain graph features, the graph features are fused with the real-time scoring results and then input into the gated graph convolution network to obtain graph enhanced node representation, the push probability of the live broadcast room is calculated based on the graph enhanced node representation, and the display weights of different content groups are adaptively adjusted;

[0136] A push strategy is generated based on the push probability and the graph enhanced node representation. The push strategy sorts the live broadcast rooms in real time based on the real-time scoring results, determines the personalized push timing based on the user's long-term interest representation, and dynamically adjusts the push ratio of different types of live broadcast rooms according to the display weight.

[0137] For example, the live broadcast recommendation system uses a deep reinforcement learning model, combined with content portraits, associated networks, content group information, and user historical viewing behavior data, to dynamically adjust the display weights and push strategies of different types of live broadcast rooms to improve user experience and platform benefits. The following describes in detail its implementation steps and technical details:

[0138] First, construct feature representations of live broadcast rooms and users. Collect information such as the title, label, cover image, anchor information, and group of each live broadcast room to construct a multimodal content portrait of the live broadcast room. Use common labels between live broadcast rooms, anchor cooperation relationships, etc. to build a live broadcast room association network. Divide the live broadcast content into groups such as games, entertainment, and life. Collect users' historical viewing records, likes, comments, and sharing behavior data to construct user portraits. For example, the content portrait of live broadcast room A can be expressed as {"Game", "League of Legends", "Anchor X", "High Quality"}, and there is a "common anchor" connection relationship between live broadcast room A and live broadcast room B in the association network, and live broadcast room A belongs to the "Game" group. User Xiao Wang's viewing history includes {Live Broadcast Room A, Live Broadcast Room C, Live Broadcast Room D}.

[0139] Next, the live broadcast room content portrait, associated network, content group information and user historical viewing behavior data are feature fused to obtain a state vector. Specifically, the text features are converted into word vectors, the image features are extracted using a pre-trained model, and the features of different modalities are spliced ​​into a high-dimensional vector. The user's historical viewing behavior data can construct a feature vector by counting the viewing time, number of interactions, etc. For example, the state vector of live broadcast room A can be expressed as [0.2, 0.5, 0.3, ...], and the historical viewing behavior vector of user Xiao Wang can be expressed as [10, 5, 2, ...]. These features are spliced ​​to obtain the final state vector.

[0140] Then, a deep reinforcement learning model is used to model user behavior and generate real-time ratings. A hierarchical long short-term memory network is used to model the user's behavior sequence. The bottom long short-term memory network encodes the behavior pattern within a single session, such as the user's click, watch, comment and other behavior sequences in a visit. The top long short-term memory network integrates multiple session representations to obtain the user's long-term interest representation. The user's long-term interest representation is used to update the policy network parameters of the deep reinforcement learning model. For example, if user Xiao Wang has watched multiple game live broadcasts in the past week, his long-term interest representation will be biased towards game content. A hierarchical attention value network is constructed based on the user's real-time interactive behavior and stay time data in the live broadcast room. Through the content layer, time layer and group layer attention mechanisms, content features, behavior features and group features are obtained respectively. For example, if user Xiao Wang is watching a game live broadcast, the content layer attention mechanism will pay more attention to the game-related content features. The content features, behavior features and group features are fused through the cross-layer adaptive gating mechanism to obtain multi-layer attention fusion features and generate the real-time rating results of the live broadcast room. For example, the real-time rating of live broadcast room A is 8.5 points.

[0141] Subsequently, a dynamic heterogeneous graph is constructed based on the state vector, user long-term interest representation, and multi-layer attention fusion features. The graph contains live broadcast room nodes and user nodes, and the edges represent the interactive relationship between users and live broadcast rooms. The local subgraph structure of each node is extracted, and the local subgraph structure is input into the graph attention network to obtain graph features. The graph features are fused with the real-time scoring results and input into the gated graph convolutional network to obtain the graph enhanced node representation. For example, the graph enhanced node representation of live broadcast room A includes its own features, the features of the live broadcast room associated with it, and the features of the users interacting with it. The push probability of the live broadcast room is calculated based on the graph enhanced node representation, and the display weights of different content groups are adaptively adjusted. For example, the display weight of the game group is 0.8, and the display weight of the entertainment group is 0.2.

[0142] Finally, a push strategy is generated based on the push probability and the graph enhancement node representation. The push strategy sorts the live broadcast rooms in real time based on the real-time scoring results, determines the personalized push timing based on the user's long-term interest representation, and dynamically adjusts the push ratio of different types of live broadcast rooms based on the display weight. For example, the system will give priority to pushing game live broadcast rooms with high real-time scores and that meet the long-term interests of user Xiao Wang.

[0143] The personalized recommendations of the present invention are more in line with user tastes, improve user satisfaction and retention rate, dynamically adjust the display weight of the live broadcast room, optimize traffic distribution, and improve the overall efficiency of the platform. It combines multimodal content understanding, deep reinforcement learning, graph neural network and other technologies to achieve more accurate and intelligent live broadcast recommendations.

[0144] In an optional embodiment,

[0145] Constructing a hierarchical attention value network, obtaining content features, behavior features and group features through the content layer, time layer and group layer attention mechanisms respectively, fusing the content features, behavior features and group features through a cross-layer adaptive gating mechanism to obtain multi-layer attention fusion features, and generating real-time scoring results for the live broadcast room includes the following steps:

[0146] The live broadcast room state information is encoded to obtain a live broadcast room feature vector; a nonlinear transformation is performed on the state vector to obtain a content-aware query vector, the correlation between the content-aware query vector and the live broadcast room feature vector is calculated to obtain an attention correlation, the attention correlation is normalized to obtain a content layer attention weight, and a weighted sum is performed on the live broadcast room feature vector based on the content layer attention weight to obtain a content feature;

[0147] The behavior sequence and the session representation are concatenated to obtain a behavior time series vector, the behavior time series vector is nonlinearly transformed to obtain a behavior importance score, the behavior importance score is normalized to obtain a behavior attention distribution, and the behavior sequence is weightedly fused based on the behavior attention distribution to obtain a behavior feature;

[0148] The content feature is concatenated with the user's long-term interest representation to obtain a group input vector, the group input vector is nonlinearly transformed to obtain a group attention score, the group attention score is normalized to obtain a group attention weight, and the group input vector is weightedly fused based on the group attention weight to obtain a group feature;

[0149] The content feature, the behavior feature and the group feature are concatenated, the importance score of each feature is calculated by a sigmoid function to obtain a feature weight, and the content feature, the behavior feature and the group feature are weightedly fused based on the feature weight to obtain a multi-layer attention fusion feature;

[0150] Perform multi-layer nonlinear transformation on the multi-layer attention fusion features to obtain initial features for score prediction; based on the initial features for score prediction and the state at the previous moment, apply a gated recurrent unit to calculate a reset gating vector and an update gating vector, fuse the reset gating vector, the update gating vector, the state at the previous moment and the initial features for score prediction to obtain a current state, and convert the current state into a real-time scoring result for the live broadcast room through linear mapping.

[0151] Exemplarily, first, obtain the live broadcast room status information. The live broadcast room status information includes but is not limited to the number of people online in the live broadcast room, the number of bullet screens, the number of likes, the value of gifts, the frequency of anchor speech, the frequency of user interaction, etc. This information will be encoded into a vector of fixed length, called the live broadcast room feature vector. For example, a live broadcast room feature vector can be [1000, 500, 2000, 10000, 20, 10], which respectively represents 1000 people online, 500 bullet screens, 2000 likes, 10000 virtual coins worth of gifts, 20 times per minute of anchor speech, and 10 times per minute of user interaction.

[0152] Next, calculate the content features. Perform a nonlinear transformation on the live broadcast room feature vector, such as using the ReLU activation function, to obtain a content-aware query vector. Calculate the correlation between the content-aware query vector and the live broadcast room feature vector to obtain the attention correlation. Normalize the attention correlation, such as using the Softmax function, to obtain the content layer attention weight. Perform a weighted summation of the live broadcast room feature vector based on the content layer attention weight to obtain the content feature. For example, assuming that the content layer attention weight is [0.2, 0.1, 0.3, 0.2, 0.1, 0.1], the content feature is the weighted average of the live broadcast room feature vector.

[0153] Then, the behavior features are calculated. The user's behavior sequence, such as viewing time, sending barrages, sending gifts, etc., is concatenated with the session representation, such as the user's historical viewing record, to obtain a behavior time series vector. The behavior time series vector is transformed nonlinearly, such as using the Sigmoid activation function, to obtain a behavior importance score. The behavior importance score is normalized, such as using the Softmax function, to obtain a behavior attention distribution. The behavior sequence is weighted and fused based on the behavior attention distribution to obtain the behavior features. For example, a user watched for 10 minutes, sent 5 barrages, and sent a gift worth 100 virtual coins. The behavior sequence can be expressed as [10, 5, 100].

[0154] Then, the group features are calculated. The content features are concatenated with the user's long-term interest representation, such as the type of anchor the user follows and the live content the user likes, to obtain the group input vector. The group input vector is nonlinearly transformed, such as using the Tanh activation function, to obtain the group attention score. The group attention score is normalized, such as using the Softmax function, to obtain the group attention weight. The group input vector is weighted fused based on the group attention weight to obtain the group features.

[0155] After that, multi-layer attention features are fused. Content features, behavior features, and group features are concatenated. The importance score of each feature is calculated by the Sigmoid function to obtain the feature weight. Based on the feature weight, the content features, behavior features, and group features are weighted and fused to obtain multi-layer attention fusion features. For example, assuming that the importance scores of content features, behavior features, and group features are 0.5, 0.3, and 0.2 respectively, the multi-layer attention fusion feature is the weighted average of the three.

[0156] Finally, generate real-time scoring results. Perform multi-layer nonlinear transformations on the multi-layer attention fusion features, such as using a multi-layer perceptron, to obtain the initial features for score prediction. Based on the initial features for score prediction and the state at the previous moment, apply the gated recurrent unit (GRU) to calculate the reset gate vector and the update gate vector. Fuse the reset gate vector, the update gate vector, the state at the previous moment, and the initial features for score prediction to obtain the current state. Convert the current state into the real-time scoring result of the live broadcast room through linear mapping, such as a score between 0 and 100.

[0157] Through the multi-layer attention mechanism, the model of the present invention can capture the information of the live broadcast room more comprehensively, thereby improving the accuracy of the scoring. The content layer attention focuses on the quality of the live broadcast content itself, the behavior layer attention focuses on the real-time feedback of users on the live broadcast, and the group layer attention takes into account the long-term preferences of the user group. The combination of the three can more accurately reflect the comprehensive value of the live broadcast room. The use of gated recurrent units can effectively process time series data and dynamically adjust the score according to the latest live broadcast room status, thereby enhancing the real-time nature of the score. The hierarchical attention mechanism makes the scoring results of the model more interpretable. By analyzing the attention weights at different levels, we can understand which factors contribute more to the final score, such as content quality, user interaction or user group preference, thereby providing more specific guidance for the improvement of the live broadcast room.

[0158] Figure 2 FIG. 1 is a schematic diagram of a structure of a live broadcast room content recognition and intelligent distribution system based on multimodal fusion according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0159] The first unit is used to collect the video stream, audio stream and bullet screen data of the live broadcast room; the resolution and frame rate of the video stream are monitored in real time. When abnormal video quality is detected, the backup encoder is triggered to repair the image quality, and then the key frame processing is performed through the convolutional neural network to obtain the video feature vector; the signal-to-noise ratio of the audio stream is detected. When abnormal audio quality is detected, the noise reduction processing is triggered and the audio feature vector is obtained through the acoustic feature extraction network; the bullet screen data is filtered for sensitive words, and then the word segmentation and semantic understanding are performed through the natural language processing model to obtain the text feature vector;

[0160] The second unit is used to input the multimodal initial feature data including the video feature vector, the audio feature vector and the text feature vector into a preset multimodal feature fusion model, wherein the multimodal feature fusion model uses an attention mechanism to perform cross-modal interaction and feature alignment on the video feature vector, the audio feature vector and the text feature vector. During the feature fusion process, when it is detected that a certain modality data is abnormally missing, feature reconstruction is performed through a feature compensation mechanism of other modality data to generate cross-modal fusion features; multi-dimensional content recognition is performed on the cross-modal fusion features through a pre-trained content classification model to obtain category labels and time series features of the live broadcast room content;

[0161] The third unit is used to construct a content portrait of the live broadcast room based on the category label and time series features, and to construct an association network between the contents of the live broadcast room based on the knowledge graph, and to cluster and group the live broadcast rooms with similar subject features using a graph neural network to obtain content group information; the content portrait, the association network and the content group information are input into a deep reinforcement learning model in combination with the user's historical viewing behavior data, and the deep reinforcement learning model is updated online based on the user's real-time interactive behavior and stay time data to generate a real-time scoring result for the live broadcast room, and the display weights and push strategies of different types of live broadcast rooms are dynamically adjusted according to the real-time scoring results.

[0162] According to a third aspect of the embodiments of the present invention,

[0163] An electronic device is provided, comprising:

[0164] processor;

[0165] a memory for storing processor-executable instructions;

[0166] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0167] A fourth aspect of the embodiments of the present invention is:

[0168] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0169] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying and intelligently distributing live broadcast content based on multimodal fusion, characterized in that: include: Collect video stream, audio stream and bullet comment data of the live broadcast room; The resolution and frame rate of the video stream are monitored in real time. When abnormal video quality is detected, the backup encoder is triggered to repair the image quality, and then the key frame processing is performed through the convolutional neural network to obtain the video feature vector; the signal-to-noise ratio of the audio stream is detected. When abnormal audio quality is detected, the noise reduction processing is triggered and the audio feature vector is obtained through the acoustic feature extraction network; the barrage data is filtered for sensitive words, and then the natural language processing model is used to perform word segmentation and semantic understanding to obtain the text feature vector; Inputting the multimodal initial feature data including the video feature vector, the audio feature vector and the text feature vector into a preset multimodal feature fusion model, the multimodal feature fusion model adopts the attention mechanism to perform cross-modal interaction and feature alignment on the video feature vector, the audio feature vector and the text feature vector. In the feature fusion process, when it is detected that a certain modality data is abnormally missing, the feature is reconstructed through the feature compensation mechanism of other modality data to generate cross-modal fusion features; performing multi-dimensional content recognition on the cross-modal fusion features through the pre-trained content classification model to obtain the category label and time series features of the live broadcast room content; A content portrait of the live broadcast room is constructed based on the category labels and temporal features, and an association network between the contents of the live broadcast room is constructed based on the knowledge graph. The live broadcast rooms with similar subject features are clustered and grouped using a graph neural network to obtain content group information. The content portrait, the association network and the content group information are input into a deep reinforcement learning model in combination with the user's historical viewing behavior data. The deep reinforcement learning model is updated online based on the user's real-time interactive behavior and stay time data to generate a real-time scoring result for the live broadcast room, and the display weights and push strategies of different types of live broadcast rooms are dynamically adjusted according to the real-time scoring results.

2. The method according to claim 1, characterized in that Inputting the multimodal initial feature data including the video feature vector, the audio feature vector and the text feature vector into a preset multimodal feature fusion model, the multimodal feature fusion model uses an attention mechanism to perform cross-modal interaction and feature alignment on the video feature vector, the audio feature vector and the text feature vector. During the feature fusion process, when it is detected that a certain modality data is abnormally missing, feature reconstruction is performed through a feature compensation mechanism of other modality data. The steps of generating cross-modal fusion features include: Performing dimensional unification processing on the multimodal initial feature data, including: mapping the video feature vector through a fully connected layer, the audio feature vector through a two-layer perceptron, and the text feature vector through a mapping layer to a feature space of uniform dimension, and performing regularization processing on the mapped features to obtain preprocessed features; In a multimodal feature fusion model, the preprocessed features are subjected to cross-modal interaction, including: the features after unified dimension are subjected to feature transformation through three independent linear projection layers, each linear projection layer comprises a query matrix for generating a cross-modal query vector, a key matrix for generating a key vector, and a value matrix for generating a value vector, and the query matrix, the key matrix, and the value matrix are initialized using a Xavier initialization method that obeys a uniform distribution; the preprocessed features are distributed in parallel to multiple attention heads, each attention head calculates an inter-modal attention score using a corresponding cross-modal query vector and a key vector and obtains a modal fusion attention weight through a softmax function, the modal fusion attention weight is matrix multiplied with the corresponding value vector to obtain an output feature of the attention head, and a random dropout operation is applied to the output feature; the output features of all attention heads are concatenated and processed through a multi-layer stacked attention module to obtain an interactive feature; A dynamic programming algorithm is used to perform time-series alignment on the interaction features, the interaction features are organized into a feature sequence according to the time dimension, a distance matrix of the feature sequence is constructed, the minimum alignment cost of the feature sequence is calculated based on the distance matrix, and the interaction features after time-series alignment are adaptively fused through a gating mechanism to obtain cross-modal fusion features; During the feature fusion process, the missing modality is detected. When the missing modality is detected, a conditional variational autoencoder is used to reconstruct the missing features. The encoder of the conditional variational autoencoder encodes the existing modal features into latent variables, and the decoder reconstructs the missing features based on the latent variables.

3. The method according to claim 2, characterized in that The steps of using a dynamic programming algorithm to perform time-series alignment on the interaction features, organizing the interaction features according to the time dimension to form a feature sequence, constructing a distance matrix of the feature sequence, calculating the minimum alignment cost of the feature sequence based on the distance matrix, and adaptively fusing the time-series aligned interaction features through a gating mechanism to obtain cross-modal fusion features include: Calculating the mutual correlation coefficients of the interaction features at adjacent time steps and constructing a time series correlation matrix, wherein each element in the time series correlation matrix represents the degree of correlation between the interaction features of two corresponding time steps calculated by cosine similarity; A dynamic time warping algorithm is used to construct a distance matrix of feature sequences. The dynamic time warping algorithm recursively calculates the minimum cumulative distance between two feature sequences and introduces a bandwidth constraint to limit the search range of the alignment path. The range of the bandwidth constraint is determined according to the length of the feature sequence. Each element of the distance matrix represents the Euclidean distance between the feature sequences. Based on the distance matrix, a dynamic programming algorithm is used to solve the optimal alignment path. The dynamic programming algorithm calculates the optimal path by constructing a state transfer equation. The state transfer equation considers three state transfer situations: insertion cost, deletion cost, and replacement cost. The alignment sequence subscript pair is obtained by backtracking the state transfer equation; Based on the aligned sequence index pairs, feature adaptive fusion is performed, and the feature adaptive fusion adopts a long short-term memory network for a gating mechanism. The long short-term memory network consists of a forget gate, an input gate, and an output gate, wherein the forget gate controls the degree of forgetting of historical information, the input gate controls the degree of updating of current input information, and the output gate controls the degree of output at the current time step. The hidden state of the long short-term memory network is linearly transformed to obtain the final cross-modal fusion feature.

4. The method according to claim 1, characterized in that: The steps of performing multi-dimensional content recognition on the cross-modal fusion features through a pre-trained content classification model to obtain category labels and time series features of the live broadcast room content include: Acquire a knowledge base of a pre-trained content classification model, perform a bidirectional cross-attention calculation on the cross-modal fusion feature and the knowledge base to obtain a first interaction feature and a second interaction feature, and adaptively fuse the first interaction feature and the second interaction feature through a gating mechanism to obtain a semantic representation; A hierarchical content classifier is constructed based on the semantic representation, and projection mapping is used to perform feature transformation on the semantic representation to obtain topic features, and the topic features are compared with similar samples and heterogeneous samples to obtain topic classification results; multi-scale convolution is performed on the semantic representation to obtain local features, and global features are obtained through a self-attention mechanism, and the local features and the global features are fused to obtain a sentiment classification result; the semantic representation is encoded through a bidirectional gated recurrent unit to obtain an interaction type classification result, wherein the topic classification result, the sentiment classification result, and the interaction type classification result share the semantic representation; Construct a feature memory bank to store historical semantic representations obtained by historical processing, calculate the similarity between each semantic representation in a semantic representation sequence within a time window and the historical semantic representation, perform feature correction on the semantic representation in the semantic representation sequence based on the similarity to obtain a corrected semantic representation, and update the semantic representation in the semantic representation sequence to the feature memory bank when the similarity is less than a preset threshold; use a long short-term memory network to perform temporal encoding on the corrected semantic representation to obtain a hidden state feature, calculate a temporal dynamic attention weight based on the hidden state feature, and weight the corrected semantic representation according to the temporal dynamic attention weight to obtain a dynamic temporal feature; A confidence assessment network is constructed based on the content classification model. The confidence assessment network performs confidence assessment on the topic classification results, the sentiment classification results and the interaction type classification results to obtain confidence weights. The topic classification results, the sentiment classification results and the interaction type classification results are weighted based on the confidence weights to obtain category labels. The dynamic time series features are feature spliced ​​with the topic classification results, the sentiment classification results, the interaction type classification results and the confidence weights of the category labels to obtain time series features.

5. The method according to claim 4, characterized in that The steps of constructing a content portrait of the live broadcast room based on the category label and the time series feature, constructing an association network between the content of the live broadcast room based on the knowledge graph, and clustering and grouping the live broadcast rooms with similar subject features using a graph neural network to obtain content group information include: The topic classification results, sentiment classification results and interaction type classification results in the category label are respectively mapped to the corresponding feature vector space to obtain the topic feature vector, sentiment feature vector and interaction feature vector, the temporal feature is subjected to the temporal attention mechanism to obtain the temporal feature vector, and the topic feature vector, the sentiment feature vector, the interaction feature vector and the temporal feature vector are feature-fused to obtain the content portrait vector of the live broadcast room; Extracting semantic feature entities from the content portrait vector, retrieving adjacent entities of the semantic feature entity based on the pre-trained knowledge graph, calculating the semantic association strength between the semantic feature entity and the adjacent entity using knowledge graph entity embedding, constructing the association degree between the live broadcast rooms based on the semantic association strength, and generating an adjacency matrix of the association network of the live broadcast rooms according to the association degree, wherein each element of the adjacency matrix represents the degree of association between the content portrait vectors of the corresponding live broadcast rooms; Determine a neighbor node set of the live broadcast room according to the adjacency matrix, perform weighted aggregation on the content portrait vectors in the neighbor node set based on the graph attention mechanism to obtain node messages, perform feature fusion on the node messages and the corresponding content portrait vectors, and perform information propagation on the fused features through a multi-layer graph attention network to obtain node representation features of the association network; A similarity matrix of the node representation features of the association network is constructed, a standardized Laplace matrix is ​​calculated based on the similarity matrix, the eigenvalues ​​and eigenvectors of the standardized Laplace matrix are solved, and the solution results are used to cluster the live broadcast rooms with similar theme features to obtain content group information.

6. The method according to claim 1, characterized in that The content portrait, the associated network and the content group information are combined with the user's historical viewing behavior data into a deep reinforcement learning model, the deep reinforcement learning model is updated online based on the user's real-time interactive behavior and stay time data, and a real-time scoring result of the live broadcast room is generated. The steps of dynamically adjusting the display weights and push strategies of different types of live broadcast rooms according to the real-time scoring results include: The content portrait, association network, content group information and user historical viewing behavior data are feature-fused to obtain a state vector; based on the state vector, the user's behavior sequence in the live broadcast room is obtained, and the behavior sequence is modeled using a hierarchical long short-term memory network, wherein the bottom long short-term memory network encodes the behavior pattern in a single session to obtain a session representation, and the top long short-term memory network integrates multiple session representations to obtain a user's long-term interest representation, and the policy network parameters of the deep reinforcement learning model are updated based on the user's long-term interest representation; Construct a hierarchical attention value network, obtain content features, behavior features and group features through the content layer, time layer and group layer attention mechanisms respectively, fuse the content features, behavior features and group features through a cross-layer adaptive gating mechanism to obtain multi-layer attention fusion features, and generate real-time scoring results for the live broadcast room; Based on the state vector, the user's long-term interest representation and the multi-layer attention fusion feature, a dynamic heterogeneous graph including live broadcast room nodes and user nodes is constructed, the local subgraph structure of each node is extracted, the local subgraph structure is input into the graph attention network to obtain graph features, the graph features are fused with the real-time scoring results and then input into the gated graph convolution network to obtain graph enhanced node representation, the push probability of the live broadcast room is calculated based on the graph enhanced node representation, and the display weights of different content groups are adaptively adjusted; A push strategy is generated based on the push probability and the graph enhanced node representation. The push strategy sorts the live broadcast rooms in real time based on the real-time scoring results, determines the personalized push timing based on the user's long-term interest representation, and dynamically adjusts the push ratio of different types of live broadcast rooms according to the display weight.

7. The method according to claim 6, characterized in that Constructing a hierarchical attention value network, obtaining content features, behavior features and group features through the content layer, time layer and group layer attention mechanisms respectively, fusing the content features, behavior features and group features through a cross-layer adaptive gating mechanism to obtain multi-layer attention fusion features, and generating real-time scoring results for the live broadcast room includes the following steps: The live broadcast room state information is encoded to obtain a live broadcast room feature vector; a nonlinear transformation is performed on the state vector to obtain a content-aware query vector, the correlation between the content-aware query vector and the live broadcast room feature vector is calculated to obtain an attention correlation, the attention correlation is normalized to obtain a content layer attention weight, and a weighted sum is performed on the live broadcast room feature vector based on the content layer attention weight to obtain a content feature; The behavior sequence and the session representation are concatenated to obtain a behavior time series vector, the behavior time series vector is nonlinearly transformed to obtain a behavior importance score, the behavior importance score is normalized to obtain a behavior attention distribution, and the behavior sequence is weightedly fused based on the behavior attention distribution to obtain a behavior feature; The content feature is concatenated with the user's long-term interest representation to obtain a group input vector, the group input vector is nonlinearly transformed to obtain a group attention score, the group attention score is normalized to obtain a group attention weight, and the group input vector is weightedly fused based on the group attention weight to obtain a group feature; The content feature, the behavior feature and the group feature are concatenated, the importance score of each feature is calculated by a sigmoid function to obtain a feature weight, and the content feature, the behavior feature and the group feature are weightedly fused based on the feature weight to obtain a multi-layer attention fusion feature; Perform multi-layer nonlinear transformation on the multi-layer attention fusion features to obtain initial features for score prediction; based on the initial features for score prediction and the state at the previous moment, apply a gated recurrent unit to calculate a reset gating vector and an update gating vector, fuse the reset gating vector, the update gating vector, the state at the previous moment and the initial features for score prediction to obtain a current state, and convert the current state into a real-time scoring result for the live broadcast room through linear mapping.

8. A live broadcast room content recognition and intelligent distribution system based on multimodal fusion, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to collect the video stream, audio stream and bullet screen data of the live broadcast room; The resolution and frame rate of the video stream are monitored in real time. When abnormal video quality is detected, the backup encoder is triggered to repair the image quality, and then the key frame processing is performed through the convolutional neural network to obtain the video feature vector; the signal-to-noise ratio of the audio stream is detected. When abnormal audio quality is detected, the noise reduction processing is triggered and the audio feature vector is obtained through the acoustic feature extraction network; the barrage data is filtered for sensitive words, and then the natural language processing model is used to perform word segmentation and semantic understanding to obtain the text feature vector; The second unit is used to input the multimodal initial feature data including the video feature vector, the audio feature vector and the text feature vector into a preset multimodal feature fusion model, wherein the multimodal feature fusion model uses an attention mechanism to perform cross-modal interaction and feature alignment on the video feature vector, the audio feature vector and the text feature vector. During the feature fusion process, when it is detected that a certain modality data is abnormally missing, feature reconstruction is performed through a feature compensation mechanism of other modality data to generate cross-modal fusion features; multi-dimensional content recognition is performed on the cross-modal fusion features through a pre-trained content classification model to obtain category labels and time series features of the live broadcast room content; The third unit is used to construct a content portrait of the live broadcast room based on the category label and time series features, and to construct an association network between the contents of the live broadcast room based on the knowledge graph, and to cluster and group the live broadcast rooms with similar subject features using a graph neural network to obtain content group information; the content portrait, the association network and the content group information are input into a deep reinforcement learning model in combination with the user's historical viewing behavior data, and the deep reinforcement learning model is updated online based on the user's real-time interactive behavior and stay time data to generate a real-time scoring result for the live broadcast room, and the display weights and push strategies of different types of live broadcast rooms are dynamically adjusted according to the real-time scoring results.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images