Multi-mode live broadcast data processing method and device

Abstract

The invention relates to the technical field of internet live broadcast platforms, in particular to a multi-modal live broadcast data processing method and device, and the method comprises the steps: obtaining a plurality of modal data of a target live broadcast room, and carrying out the materialization of the modal data through a knowledge graph; normalizing the entity by using an L2 norm; on the basis of multiple rounds of cross attention mechanisms, in each round, a certain mode serves as a query set, other modes serve as a key value set in sequence, a pre-trained shared matrix is used for conducting parameterization representation on the query set, Q, K and V are obtained, then activation function normalization is used for obtaining cross attention weights, and the cross attention weights are obtained. And finally, carrying out weighted summation on the attention distribution to obtain modal output, and splicing embedded features subjected to modal output to form complementary modal data so as to carry out modal fusion on multi-modal live broadcast data. Modal information such as relations, time sequences, attributes, images and numerical values is comprehensively considered, the modes are jointly fused into joint embedding, and the problem of mode missing in entity fusion is effectively relieved.

Description

Technical Field

[0001] The present application relates to the technical field of Internet live broadcast platform, and in particular to a multimodal live broadcast data processing method and device. Background Art

[0002] In recent years, multimodal applications have emerged in various fields. Researchers are concerned about how to fuse data from different modalities to achieve the complementarity of multiple heterogeneous information. For example, research on speech recognition has shown that the temporal modality provides lip movement and pronunciation information, which helps to improve speech recognition performance. Therefore, it is very valuable for deep learning to use the comprehensive semantics provided by multiple modes. At present, research on multimodal machine learning can be divided into five directions: multimodal representation, multimodal conversion, multimodal alignment, multimodal fusion, and multimodal collaborative learning. Multimodal representation learning extracts complementary and correlation information from multiple modal data to obtain information-rich multimodal feature representations, which are input into the model for downstream tasks. Existing multimodal representation learning can be divided into two categories: joint representation and collaborative representation. The difference between the two is that joint representation learning will fuse the features of multiple modalities and use the fused features to perform certain predictive tasks. It is suitable for tasks with multiple modalities input at the same time, such as multimodal object detection; while collaborative representation learning does not require fusion, but only establishes the correlation between multiple modal data, and establishes the association of modalities by constraining the similarity of intermediate features. It is suitable for processing task scenarios with only one modality as input, such as cross-modal retrieval, translation and other tasks. Multimodal representation also faces some problems: how to combine heterogeneous data from different modalities, how to deal with the lack or noise of a certain modality, etc. Multimodal conversion is used to learn the mapping of the same object from one modality to another. The problem to be solved is to generate other modal data of the same object given the data of a certain modality, such as generating its corresponding text description given an image. Its most common applications are generating video descriptions and cross-modal translation. The methods of multimodal conversion mainly include example-based conversion and generative model-based conversion: example-based multimodal conversion converts modalities by establishing a dictionary, which records the conversion relationship between modalities. Modal conversion is achieved by querying the conversion relationship in the dictionary; while the generation-based method trains a generative model, the input of the model is the data of a certain modality, and the output is the data of other modalities. Multimodal alignment is defined as finding the correspondence between subcomponents of the same object in two or more modalities, that is, it is used to establish the correspondence between different modalities of the same entity object. It can be used as a separate multimodal task, such as establishing the correspondence between words in a sentence and objects in a picture; it can also be used as the basis for other tasks, such as pre-aligning input data before multimodal fusion. The literature divides multimodal alignment into explicit alignment and implicit alignment. If the goal of model optimization is to maximize the alignment of multimodal data, it is explicit alignment. If the final optimization goal of the model is not modal alignment, and alignment is only used as an intermediate step, it is implicit alignment. The main problems currently faced by multimodal alignment are the lack of datasets that show alignment annotations and the possibility of missing some modalities.Multimodal fusion is used to make some kind of prediction by fusing the data of two modalities. Common applications include video-audio recognition, multimodal sentiment analysis, etc. Modal fusion is the core content of multimodal learning research. Many existing works are exploring how to better fuse multiple modalities to make more accurate predictions. According to the stage of modal fusion, multimodal fusion can be divided into early fusion (data-level fusion), mid-term fusion (feature-level fusion), and late fusion (decision-level fusion). Usually, modal fusion is based on modal alignment, that is, at least the input multimodal data must be corresponding.

[0003] Chinese patent CN201710390460.3 discloses a virtual robot multimodal interaction method and system applied to a video live broadcast platform. The application of the video live broadcast platform is configured with a virtual robot that assists in live broadcasting. The virtual robot has multimodal interaction capabilities. The public opinion monitoring method includes the following steps: information collection step, collecting public opinion information of the live broadcast of the current specific live broadcast room, and the public opinion information includes: watched text feedback information; public opinion monitoring step, calling text semantic understanding ability and generating public opinion monitoring results for the specific live broadcast room; scenario event response step, judging the event represented by the public opinion monitoring result, calling multimodal interaction capabilities and outputting multimodal response data through the virtual robot. It can monitor and prompt the audience's viewing feedback information in real time, use virtual robots to assist video live broadcast operations, always maintain stickiness with users, and improve user experience.

[0004] Chinese patent CN202410577482.0 discloses a real-time question-answering system and method for e-commerce live broadcast based on multimodal retrieval enhanced generation, including: constructing a vector database unit for establishing a knowledge base containing product information, return methods, promotional activities, and logistics information knowledge; a real-time question acquisition unit for receiving questions from users in the live broadcast room, filtering invalid questions and replacing abbreviations; a historical question library for storing each type of question and its frequency, and updating the question category frequency according to the similarity of the questions. Combining multimodal knowledge base retrieval with deep learning, a pre-trained large model is introduced, and fine-tuning is performed through interactive records and knowledge base data of e-commerce live broadcast scenes under multiple categories, thereby achieving high-quality answer generation; it can generate high-quality answers that meet the current live broadcast category based on the real-time live broadcast room status and product information knowledge base, rather than just relying on preset answers.

[0005] The above are all based on the application of multimodality in live broadcast scenarios, but the modality construction is not extensive enough. In live broadcast scenarios, there are real-time comments, bullet screens, anchor scripts or subtitles in chat rooms, anchor audio and video modalities, and user interactions with live broadcast content, such as likes, shares, gift giving, clicks and other behavioral data, as well as user social network information, such as attention, fans, friend relationships, etc., and purchase behaviors in live broadcasts, such as virtual goods purchases, paid subscriptions, etc. These data are crucial to the commercialization of live broadcasts, etc. It is far from enough to use only a few modal data for live broadcast room labeling, user favorite live broadcast recommendations, and anchor sentiment analysis, etc. Therefore, a multimodal live broadcast data processing method is proposed to be applicable to multiple live broadcast scenarios. Summary of the invention

[0006] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description.

[0007] The present application provides a multimodal live broadcast data processing method and device, aiming to solve the following technical problems: 1) Data dependency problem: Existing multimodal data fusion technology mainly relies on deep learning models, which require a large amount of data for training. However, the data collection and annotation process is often time-consuming, labor-intensive, and costly. By fusing data of multiple modalities, the present technical solution can reduce the demand for single modality data, reduce dependence on data, and improve the generalization ability of the model. 2) Modality correlation problem: Existing technologies often ignore the correlation between modalities when understanding and fusing data of different modalities, which may lead to inaccurate fusion results. Through the attention mechanism and deep learning model, the present technical solution can identify key information in different modalities, make full use of the correlation between modalities, and improve the accuracy of the fusion results.

[0008] In a first aspect, the present application provides a multimodal live broadcast data processing method for performing modal fusion on multimodal live broadcast data, comprising:

[0009] Obtain multiple modal data of the target live broadcast room, and use the knowledge graph to entity the modal data, specifically represented as entity e i Each sub-modality M∈{r,a,v,n,g}, where r,a,v,n,g represent the feature embedding of relation, attribute, time series, value, and graph structure respectively;

[0010] Use the L2 norm to transform the entity e i Perform normalization;

[0011] Based on the multi-round cross-attention mechanism, each round first takes a certain modality as the query set, and the other modalities are taken as the key-value set in turn, and the pre-trained shared matrices Wq, Wk, and Wv are used to parameterize them to obtain Q, K, and V. Then, the Sigmoid activation function is used to normalize them to obtain the cross-attention weights. Finally, the attention distribution is weighted and summed to obtain the modal output. The formula is as follows:

[0012] Q=mW q

[0013] K=V=pW k

[0014]

[0015] Among them, d k represents the dimension of the Q matrix, K T represents the transpose of the K matrix, σ represents the Sigmoid activation function, m, p∈M represent different sub-modalities, Represents entity e i m-modal embedding of;

[0016] The embedded features output by the modalities are spliced ​​together to form complementary modal data, so as to perform modal fusion on the multi-modal live broadcast data. The formula is as follows:

[0017]

[0018] in, represents the concatenation operation, M is the set of all modes, Represents entity e i m-modal embedding of .

[0019] Preferably, the method of obtaining multiple modal data of the target live broadcast room and materializing the modal data with a knowledge graph includes obtaining multiple modal data of the target live broadcast room, respectively extracting relationship information, attribute information, timing information, numerical information and graph structure information in the modalities, and constructing a knowledge graph based on the relationship information, attribute information, timing information, numerical information and graph structure information.

[0020] Preferably, the entity e is transformed using the L2 norm. i Normalization is performed, including: using the L2 norm to normalize the entity e i The eigenvectors in are scaled to unit length using the following formula:

[0021]

[0022] in, It is entity e i is the eigenvector of mode m, where d is the dimension of the eigenvector.

[0023] Preferably, after concatenating the embedding features outputted by the modalities to form a complementary embedding matrix, the method further comprises:

[0024] Introduce the correlation matrix S∈R Nm×Nm , Nm represents the number of modes, R represents the set of real numbers, so that each entity considers the complementary relationship between modes while also comprehensively considering the correlation between modes. After introducing cross-modal correlation modeling, the above formula is further updated as follows:

[0025]

[0026]

[0027] where · represents the dot product of vectors, Represents entity e i The m-modal embedding, S mp is the correlation score between mode m and mode p, It is entity e i Embedding vector on modality p.

[0028] In a second aspect, the present application also provides a multi-modal live broadcast data processing device, including:

[0029] Acquisition module: used to acquire multiple modal data of the target live broadcast room, and use the knowledge graph to entity the modal data, specifically represented as entity e i Each sub-modality M∈{r,a,v,n,g}, where r,a,v,n,g represent the feature embedding of relation, attribute, time series, value, and graph structure respectively;

[0030] Normalization module: used to transform the entity e using the L2 norm i Perform normalization;

[0031] Modality alignment module: It is used based on a multi-round cross-attention mechanism. In each round, a certain modality is first used as the query set, and the other modalities are used as the key value set in turn. The pre-trained shared matrices Wq, Wk, and Wv are used to parameterize them to obtain Q, K, and V. Then, the Sigmoid activation function is used to normalize them to obtain the cross-attention weights. Finally, the attention distribution is weighted and summed to obtain the modality output. The formula is as follows:

[0032] Q=mW q

[0033] K=V=pW k

[0034]

[0035] Among them, dk represents the dimension of the Q matrix, K T represents the transpose of the K matrix, σ represents the Sigmoid activation function, m, p∈M represent different sub-modalities, Represents entity e i m-modal embedding of;

[0036] Modal fusion module: It is used to splice the embedded features output by the modalities to form complementary modal data, so as to perform modal fusion on multi-modal live broadcast data. The formula is as follows:

[0037]

[0038] in, represents the concatenation operation, M is the set of all modes, Represents entity e i m-modal embedding of .

[0039] In a third aspect, the present application further provides a computing device, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, including:

[0040] When the processor executes the computer instructions, the steps of the method described in the first aspect are implemented.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium storing computer instructions, including:

[0042] When the computer instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0043] Multimodal conversion is used to learn the mapping of the same object from one modality to another. The problem to be solved is to generate data of other modalities of the same object given the data of a certain modality, for example, given an image, generate its corresponding text description. Its most common applications are generating video descriptions and cross-modal translation. The methods of multimodal conversion mainly include example-based conversion and generative model-based conversion: instance-based multimodal conversion converts modalities by establishing a dictionary, which records the conversion relationship between modalities, and modal conversion is achieved by querying the conversion relationship in the dictionary; while the generation-based method trains a generative model, the input of the model is data of a certain modality, and the output is data of other modalities.

[0044] Compared with the existing technology, the beneficial effects of this technical solution are as follows: 1. More efficient data processing: The present invention adopts multimodal data fusion technology, which can integrate multiple data sources such as video, audio, text and user interaction to provide more comprehensive and accurate content understanding. Compared with the existing technology, the present invention can process large-scale live data more efficiently and improve the real-time and accuracy of data processing. 2. More accurate emotion recognition: The present invention uses deep learning models and attention mechanisms to accurately identify the emotional states of anchors and viewers. Compared with the existing technology, the present invention is more accurate in emotion recognition, can provide richer emotion analysis results, and provide more powerful support for content recommendation and social marketing. 3. More accurate user portrait construction: The present invention combines user interaction and text analysis to construct user portraits to provide support for personalized marketing and advertising. Compared with the existing technology, the present invention can more accurately construct user portraits and improve the effects of marketing and advertising. 4. It comprehensively considers modal information such as relationships, time series, attributes, images and numerical values, and integrates the above modalities into joint embedding, which effectively alleviates the modality missing problem in entity fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0046] Figure 1 A flow chart of a multi-modal live broadcast data processing method provided in one embodiment of the present application; Figure 2 A structural schematic diagram of a multimodal live broadcast data processing device provided in one embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] In order to make the purpose, features, and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0048] The following is an explanation of the prior art used in this application:

[0049] Attention mechanism: The task of the attention mechanism is to obtain local attention information. The introduction of the attention mechanism allows the task to know which parts of the input data are more worthy of attention. It is usually expressed as Q, K, V, namely Query, Key and Value. All three are derived from the input features themselves and are vectors generated based on the input features. Q can be regarded as a vector of a single input feature. When a set of Q is directly input into the network for training, the network is a network without the introduction of the attention mechanism. However, if the attention mechanism is introduced, it is necessary to multiply this set of Q by a set of weights W (K, V) respectively, so that the local input features can be paid attention to. W (K, V) is used to calculate the similarity between K and V. The common one is the dot-product attention mechanism that uses dot product calculation, that is, Q is the vector representing the input features, K and V are the feature vectors for calculating the weights of the attention mechanism, and all three are obtained from the input features. It means that the similarity of the current Key and Value is calculated, and this similarity value is calculated through the Softmax layer to obtain a set of weights. The Value value under the attention mechanism is obtained by summing the product of this set of weights and the corresponding Value.

[0050] Knowledge Graph (KG) aims to represent relevant knowledge in the real world in a structured way, where concepts are represented as nodes and the relationships between concepts are represented as edges. These structured knowledge can also be represented as triples[1].<s,p,o> The representation form means <entity, relationship, entity> in the relationship triple and <entity, attribute, attribute value> in the attribute triple.

[0051] NLP: It is a technology that studies language problems in the interaction between people and computers.

[0052] GCN, the full name of which is Graph Convolutional Networks, is a deep learning model that specializes in processing graph structured data. It processes graph data by defining graph convolution operations, which allow the model to learn embedded representations of nodes while considering both the feature information of the nodes and the structural information of the graph. GCN can be used for tasks such as node classification, graph classification, and edge prediction, and obtains embedded representations of graphs. This network structure is particularly useful in processing non-Euclidean structured data such as social networks and chemical molecular structures, because it does not require a fixed input format and can process irregularly structured data.

[0053] In the present application, a multimodal live broadcast data processing method and device are provided, which are described in detail one by one in the following embodiments.

[0054] like Figure 1As shown, according to a multimodal live broadcast data processing method provided by an embodiment of the present application, the cover image data and text data of the target live broadcast room are fused to recommend favorite live broadcast content to users. Because when users choose their favorite live broadcast content, they will not only refer to the text description during the live broadcast, but may also be attracted by multimodal content such as image descriptions and user reviews. Therefore, it is necessary to introduce these multimodal information into the recommendation system. Compared with traditional recommendation systems, multimodal recommendation systems not only consider the user's historical behavior data, but also cover data across different media types, including text, images, audio, and video. Multimodal recommendation uses auxiliary multimodal information, which can not only supplement the historical interaction between users and live broadcast content, but also help alleviate the data sparsity problem in the recommendation system, thereby improving recommendation performance. In addition, multimodal recommendation can reveal the intrinsic relationship between different modalities, and can mine supplementary information that cannot be captured by single-modal recommendation and implicit interaction. Specifically including the following steps:

[0055] S10, obtain multiple modal data of the target live broadcast room, and use the knowledge graph to entity the modal data, specifically represented as entity e i Each sub-modality M∈{r,a,v,n,g}, where r represents the relationship, a represents the attribute, v represents the time series, n represents the value, and g represents the graph structure;

[0056] Acquire multiple modal data of the target live broadcast room, extract the relationship information, attribute information, timing information, numerical information and graph structure information in the modalities respectively, and construct a knowledge graph based on the relationship information, attribute information, timing information, numerical information and graph structure information. Before constructing a knowledge graph based on the acquired information, it is necessary to extract knowledge from the acquired information; specifically, extract the timing information, relationship information, attribute information, numerical information and graph structure information from the audio, video and text information, and construct a modal knowledge graph based on the timing information, relationship information and attribute information of the audio and video text data and the historical live broadcast text data. For example, if the audio and video text data is {Zhang San's age is 24 years old, Zhang San's girlfriend is Wang Wu}, then the graph structure information extracted from the audio and video text data is {Zhang San}, {Wang Wu}, the attribute information is {age}, the numerical information is {24}, the relationship information is {girlfriend}, and the timing information is empty. In practical applications, historical live broadcast data may be stored in a structured database or a graph database. When the historical live broadcast text data is stored in structured data, it is also necessary to extract knowledge from the historical live broadcast text data to obtain the time series information, attribute information, and relationship information in the historical live broadcast text data. Extract user information from the personal live broadcast platform data set, including user personal information such as the user's country, age, language, time zone, etc. to form a user portrait. Then save the user portrait as a node attribute in the user node of the knowledge graph. The extracted objects include all users of the live broadcast platform, including both anchors and users watching the broadcast. Further explanation: All users on the live broadcast platform can watch live broadcasts or start broadcasts themselves, so all user entities are extracted here. Extract the user's behavior of watching anchors, and save the user's historical behavior of watching anchors as a relationship between entities in the graph. The relationship between entities is specifically manifested as directed edges, edge directions, and edge interest weights. The direction of the edge represents the active and passive viewing relationship between users. For example, if a user on the viewing end watches a live broadcast of the anchor, then this direction is from the viewing end user to the anchor. The interest weight of the edge represents the user's preference for different anchors. The degree of preference is mainly based on the historical viewing frequency (number of views) of the anchor by the viewing end user in the statistical log data. The number of times represents the interest weight of the viewing user in the anchor node. The prediction of user viewing anchors based on the knowledge graph refers to the user behavior knowledge graph constructed based on the above steps, adding the temporal characteristics of user behavior, that is, the exponential decay model to update the graph weight, and considering the list of viewing anchors between similar users, so as to greatly improve the accuracy and richness of the prediction results. The update process is used in the recommendation process. In the following formula, m is the previous m days of the user predicted at this time (the 8th day in this embodiment). For example, the previous day m=1, so the weight of the viewing interest of e-1 on the 8th-1=7th day is large, and the previous 5 days m=5, so the weight of e-3 on the 8th-5=3rd day is small, so as to reflect the temporal characteristics of user behavior.Among them, audio and video text data refers to text data obtained by parsing historical audio and video data. For example, live video content is sorted through NLP to obtain live video text data.

[0057] S20, normalize using L2 norm;

[0058] The purpose of L2 norm normalization is to scale the feature vector to unit length. The formula is as follows:

[0059]

[0060] in, Representation vector The L2 norm of It is entity e i The eigenvector of mode m, i represents the index of the vector in the mode, d is the dimension of the eigenvector, It means to sum all elements of the vector (from the 1st to the dth), Representation vector The kth element in the normalized vector can avoid the scale difference between the features of different modalities, making the feature vectors of different modalities comparable when calculating similarity. The sub-modalities in the entity obtain a low-dimensional vector representation in the same representation space. The L2 distance minimization is used in the entity and relationship vector representation training to optimize the training process. In multimodal learning, normalization helps the model better capture the complementarity and correlation between different modalities. Through this formula, we can normalize the feature vectors of different modalities to unit length, preparing for subsequent feature fusion and entity alignment.

[0061] S30, using multiple rounds of cross attention mechanism, in each round, first take a certain modality as the query set, and the other modalities as the key value set in turn, and use the shared matrix Wq, Wk, Wv to parameterize them to get Q, K, V, and then use the Sigmoid activation function to normalize to get the cross attention weight, and finally get the modal output by weighted summation of the attention distribution, the formula is as follows:

[0062] Q=mW q

[0063] K=V=pW k

[0064]

[0065] Among them, d k Represents the dimension of the Q matrix, which is used to scale the dot product result to avoid gradient disappearance or explosion. T represents the transpose of the K matrix, QK Tis the dot product of the query vector and the key vector, σ represents the Sigmoid activation function, m, p∈M represent different modes, Represents entity e i m-modal embedding of .

[0066] The source of the shared matrix is ​​learned through gradient descent or other optimization algorithms during model training. At the beginning of training, these matrices are usually initialized to random values, and then continuously updated through the back-propagation algorithm to minimize the loss function of the model on the training data. Calculating the modal output is divided into two steps: cross-attention weight calculation and modal output. In the cross-attention mechanism, the cross-attention weight refers to the attention score calculated by the dot product between the query vector (Q) and the key vector (K), and then normalized by an activation function (such as Sigmoid). These weights represent the correlation or importance between different modal features. Specifically, the weights are calculated as follows:

[0067]

[0068] These attention weights are then multiplied with the value vector (V) to obtain the weighted sum, which is the modality output. This process can be viewed as the model distributing attention across different modalities to identify and integrate relevant information.

[0069] The Wq matrix is ​​used to map the features of the input modality to a query vector (Q), which represents the target to be found. The Wk matrix is ​​used to map the features of other modalities to a key vector (K), which is used to compare with the query vector to determine the correlation between the features of different modalities. The Wv matrix is ​​used to map the features of other modalities to a value vector (V), which contains the selected feature information and will be weighted and summed according to the matching degree between the query and the key. The above formulas map the features of modality m to the query vector Q through Wq, and map the features of modality p to the key vector K and the value vector V through Wk and Wv. Here, m and p represent different modalities, for example, m can be a time series modality, and p can be a text modality. In this way, the model can learn complementary information between different modalities. The benefit of using a shared matrix is ​​that it allows the model to share parameters between different modalities, which can improve the efficiency and generalization ability of the model while reducing the number of parameters of the model. In addition, the shared matrix can also help the model learn common features across modalities, which is very useful for understanding and fusing multimodal data. For example, after constructing the first knowledge graph of video information and the second knowledge graph of text, the knowledge graphs have relationship feature vectors, attribute feature vectors, time series feature vectors, numerical feature vectors and graph structure feature vectors. The graph topology information is also integrated into the graph structure feature vector based on the GCN feature representation.

[0070] S40. After multiple rounds of cross-attention mechanism processing, each modality m has considered the complementarity with other modalities. These complementarity-enhanced embedding vectors are concatenated to form a complementary embedding matrix, which is expressed as follows:

[0071]

[0072] in, represents the concatenation operation, M is the set of all modes, Represents entity e i m-modal embedding. After attention weighting, the embeddings of different modalities have a certain degree of complementarity, and each modality considers the contribution of other modal features and adds them to its own embedding. Cross-modal correlation modeling is used to capture the connection and shared information between different modalities. The first knowledge graph and the second knowledge graph contain some known aligned quintuples, so that a preliminary connection is also established in the representation vector. The representation of all relationship feature vectors, attribute feature vectors, time series feature vectors, numerical feature vectors and graph structure feature vectors in the first knowledge graph and the second knowledge graph is obtained. After the first knowledge graph and the second knowledge graph are embedded in the same preset vector space to obtain the first vector of all time series and relationships in the first knowledge graph and the second vector of all time series and relationships in the second knowledge graph, the vector product similarity between all vectors in the first vector and the second vector is determined based on the correlation matrix in S50. For example, the first vector contains 10 vectors and the second vector contains 10 vectors, then there will be a total of 20 data, and any two of them multiplied together will obtain 190 vector product similarities.

[0073] Introducing the correlation matrix S∈R when modeling modal complementarity Nm×Nm ,S: represents the correlation matrix, which is used to describe the correlation between different modalities. R represents the set of real numbers, which means that all elements of the matrix S are real numbers. Nm represents the number of modalities. In multimodal learning, Nm can be the total number of different modalities such as vision, text, and audio. The introduction of the correlation matrix can adjust the modal embedding so that each entity considers the complementary relationship between modalities while also comprehensively considering the correlation relationship between modalities. After introducing cross-modal correlation modeling, the above formula is further updated as follows:

[0074]

[0075]

[0076] where · represents the dot product of vectors, Represents entity e i The m-modal embedding, S mp is the correlation score between mode m and mode p, It is entity ei Embedding vector on modality p. In this way, the model not only considers the complementarity between different modalities, for example, the temporal modality may provide the physical characteristics of the entity, while the textual modality provides the semantic description of the entity. The correlation matrix helps the model understand how these modalities complement each other. The correlation between modalities is also considered, so as to understand and fuse multimodal data more comprehensively. For example, in video content, temporal information (video frames) and audio information (dialogue, background music) are closely related. The correlation matrix can help the model identify and utilize these relationships. This fusion method helps to improve the accuracy of entity alignment because it integrates rich information from different modalities. For example, the live broadcast interface shows that the anchor is singing. From the perspective of temporal and graph structure features, this is a singing entertainment live broadcast room, but in fact this is a game anchor who is just singing during a short break in the game, so it will cause ambiguity. But with the addition of game attributes, the entity can be quickly located using game attributes, so the attributes can make up for the problem of missing modal information to a certain extent.

[0077] Corresponding to the above method embodiment, the present application also provides a multi-modal live broadcast data processing device embodiment, Figure 2 FIG. 1 is a schematic diagram showing the structure of a multi-modal live broadcast data processing device provided by an embodiment of the present application. Figure 2 As shown, the device comprises:

[0078] Acquisition module 10: used to acquire multiple modal data of the target live broadcast room, and use the knowledge graph to entity the modal data, specifically represented as entity e i Each sub-modality M∈{r,a,v,n,g}, where r,a,v,n,g represent the feature embedding of relation, attribute, time series, value, and graph structure respectively;

[0079] Normalization module 20: used to normalize the entity e using the L2 norm i Perform normalization;

[0080] Modality alignment module 30: Based on a multi-round cross-attention mechanism, in each round, a certain modality is first used as a query set, and other modalities are used as key value sets in turn, and the pre-trained shared matrices Wq, Wk, and Wv are used to parameterize them to obtain Q, K, and V. Then, the cross-attention weights are normalized using the Sigmoid activation function, and finally the weighted sum of the attention distribution is obtained to obtain the modality output. The formula is as follows:

[0081] Q=mW q

[0082] K=V=pW k

[0083]

[0084] Among them, d k represents the dimension of the Q matrix, K T represents the transpose of the K matrix, σ represents the Sigmoid activation function, m, p∈M represent different sub-modalities, Represents entity e i m-modal embedding of;

[0085] Modal fusion module 40: used to splice the embedded features output by the modalities to form complementary modal data, so as to perform modal fusion on the multi-modal live broadcast data. The formula is as follows:

[0086]

[0087] in, represents the concatenation operation, M is the set of all modes, Represents entity e i m-modal embedding of .

[0088] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document does not limit this.

[0089] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0090] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multi-modal live broadcast data processing method for performing modal fusion on multi-modal live broadcast data, characterized in that: include: Obtain multiple modal data of the target live broadcast room, and use the knowledge graph to materialize the modal data, specifically representing it as an entity Each sub-modality M∈{r,a,v,n,g}, where r,a,v,n,g represent the feature embedding of relation, attribute, time series, value, and graph structure respectively; Use the L2 norm to transform the entity Perform normalization; Based on the multi-round cross-attention mechanism, each round first takes a certain modality as the query set, and the other modalities are taken as the key-value set in turn, and the pre-trained shared matrices Wq, Wk, and Wv are used to parameterize them to obtain Q, K, and V. Then, the Sigmoid activation function is used to normalize them to obtain the cross-attention weights. Finally, the attention distribution is weighted and summed to obtain the modal output. The formula is as follows: ， ， ， in, represents the dimension of the Q matrix, represents the transpose of the K matrix, represents the Sigmoid activation function, m, p∈M represent different sub-modalities, Representing Entities m-modal embedding of; The embedded features output by the modalities are spliced ​​together to form complementary modal data, so as to perform modal fusion on the multi-modal live broadcast data. The formula is as follows: ， in, represents the concatenation operation, M is the set of all modes, Representing Entities m-modal embedding of .

2. A multimodal live broadcast data processing method according to claim 1, characterized in that: The method of obtaining multiple modal data of a target live broadcast room and materializing the modal data using a knowledge graph includes obtaining multiple modal data of a target live broadcast room, respectively extracting relationship information, attribute information, timing information, numerical information and graph structure information in the modalities, and constructing a knowledge graph based on the relationship information, attribute information, timing information, numerical information and graph structure information.

3. A multimodal live broadcast data processing method according to claim 1, characterized in that: The entity is transformed using the L2 norm Normalization is performed, including: using the L2 norm to normalize the entity The eigenvectors in are scaled to unit length using the following formula: ， in, Is Entity is the eigenvector of mode m, where d is the dimension of the eigenvector.

4. A multi-modal live broadcast data processing method according to claim 1, characterized in that: After the embedding features outputted by the modalities are concatenated to form a complementary embedding matrix, the method further includes: Introducing the Correlation Matrix , Nm represents the number of modes, R represents the set of real numbers, so that each entity considers the complementary relationship between modes while also comprehensively considering the correlation between modes. After introducing cross-modal correlation modeling, the above formula is further updated as follows: ， ， in, represents the dot product of vectors, Representing Entities The m-modal embedding of is the correlation score between mode m and mode p, Is Entity Embedding vector on modality p.

5. A multi-modal live broadcast data processing device, characterized in that: include: Acquisition module: used to acquire multiple modal data of the target live broadcast room, and use the knowledge graph to materialize the modal data, specifically represented as entities Each sub-modality M∈{r,a,v,n,g}, where r,a,v,n,g represent the feature embedding of relation, attribute, time series, value, and graph structure respectively; Normalization module: used to transform the entity Perform normalization; Modality alignment module: It is used based on a multi-round cross-attention mechanism. In each round, a certain modality is first used as the query set, and the other modalities are used as the key value set in turn. The pre-trained shared matrices Wq, Wk, and Wv are used to parameterize them to obtain Q, K, and V. Then, the Sigmoid activation function is used to normalize them to obtain the cross-attention weights. Finally, the attention distribution is weighted and summed to obtain the modality output. The formula is as follows: ， ， ， in, represents the dimension of the Q matrix, represents the transpose of the K matrix, represents the Sigmoid activation function, m, p∈M represent different sub-modalities, Representing Entities m-modal embedding of; Modal fusion module: It is used to splice the embedded features output by the modalities to form complementary modal data, so as to perform modal fusion on multi-modal live broadcast data. The formula is as follows: ， in, represents the concatenation operation, M is the set of all modes, Representing Entities m-modal embedding of .

6. A computing device comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that: include: When the processor executes the computer instructions, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium storing computer instructions, characterized in that: include: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images