Multi-modal content aggregation and expression normalization method and device for social big data
By employing multimodal feature fusion and self-supervised aggregation mechanisms, the problem of insufficient adaptability in social content normalization technology is solved, achieving high-quality social content aggregation and expression. This technology is suitable for content redundancy removal and intelligent knowledge aggregation on large-scale social platforms.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT
- Filing Date
- 2025-09-27
- Publication Date
- 2026-07-03
Smart Images
Figure CN121278085B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing technology, and in particular to a method and apparatus for multimodal content aggregation and expression normalization for social big data. Background Technology
[0002] With the widespread adoption of social media, user content has become increasingly diverse, redundant, and expressive. In social media scenarios, traditional content aggregation and deduplication techniques often rely on feature distance (such as cosine similarity) and static clustering, leading to more complex problems and amplifying the limitations of traditional methods. Traditional methods generally suffer from the following issues:
[0003] (1) It is impossible to distinguish between primary and secondary information and information density, and it is easy to retain a large number of advertisements, short sentences and useless information;
[0004] (2) Lack of multimodal (such as text and time) fusion capabilities, and insufficient adaptability to the diversity of content displayed in different scenarios and accounts;
[0005] (3) A single threshold rule is prone to over-grouping or over-merging, making it difficult to automatically and adaptively output high-quality representative content.
[0006] Therefore, there is an urgent need for a new social content normalization technology that can integrate deep semantic features and multimodal information, and has self-supervised discrimination and aggregation capabilities. Summary of the Invention
[0007] A brief overview of the invention is given below to provide a basic understanding of certain aspects of it. It should be understood that this overview is not an exhaustive summary of the invention. It is not intended to identify key or essential parts of the invention, nor is it intended to limit the scope of the invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.
[0008] In view of this, in order to solve the above problems, the present invention proposes a method and apparatus for multimodal content aggregation and expression normalization for social big data, which realizes the normalization of social text content and automatic extraction of representative expressions.
[0009] This invention provides a method for multimodal content aggregation and expression normalization for social big data, including:
[0010] The original social text set is filtered to obtain a set of valid text content, which includes multiple valid text contents.
[0011] Each valid text content is text-encoded to obtain a text vector; and the text vector is concatenated with the time feature vector corresponding to the valid text content to obtain the multimodal fusion vector corresponding to the valid text content.
[0012] Calculate the similarity between any two multimodal fusion vectors within the set of effective text content, and group the set of effective text content into multiple content groups based on the similarity.
[0013] A multi-way query-driven attention Top-k mechanism is used to select key information, and the aggregated result vector is obtained through fusion gating.
[0014] The similarity between each multimodal fusion vector within each content group and the aggregation result vector is calculated to determine the aggregation data of the multimodal fusion vector, and the central content of each content group is determined based on the aggregation data.
[0015] The aggregated data of the central content of each content group is sorted, and a preset number of central contents are selected as the content output for multimodal content aggregation and expression normalization.
[0016] Preferably, content filtering of the original social text set includes: using regular expressions and length discrimination mechanisms to filter the original social text set, determining the set of valid text content, and retaining the timestamp information corresponding to each of the valid text contents in the set of valid text content.
[0017] Preferably, text encoding is performed on each of the valid text contents to obtain a text vector, including:
[0018] A pre-trained deep language model is used to encode each piece of valid text content and extract text vectors.
[0019] Preferably, concatenating the text vector with the time feature vector corresponding to the effective text content to obtain the multimodal fusion vector corresponding to the effective text content includes:
[0020] The timestamp information corresponding to the effective text content is converted according to the proportion of hours and the proportion of days of the week to obtain the time feature vector;
[0021] The text vector and the time feature vector can be directly concatenated to form a high-dimensional multimodal fusion vector, or the text vector and the time feature vector can be nonlinearly mapped to obtain a multimodal fusion vector.
[0022] Preferably, the similarity calculation is performed on any two of the multimodal fusion vectors within the effective text content set, and the effective text content set is grouped into multiple content groups based on the similarity, including:
[0023] Calculate the cosine similarity between any two multimodal fusion vectors within the set of effective text content, and group the multimodal fusion vectors with a similarity greater than a preset similarity threshold into a content group.
[0024] Preferably, the key information is selected using a multi-way query-driven attention Top-k mechanism, and the aggregated result vector is obtained through fusion gating, including:
[0025] Construct a query vector, which includes: a globally trainable vector, a sample mean vector, and a principal direction vector;
[0026] For any of the aforementioned query vectors, an attention Top-k mechanism is used to determine the aggregation vector;
[0027] The aggregation result is processed through a fusion gating mechanism to obtain the weight of each query vector, and the weighted sum of each query vector is used to determine the aggregation result vector.
[0028] Preferably, the method further includes:
[0029] Under unlabeled conditions, the total loss of the effective text content set is obtained, and the attention mechanism and gating parameters are updated through optimizer and backpropagation. When the learning rate drops to a preset loss threshold, the update is stopped.
[0030] Preferably, the total loss for obtaining the set of valid text content includes:
[0031] The total loss is determined using the following formula:
[0032]
[0033] in, The consistency loss for two random multimodal fusion vectors that are not in the same content group. Stability weight parameters The stability loss for two random multimodal fusion vectors that are not in the same content group. Due to attention bandwidth constraints, To incorporate the weighting coefficients of the gated entropy regularization term, This is the regularization term for the fusion weight entropy.
[0034] Secondly, the present invention also provides a device for multimodal content aggregation and expression normalization for social big data, comprising:
[0035] The content filtering module is used to filter the original social text set to obtain a set of valid text content, which includes multiple valid text contents.
[0036] The feature encoding module is used to encode each of the effective text contents to obtain a text vector; and to concatenate the text vector with the time feature vector corresponding to the effective text content to obtain the multimodal fusion vector corresponding to the effective text content.
[0037] The similarity deduplication module is used to calculate the similarity between any two multimodal fusion vectors within the effective text content set, and to group the effective text content set into multiple content groups based on the similarity.
[0038] The gating aggregation module is used to select key information using a multi-way query-driven attention Top-k mechanism and obtain the aggregation result vector through fusion gating;
[0039] The content extraction module is used to calculate the similarity between each multimodal fusion vector in each content group and the aggregation result vector, determine the aggregation data of the multimodal fusion vector, and determine the central content of each content group based on the aggregation data;
[0040] The expression output module is used to sort the aggregated data of the central content of each content group and select a preset number of central contents as the content output for multimodal content aggregation and expression normalization.
[0041] Preferably, the device further includes:
[0042] The loss calculation module is used to obtain the total loss of the effective text content set under unlabeled conditions, update the attention mechanism and gating parameters through optimizer and backpropagation, and stop updating when the learning rate drops to a preset loss threshold.
[0043] This invention relates to a method and apparatus for multimodal content aggregation and expression normalization for social big data. It addresses the problems of existing social content normalization technologies, such as the inability to adaptively distinguish content hierarchy, high information redundancy, and poor expression normalization. This invention combines multimodal feature fusion, cosine redundancy removal, and a gate circuit self-supervised aggregation mechanism to achieve accurate aggregation and hierarchy distinction of multimodal social content under unlabeled conditions. This significantly improves the quality of content normalization and representative expression, making it suitable for large-scale social platform applications involving content redundancy removal, summarization, and intelligent knowledge aggregation.
[0044] These and other advantages of the present invention will become more apparent from the following detailed description of the preferred embodiments of the invention, taken in conjunction with the accompanying drawings. A novel social content normalization technology capable of fusing deep semantic features and multimodal information, and possessing self-supervised discrimination and aggregation capabilities. Attached Figure Description
[0045] The present invention can be better understood by referring to the description given below in conjunction with the accompanying drawings, in which the same or similar reference numerals are used throughout the drawings to denote the same or similar parts. These drawings, together with the following detailed description, are incorporated in and form part of this specification, and are used to further illustrate preferred embodiments of the invention and explain the principles and advantages of the invention. In the drawings:
[0046] Figure 1 This is a flowchart illustrating the multimodal content aggregation and expression normalization method for social big data according to the present invention;
[0047] Figure 2 This is a schematic diagram illustrating the structure of the multimodal content aggregation and expression normalization device for social big data according to the present invention;
[0048] Those skilled in the art will understand that the elements in the accompanying drawings are shown for simplicity and clarity only, and are not necessarily drawn to scale. For example, the dimensions of some elements in the drawings may be enlarged relative to other elements to aid in understanding the embodiments of the invention. Detailed Implementation
[0049] Exemplary embodiments of the invention will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of actual implementations are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the development of any such actual embodiment to achieve the developer's specific goals, such as complying with constraints related to the system and business, and these constraints may vary depending on the implementation. Furthermore, it should be understood that while development work can be very complex and time-consuming, such development work is merely a routine task for those skilled in the art who benefit from this disclosure.
[0050] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the device structure closely related to the solution according to the invention is shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.
[0051] like Figure 1 As shown, this embodiment of the invention provides a method for multimodal content aggregation and expression normalization for social big data, which may include the following steps:
[0052] S100. Filter the original social text set to obtain a set of valid text content, wherein the set of valid text content includes multiple valid text contents;
[0053] S200. Encode each of the effective text contents to obtain a text vector; and concatenate the text vector with the time feature vector corresponding to the effective text content to obtain the multimodal fusion vector corresponding to the effective text content.
[0054] S300. Calculate the similarity between any two multimodal fusion vectors within the set of effective text content, and group the set of effective text content into multiple content groups based on the similarity.
[0055] S400 employs a multi-way query-driven attention Top-k mechanism to select key information, and obtains an aggregated vector through fusion gating;
[0056] S500: Calculate the similarity between each multimodal fusion vector and the aggregated vector within each content group to determine the aggregated data of the multimodal fusion vector, and determine the central content of each content group based on the aggregated data;
[0057] S600: Sort the aggregated data of the central content of each content group, and select a preset number of central contents as the content output for multimodal content aggregation and expression normalization.
[0058] In this embodiment of the invention, the content filtering of the original social text set in step S100 includes: using regular expressions and length discrimination mechanism to filter the original social text set, determining the effective text content set, and retaining the timestamp information corresponding to each effective text content in the effective text content set.
[0059] In this embodiment of the invention, step S100 can use regular expressions and length discrimination to perform initial screening of the original social text set, removing invalid content such as links, @users, tags, advertisements, and short sentences, thereby increasing the proportion of valid content and laying the foundation for subsequent multimodal coding and grouping aggregation. Perform initial filtering:
[0060] If the original social text, after removing URLs, @user tags, hashtags, and whitespace, has a remaining text length of less than 8 characters, it is considered invalid, and valid text is obtained. ;
[0061] Posting time of the original social text The filtered results retain the corresponding timestamps. .
[0062] Obtain a valid text set consisting of valid text content:
[0063]
[0064] Where N is the number of original social texts. This represents the number of valid text contents after filtering.
[0065] In this embodiment of the invention, step S200, which involves text encoding each of the valid text contents to obtain a text vector, includes:
[0066] A pre-trained deep language model is used to encode each piece of valid text content and extract text vectors.
[0067] In this embodiment of the invention, each valid text content filtered through step S100 is... ,
[0068] Text vectors are obtained by encoding text using pre-trained deep language models such as BERT.
[0069]
[0070] in, For the real number field, The dimension is the text vector.
[0071] Then concatenate the time feature vectors The final multimodal fusion vector is obtained as follows:
[0072]
[0073] in, The fusion method refers to the method of fusing the text vector and the time feature vector, which can be either direct concatenation or nonlinear mapping fusion. Specifically, for the direct concatenation method, in this embodiment of the invention, step S200 involves concatenating the text vector with the time feature vector corresponding to the effective text content to obtain the multimodal fusion vector corresponding to the effective text content, including:
[0074] The timestamp information corresponding to the effective text content is converted according to the proportion of hours and the proportion of days of the week to obtain the time feature vector;
[0075] The text vector and the time feature vector can be directly concatenated to form a high-dimensional multimodal fusion vector, or the text vector and the time feature vector can be nonlinearly mapped to obtain a multimodal fusion vector.
[0076] The direct splicing method is represented as:
[0077]
[0078] In this context, "||" represents the vector concatenation operation.
[0079] Optionally, the nonlinear mapping fusion and splicing method is represented as follows:
[0080]
[0081] Where sigma is the activation function. For trainable parameters, These are the splicing parameters. , The sample can be randomly selected or selected based on experience, and converged gradually using subsequent training and total loss constraints.
[0082] In this embodiment of the invention, when obtaining the time feature vector, the timestamp corresponding to the valid text content obtained in step S100 is required. Converting to hourly and weekday proportions yields a time feature vector:
[0083]
[0084] In this embodiment of the invention, step S300, calculating the similarity between any two multimodal fusion vectors within the effective text content set, and grouping the effective text content set into multiple content groups based on the similarity, includes:
[0085] Calculate the cosine similarity between any two multimodal fusion vectors within the set of effective text content, and group the multimodal fusion vectors with a similarity greater than a preset similarity threshold into a content group.
[0086] In this embodiment of the invention, step S300 performs cosine similarity redundancy removal. For multimodal feature matrices... The multimodal feature matrix For the matrix consisting of multimodal fusion vectors corresponding to all valid text content, deduplication is performed using the following steps:
[0087] Normalize each row vector, for any two multimodal fusion vectors and Calculate the cosine similarity of their multimodal fusion vectors:
[0088]
[0089] in, Denotes the L2 norm, if Then it is considered that the two multimodal fusion vectors and If the content is similar, it is grouped into the same content group. The multimodal feature matrix is divided into multiple content groups, and the number of multimodal fusion vectors contained in each content group can be different.
[0090] Preferably, in this embodiment of the invention, in order to speed up the calculation and reduce the computational complexity, the multimodal feature matrix can be divided into multiple sub-blocks, and similarity comparison and clustering can be performed within each block and across blocks.
[0091] In this embodiment of the invention, step S400 uses a multi-path query-driven attention Top-k mechanism to select key information, and obtains an aggregated vector through fusion gating, including:
[0092] Construct a query vector, which includes: a globally trainable vector, a sample mean vector, and a principal direction vector;
[0093] For any of the aforementioned query vectors, an attention Top-k mechanism is used to determine the aggregation result;
[0094] The aggregation result is processed through a fusion gating mechanism to obtain the weight of each query vector, and the aggregate vector is determined by weighted summation of each query vector.
[0095] In this embodiment of the invention, step S400 uses a multi-way query-driven attention Top-k mechanism to select key information and obtains an aggregated vector through fusion gating.
[0096] First, construct the query vector, including: globally trainable vectors. Sample mean vector = PCA-1 principal direction vector .
[0097] Among them, locally trainable vectors With PCA-1 principal direction vector A randomly generated vector, the sample mean vector. This is a vector obtained by averaging the filtered valid text content. They are of equal length.
[0098] For each query vector q ( Perform the following calculations on any one of the options to obtain the attention distribution corresponding to each query vector. and the corresponding aggregate vector :
[0099]
[0100] Where T is the temperature parameter. This refers to the dimension of the key vector in the attention mechanism. , These are the key vector and the value vector, respectively. The values of i are 1, 2, and 3, corresponding to the query vector indices g, m, and p. The values of r are 1, 2, and 3, corresponding to the query vector indices g, m, and p. Softmax represents the normalization operation.
[0101] Then, the aggregated vector obtained from different query vectors The weights of the aggregation result are calculated by using fusion gating. and aggregation result vector :
[0102]
[0103] in, Where Wr, Zr, and br are constants, representing the fusion temperature. This represents the weight of the r-th aggregation result.
[0104] In this embodiment of the invention, the method further includes:
[0105] Under unlabeled conditions, the total loss of the effective text content set is obtained, and the attention mechanism and gating parameters are updated through optimizer and backpropagation. When the learning rate drops to a preset loss threshold, the update is stopped.
[0106] In this embodiment of the invention, the total loss for obtaining the set of valid text content includes:
[0107] The total loss is determined using the following formula:
[0108]
[0109] in, The consistency loss for two random multimodal fusion vectors that are not in the same content group. Stability weight parameters The stability loss for two random multimodal fusion vectors that are not in the same content group. Due to attention bandwidth constraints, To incorporate the weighting coefficients of the gated entropy regularization term, This is the regularization term for the fusion weight entropy.
[0110] Under unlabeled conditions, robust aggregate representations are learned through consistency and stability constraints, and the attention distribution bandwidth and fusion weight entropy are regulated:
[0111] For two random multimodal fusion vectors that are not in the same content group , Consistency loss ;
[0112] Stable loss:
[0113] Attention bandwidth constraint: on the attention distribution entropy ;
[0114] in, Attention distribution for each query vector;
[0115] Applying a banded penalty to obtain attention bandwidth constraints:
[0116] ;
[0117] Fusion weight entropy regularization: for both fusion and weight entropy Weighting;
[0118] Total loss: In this embodiment, it is possible to take =0.1, =2.2, =3.6, =0.02, =0.02.
[0119] In this embodiment of the invention, step S500 involves calculating the similarity between each multimodal fusion vector within each content group and the aggregated result vector to determine the aggregated data of the multimodal fusion vector, and determining the central content of each content group based on the aggregated data; including:
[0120] For each content group, calculate the cosine similarity between each of the multimodal fusion vectors within the group and the final aggregated vector:
[0121]
[0122] The effective text content corresponding to the multimodal fusion vector with the highest score is selected as the core content of the group.
[0123] In this embodiment of the invention, step S600 sorts the aggregated data of the central content of each content group and selects a preset number of central contents as the content output for multimodal content aggregation and expression normalization, including:
[0124] The core content of all content groups is summarized, and the top K representative content items with the highest scores are selected as the output for multimodal content aggregation and expression normalization, taking into account group size, number of similar merges, and time indicators.
[0125] like Figure 2 As shown, this invention also provides a multimodal content aggregation and expression normalization device for social big data, comprising:
[0126] The content filtering module is used to filter the original social text set to obtain a set of valid text content, which includes multiple valid text contents.
[0127] The feature encoding module is used to encode each of the effective text contents to obtain a text vector; and to concatenate the text vector with the time feature vector corresponding to the effective text content to obtain the multimodal fusion vector corresponding to the effective text content.
[0128] The similarity deduplication module is used to calculate the similarity between any two multimodal fusion vectors within the effective text content set, and to group the effective text content set into multiple content groups based on the similarity.
[0129] The gating aggregation module is used to select key information using a multi-way query-driven attention Top-k mechanism and obtain an aggregated vector through fusion gating;
[0130] The content extraction module is used to calculate the similarity between each multimodal fusion vector and the aggregated vector within each content group, determine the aggregated data of the multimodal fusion vector, and determine the central content of each content group based on the aggregated data.
[0131] The expression output module is used to sort the aggregated data of the central content of each content group and select a preset number of central contents as the content output for multimodal content aggregation and expression normalization.
[0132] In this embodiment of the invention, the device further includes:
[0133] The loss calculation module is used to obtain the total loss of the effective text content set under unlabeled conditions, update the attention mechanism and gating parameters through optimizer and backpropagation, and stop updating when the learning rate drops to a preset loss threshold.
[0134] Example
[0135] This embodiment describes the process of using the aforementioned multimodal content aggregation and expression normalization device and method for social big data to normalize social text content and automatically extract representative expressions:
[0136] Taking the aggregation of content from a social media platform account as an example, 5000 original text entries were retrieved from this account. The following process was used to complete content normalization and output representative content:
[0137] S1 Original Content Filtering
[0138] Weak text filtering was performed on each of the 5000 original text entries: URLs, topic tags, and @ mentions were removed, and whitespace was compressed. If the net text length was less than 8 characters, it was considered invalid and discarded; timestamps were retained. After this step, 3500 valid text entries remained. .
[0139] S2 Multimodal Feature Encoding
[0140] For each valid sample Perform joint text and time encoding:
[0141] Use pre-trained BERT to obtain text CLS vectors ;
[0142] Convert timestamps into 2D periodic features ;
[0143] splicing multimodal vectors The characteristic matrix is obtained. .
[0144] S3 Cosine Similarity Deduplication
[0145] Perform L2 normalization on the samples and calculate pairwise cosine similarity. ;
[0146] When the first one appears, only the one that appears earlier is retained, and the other one is merged into its "similar group".
[0147] The results from each block were aggregated, and the deredundant text was further grouped into 100 content groups (each group contains several similar texts).
[0148] S4 Attention Gate Convergence
[0149] For the deredundant sample set, a global aggregated representation is obtained by using multi-query Top-k attention and fusion gating:
[0150] 1. Construct three types of query vectors: globally trainable vectors Mean vector PCA-1 principal direction vector .
[0151] 2. Based on multi-head attention cores (Top-k selection), select the most relevant samples from each content group E', and perform softmax weighting according to temperature T to obtain the aggregate vector for each path. With attention distribution .
[0152] 3. Through integrated gating Adaptive weighting is applied to different paths to obtain the final aggregated result vector. .
[0153] During training, temperature annealing (e.g., T: 0.95 → 0.80) is used. The Top-k training threshold and inference threshold are set to 10 and 64, respectively, and the number of multi-heads and the dimension are divisible by integers (2 in this embodiment).
[0154] S5 Self-Supervised Training Objectives
[0155] Under unlabeled conditions, robust aggregate representations are learned through consistency and stability constraints, and the attention distribution bandwidth and fusion weight entropy are regulated:
[0156] For random subsets get , , ;
[0157] Stable loss:
[0158] Attention bandwidth constraint: on the attention distribution entropy Applying strip punishment
[0159] ;
[0160] Fusion weight entropy regularization: for both fusion and weight entropy Weighting;
[0161] Total loss: In this embodiment, it is possible to take =0.1, =2.2, =3.6, =0.02, =0.02.
[0162] S6 Core Content Extraction and Top-K Output
[0163] Within each group, to be with The element with the highest cosine similarity is considered the "central content" of that group;
[0164] Perform a comprehensive sorting of all central contents: Calculation
[0165] Similarity score (Linear mapping based on cosine similarity)
[0166] Grouping and merging size (Perform min-max normalization or log-normalization on the "number of similar merges")
[0167] Time freshness score ( It is the number of days since today. 7 can be selected); and press Calculate the total score (in this example, take...) =0.5, )
[0168] Based on the total score S, select the Top-10 in descending order as the final representative content output.
[0169] In this embodiment, the final Top-10 can cover the central expression of each content group, significantly reducing duplication and redundancy, improving the expression consistency and information density of account content, and facilitating subsequent applications such as summarization, monitoring and knowledge aggregation.
[0170] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0171] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
[0172] Although the invention has been described with reference to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of interpreting or limiting the subject matter of the invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the invention is illustrative and not restrictive, and the scope of the invention is defined by the appended claims.
Claims
1. A method for multimodal content aggregation and expression normalization for social big data, characterized in that, include: The original social text set is filtered to obtain a set of valid text content, which includes multiple valid text contents. Each valid text content is text-encoded to obtain a text vector; and the text vector is concatenated with the time feature vector corresponding to the valid text content to obtain the multimodal fusion vector corresponding to the valid text content. Calculate the similarity between any two multimodal fusion vectors within the set of effective text content, and group the set of effective text content into multiple content groups based on the similarity. A multi-way query-driven attention Top-k mechanism is used to select key information, and the aggregated result vector is obtained through fusion gating. The similarity between each multimodal fusion vector within each content group and the aggregation result vector is calculated to determine the aggregation data of the multimodal fusion vector, and the central content of each content group is determined based on the aggregation data. The aggregated data of the core content of each content group is sorted, and a preset number of core contents are selected as the content output for multimodal content aggregation and expression normalization. Construct the query vector q, including: globally trainable vectors Sample mean vector PCA-1 principal direction vector ; For each query vector q, the attention distribution corresponding to each query vector is calculated as follows. and the corresponding aggregate vector : ； Where T is the temperature parameter. Let be the dimension of the key vector in the attention mechanism. , These are the key vector and the value vector, respectively. The values of i are 1, 2, and 3, corresponding to the query vector indices g, m, and p. The values of r are 1, 2, and 3, corresponding to the query vector indices g, m, and p. softmax represents the normalization operation. Aggregate vectors obtained from different query vectors The weights of the aggregation result are calculated by using fusion gating. and aggregation result vector : ； in, Where Wr, Zr, and br are constants, representing the fusion temperature. The weights represent the aggregation results of the r-th path; For each content group, calculate the cosine similarity between each of the multimodal fusion vectors within the group and the final aggregated vector: ； The effective text content corresponding to the multimodal fusion vector with the highest score is selected as the core content of that group. This is a multimodal fusion vector.
2. The method according to claim 1, characterized in that, Content filtering of the original social text set includes: using regular expressions and length discrimination mechanisms to filter the original social text set, determining the set of valid text content, and retaining the timestamp information corresponding to each of the valid text contents in the set of valid text content.
3. The method according to claim 1, characterized in that, Each valid text content is text-encoded to obtain a text vector, including: A pre-trained deep language model is used to encode each piece of valid text content and extract text vectors.
4. The method according to claim 3, characterized in that, The step of concatenating the text vector with the time feature vector corresponding to the effective text content to obtain the multimodal fusion vector corresponding to the effective text content includes: The timestamp information corresponding to the effective text content is converted according to the proportion of hours and the proportion of days of the week to obtain the time feature vector; The text vector and the time feature vector can be directly concatenated to form a high-dimensional multimodal fusion vector, or the text vector and the time feature vector can be nonlinearly mapped to obtain the multimodal fusion vector.
5. The method according to claim 1, characterized in that, Calculate the similarity between any two multimodal fusion vectors within the set of valid text content, and group the set of valid text content into multiple content groups based on the similarity, including: Calculate the cosine similarity between any two multimodal fusion vectors within the set of effective text content, and group the multimodal fusion vectors with a similarity greater than a preset similarity threshold into a content group.
6. The method according to claim 1, characterized in that, The method further includes: Under unlabeled conditions, the total loss of the effective text content set is obtained, and the attention mechanism and gating parameters are updated through optimizer and backpropagation. When the learning rate drops to a preset loss threshold, the update is stopped.
7. The method according to claim 6, characterized in that, The total loss for obtaining the set of valid text content includes: The total loss is determined using the following formula: ； in, The consistency loss for two random multimodal fusion vectors that are not in the same content group. Stability weight parameters The stability loss for two random multimodal fusion vectors that are not in the same content group. Due to attention bandwidth constraints, To incorporate the weighting coefficients of the gated entropy regularization term, This is the regularization term for the fusion weight entropy.
8. A device for multimodal content aggregation and expression normalization for social big data, characterized in that: include: The content filtering module is used to filter the original social text set to obtain a set of valid text content, which includes multiple valid text contents. The feature encoding module is used to encode each of the effective text contents to obtain a text vector; and to concatenate the text vector with the time feature vector corresponding to the effective text content to obtain the multimodal fusion vector corresponding to the effective text content. The similarity deduplication module is used to calculate the similarity between any two multimodal fusion vectors within the effective text content set, and to group the effective text content set into multiple content groups based on the similarity. The gating aggregation module is used to select key information using a multi-way query-driven attention Top-k mechanism and obtain the aggregation result vector through fusion gating; The content extraction module is used to calculate the similarity between each multimodal fusion vector in each content group and the aggregation result vector, determine the aggregation data of the multimodal fusion vector, and determine the central content of each content group based on the aggregation data; The expression output module is used to sort the aggregated data of the central content of each content group and select a preset number of central contents as the content output of multimodal content aggregation and expression normalization. Construct the query vector q, including: globally trainable vectors Sample mean vector PCA-1 principal direction vector ; For each query vector q, the attention distribution corresponding to each query vector is calculated as follows. and the corresponding aggregate vector : ； Where T is the temperature parameter. Let be the dimension of the key vector in the attention mechanism. , These are the key vector and the value vector, respectively. The values of i are 1, 2, and 3, corresponding to the query vector indices g, m, and p. The values of r are 1, 2, and 3, corresponding to the query vector indices g, m, and p. softmax represents the normalization operation. Aggregate vectors obtained from different query vectors The weights of the aggregation result are calculated by using fusion gating. and aggregation result vector : ； in, Where Wr, Zr, and br are constants, representing the fusion temperature. The weights represent the aggregation results of the r-th path; For each content group, calculate the cosine similarity between each of the multimodal fusion vectors within the group and the final aggregated vector: ； The effective text content corresponding to the multimodal fusion vector with the highest score is selected as the core content of that group. This is a multimodal fusion vector.
9. The apparatus according to claim 8, characterized in that, Also includes: The loss calculation module is used to obtain the total loss of the effective text content set under unlabeled conditions, update the attention mechanism and gating parameters through optimizer and backpropagation, and stop updating when the learning rate drops to a preset loss threshold.