Internet information analysis method based on multi-modal data fusion

Abstract

The invention discloses an Internet information analysis method based on multi-modal data fusion, comprising the following steps: S1, collecting and aligning multi-modal original data, and constructing a multi-modal input sample set; s2, inputting a U-Net fusion network, extracting and fusing modal features, and outputting a fusion semantic representation tensor; s3, constructing a stable diffusion generation model, inputting a fused semantic representation tensor, and executing denoising reconstruction; s4, verifying semantic consistency, if the matching score is insufficient, adjusting the attention parameter, and returning to execute the steps S2-S4; and S5, fusing the structured information, analyzing the generated content, and outputting a structured analysis report. According to the method, deep semantic understanding and structured analysis of Internet information are realized by fusing a multi-modal data modeling and generating technology.

Description

Technical Field

[0001] The present invention relates to the field of multimodal artificial intelligence technology, and in particular to an Internet information analysis method based on multimodal data fusion. Background Art

[0002] With the extreme richness and diversity of internet content, information analysis technology is moving from a single modality to a multimodal, deep semantic intelligent fusion stage. Most existing internet information analysis systems rely on feature extraction and rule matching for single or simply spliced ​​modal data such as text, images, and audio. Their core relies on traditional deep learning models such as convolutional neural networks, recurrent neural networks, and BERT to model single-type data. While these methods have achieved certain results in specific tasks, they struggle to adapt to the complexity and diversity of information structures on current internet platforms. This is especially true when it comes to cross-modal data fusion, deep semantic expression, and multi-scenario event reasoning. Traditional models are prone to information fragmentation, modality mismatch, and insufficient analytical depth, making it impossible to achieve efficient and robust full-process internet information understanding.

[0003] In recent years, with the development of multimodal deep learning and generative models, researchers have attempted to improve the collaborative modeling capabilities of multimodal data through models such as Transformer, variational autoencoder, and adversarial generative network. However, existing multimodal fusion networks are mostly based on simple splicing or weighting, ignoring the complex interactive relationships between modalities at the spatial, structural, and semantic levels, resulting in limited fusion feature expression capabilities. At the same time, although mainstream generative models such as GAN and VAE can achieve content generation, they lack effective global semantic constraints and dynamic feedback tuning mechanisms in the complex Internet context, which can easily cause the generated content to deviate from the real scene. In addition, existing event detection, structured analysis, and visualization solutions are usually designed in a decentralized manner, lacking the ability to adaptively verify the consistency of multimodal generated content, automatically generate analysis labels, and optimize closed-loop feedback, and cannot meet the needs of deep understanding and intelligent decision-making for the entire process of complex Internet information.

[0004] Therefore, how to provide an Internet information analysis method based on multimodal data fusion is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] One purpose of the present invention is to propose an Internet information analysis method based on multimodal data fusion. The present invention makes full use of multimodal deep learning, U-Net structural level feature fusion, stable diffusion generation mechanism and semantic consistency verification feedback method, and describes in detail the complete process of synchronous alignment of multi-source heterogeneous data on the Internet platform, multimodal feature extraction and fusion, dynamic optimization of generated content, construction of structured event units and intelligent analysis. Compared with the existing technology, the present invention has the advantages of strong multimodal collaboration ability, high semantic consistency of generated content, high degree of structuring of analysis results, strong ability to adapt to complex Internet scenarios and perfect feedback self-optimization mechanism, which can effectively improve the level of intelligent analysis, automatic understanding and decision support of Internet information.

[0006] The Internet information analysis method based on multimodal data fusion according to an embodiment of the present invention includes the following steps:

[0007] S1. Collect multimodal raw data from the Internet platform, synchronize and align the multimodal raw data, and construct a multimodal input sample set;

[0008] S2. Input the multimodal input sample set into a U-Net fusion network. The U-Net fusion network includes an image encoder, a text encoder, an audio encoder, and a structured modality encoder. The U-Net fusion network outputs feature representation tensors of each modality respectively, fuses the feature representation tensors of each modality to form a multimodal fusion feature map, establishes skip connections between layers of the U-Net encoding path, and outputs a fused semantic representation tensor.

[0009] S3. Construct a stable diffusion generation model, take the fused semantic representation tensor as the conditional control vector as input, perform multi-step denoising and reconstruction through the reverse diffusion process, and generate multimodal joint output content;

[0010] S4, verifying the semantic consistency of the multimodal joint output content and the multimodal input sample set, calculating the semantic matching score, and if the semantic matching score is lower than the set threshold, adjusting the attention mechanism parameters in the U-Net fusion network, and returning to step S2 to step S4 until the semantic matching score meets the set conditions;

[0011] S5. Based on the semantic consistency verification results, combined with the structured Internet information associated with the multimodal input sample set, a comprehensive analysis of the multimodal joint output content is performed, event expression units and analysis labels are constructed, and a structured analysis report of the Internet information is generated and output to the visualization terminal.

[0012] Optionally, the multimodal original data specifically includes image data, text data, audio data and web page structure data.

[0013] Optionally, the synchronous alignment of multimodal raw data refers to size normalization of image data, word segmentation and embedding vector conversion of text data, extraction of Mel-spectrogram features of audio data, label hierarchical analysis of web page structure data, and synchronous alignment of all modal data according to timestamps.

[0014] Optionally, the S2 specifically includes:

[0015] S21. Construct a U-Net fusion network, where the U-Net fusion network includes an image encoder, a text encoder, an audio encoder, and a structured modality encoder;

[0016] S22. Input the multimodal input sample set into the image encoder, text encoder, audio encoder, and structured modality encoder in the U-Net fusion network respectively. The image encoder adopts a multi-layer convolutional neural network structure, the text encoder adopts a Transformer network structure, the audio encoder adopts a hybrid structure of convolutional and recurrent neural networks, and the structured modality encoder adopts a joint structure of multi-layer perceptron and graph neural network.

[0017] S23, the image encoder extracts features from the image data input to obtain the image feature representation tensor T I ; The text encoder extracts features from the text data input and obtains the text feature representation tensor T T ; The audio encoder extracts features from the audio data input and obtains the audio feature representation tensor T A ; The structured modal encoder extracts features from the web page structure data input and obtains the structured modal feature representation tensor T S ;

[0018] S24. Setting a structurally shared sub-encoder module in the intermediate layer of each encoding path of the U-Net fusion network to apply a shared encoding operation to the intermediate feature tensors of each modality, where the intermediate feature tensors of each modality refer to the output feature tensors of each modality encoder in the non-final output stage and each intermediate layer in the U-Net fusion network;

[0019] S25. Introduce a cross-modal jump connection mechanism in the U-Net fusion network, perform cross-modal jump connections between the intermediate feature tensors of the text encoder and the same-layer feature tensors of the image encoder, and perform cross-modal jump connections between the intermediate feature tensors of the audio encoder and the structured modality encoder, to achieve alignment and information collaboration between the intermediate layer features of different modalities;

[0020] S26. Establish the downsampling structure of each encoding path in the U-Net fusion network, and represent the image feature tensor T respectively. I , text feature representation tensor T T , audio feature representation tensor TA , structured modal feature representation tensor T S Perform downsampling processing;

[0021] S27. Establish a corresponding upsampling structure in the U-Net fusion network, and connect the feature tensors between each layer of the encoding path to the corresponding positions of the decoding path through the skip connection mechanism to ensure the synchronous transmission of context and fine-grained information;

[0022] S28, performing feature fusion on the downsampled feature tensors of each modality processed by the structure-sharing sub-encoder module and the cross-modal jump connection mechanism to form a multimodal fusion feature map;

[0023] S29. In the decoding path of the U-Net fusion network, the spatial dimension is restored layer by layer and the skip connection features are fused, and finally the fused semantic representation tensor is output.

[0024] Optionally, the S3 specifically includes:

[0025] S31. Construct a stable diffusion generation model, wherein the stable diffusion generation model adopts a U-Net backbone structure and is provided with a multimodal condition control mechanism, a multi-level semantic condition adaptive injection module, a cross-modal attention guidance module, and an adaptive diffusion step size and dynamic convergence scheduling module;

[0026] S32, set the total number of steps in the diffusion process to N, and the diffusion state of each step to x t , where t=N,N-1,…,1,0;

[0027] S33, the initial diffusion state x N Set to a noise tensor that obeys a Gaussian distribution, i.e. Where I is the unit covariance matrix;

[0028] S34. Through the multimodal conditional control mechanism, the output fused semantic representation tensor is used as the multimodal conditional control vector, which is input into the stable diffusion generation model and serves as a guiding signal for generating content in the entire diffusion reverse denoising process;

[0029] S35. The multi-level semantic condition adaptive injection module hierarchically injects different semantic feature components of the fused semantic representation tensor into different layers of the U-Net backbone structure of the stable diffusion generative model. It injects spatial feature information into the shallow layers of the encoder and high-order semantic feature information into the deep layers. Through adaptive weight control, the adaptive injection of multi-level semantic conditions is completed.

[0030] S36, through the cross-modal attention guidance module, in each step of the reverse denoising process, the current noise state x is used tTogether with the multimodal conditional control vector, multi-head cross-modal attention is performed on the spatial and semantic information of the fused semantic representation tensor and the current generated tensor;

[0031] S37, in each step of the reverse denoising process, through the multimodal conditional control mechanism and the cross-modal attention guidance module, based on the current generated state x t , fuse the semantic representation tensor, dynamically calculated adaptive weights of each modality, and adaptive gating units, fuse the feature flows of each modality, and use the semantic consistency deviation between the tensor generated at the previous moment and the fused semantic representation tensor at the current moment as the residual signal to participate in denoising prediction, and output the denoising prediction tensor

[0032]

[0033] Among them, m represents each mode category, λ m is the dynamically calculated adaptive weight of the mth modality, is the denoising prediction under the mth mode condition, R t is the semantic residual feedback term, γ t is the gating weight of the residual signal, α t is the noise scaling factor at step t, Represent tensors for fusion semantics;

[0034] S38. In each step of the reverse denoising process, the multi-level semantic condition adaptive injection module dynamically adjusts the injection weight of the multimodal condition control vector according to the semantic consistency between the fused semantic representation tensor and the current generation state, thereby optimizing the contribution ratio of each modality to the generation result;

[0035] S39. In each step of the reverse denoising process, the adaptive diffusion step size and dynamic convergence scheduling module, through the semantic discrimination method, calculates the semantic matching score between the current generation state and the multimodal condition control vector in real time to judge the quality of the current generation state;

[0036] S310: During the reverse denoising iteration, when the adaptive diffusion step and dynamic convergence scheduling module determines that the semantic matching score is higher than the set threshold, the current generation state is output, and the adaptive diffusion step and dynamic convergence scheduling are completed; if the semantic matching score does not reach the set threshold, steps S36 to S39 are continued to continue the reverse denoising process;

[0037] S311, iteratively executing steps S36 to S310 in sequence, gradually reconstructing the noise tensor into the final multimodal joint output content generation tensor x0;

[0038] S312. The resulting generated tensor x0 is restored to multimodal joint output content through a decoding process, and the multimodal joint output content is output.

[0039] Optionally, the S4 specifically includes:

[0040] S41, establishing a correspondence between the generated multimodal joint output content and the multimodal input sample set, using a unified sample identifier to form a one-to-one corresponding input and output sample pair;

[0041] S42. For each pair of multimodal joint output content and the corresponding multimodal input sample set, extract the fusion semantic representation tensor and As input features for semantic consistency verification;

[0042] S43. For each pair of fused semantic representation tensors and Calculate the semantic matching score S match :

[0043]

[0044] in, represents the vector inner product of the two, and They represent the L2 norm of the two respectively, the first term is the cosine similarity, the second term is the square of the Euclidean distance, β is the balance factor, S match Reflects the comprehensive consistency of the fused semantic representation tensor in terms of direction and distribution;

[0045] S44, the semantic matching score S of each pair of input and output samples match Compare with the preset threshold δ, if S match <δ, the semantic consistency of the generated content is judged to be insufficient, and the inconsistent sample index is recorded;

[0046] S45. For samples whose semantic consistency does not meet the requirements, adjust the attention mechanism parameter α in the U-Net fusion network, use the feedback optimization method to correct the attention allocation weight, and obtain a new attention parameter set α′;

[0047] S46. Based on the modified attention mechanism parameter set α′, return to step S2 to step S4 and repeat the above semantic consistency verification and feedback optimization process until the semantic matching scores of all samples reach the preset threshold;

[0048] S47. When the semantic matching scores of all input and output samples are higher than the preset threshold δ, it is confirmed that the current combination of the U-Net fusion network and the stable diffusion generation model has achieved full-process semantic consistency between the multimodal output content and the input samples, and the process enters the next step of analysis.

[0049] Optionally, the S5 specifically includes:

[0050] S51. Based on the obtained semantic consistency verification results, mark the matching relationship between each multimodal joint output content and the multimodal input sample set, and screen out content pairs whose semantic consistency meets a set threshold;

[0051] S52. For multimodal joint output content that meets the semantic consistency requirements, combine it with structured Internet information associated with the multimodal input sample set, and use data fusion methods to achieve joint analysis of different information sources to form a multimodal analysis input unit;

[0052] S53, the fusion semantic representation tensor of the multimodal joint output content and the feature representation of structured Internet information S I Perform feature cascade;

[0053] S54, based on the fused multimodal analysis feature tensor T FA ,Through the event detection algorithm, the Internet event expression unit E is extracted, and each event expression unit is jointly represented by relevant text, image, audio and structured information;

[0054] S55. For each event expression unit, a label generation algorithm is used to generate a corresponding analysis label, where the analysis label includes structured analysis information such as event type, key information points, sentiment attributes, or source identification;

[0055] S56. Summarize and construct a structured analysis report of Internet information based on all event expression units and their analysis tags. The structured analysis report includes multimodal semantic descriptions, event links, statistical features, and visual analysis elements.

[0056] S57. Output the structured analysis report to the visualization terminal to realize the result display and decision support of Internet multimodal information.

[0057] The beneficial effects of the present invention are:

[0058] The present invention realizes deep collaborative modeling and intelligent analysis of multi-source heterogeneous data on the Internet platform by integrating the multimodal U-Net network structure and the stable diffusion generation model, significantly improving the information analysis system's ability to understand complex scenarios and diverse content. Compared with the existing analysis methods that only rely on a single modality or simple fusion, the present invention can give full play to the complementary advantages of multimodal features at the spatial, semantic and structural levels, and effectively enhances the expressive power of fused features and the semantic consistency of generated content through multi-level conditional adaptive injection and cross-modal attention mechanism. At the same time, the present invention introduces a semantic consistency check and feedback tuning link, which can perform dynamic consistency evaluation on the multimodal joint output content and the original input sample, achieve accurate alignment of the generated results with the actual semantic requirements, and greatly improve the credibility and robustness of the analysis results.

[0059] Furthermore, the present invention combines structured internet information to construct event units and automatically generate tags for multimodal content, and outputs structured analysis reports, thus promoting the automation and intelligentization of internet information analysis. Overall, the present invention not only achieves efficient collaboration and in-depth understanding of multimodal data, but also builds a closed-loop optimization system through generative models and feedback mechanisms. This effectively addresses the bottlenecks of existing technologies in terms of fusion depth, generation accuracy, and structured output, significantly improving the efficiency of the entire process of analyzing complex internet information and the ability to support intelligent decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0061] Figure 1 This is a flow chart of the Internet information analysis method based on multimodal data fusion proposed by the present invention;

[0062] Figure 2 This is a schematic diagram of the structure of the stable diffusion generation model of the Internet information analysis method based on multimodal data fusion proposed in the present invention. DETAILED DESCRIPTION

[0063] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0064] refer to Figure 1 and Figure 2 , an Internet information analysis method based on multimodal data fusion, comprises the following steps:

[0065] S1. Collect multimodal raw data from the Internet platform, synchronize and align the multimodal raw data, and construct a multimodal input sample set;

[0066] S2. Input the multimodal input sample set into a U-Net fusion network. The U-Net fusion network includes an image encoder, a text encoder, an audio encoder, and a structured modality encoder. The U-Net fusion network outputs feature representation tensors of each modality respectively, fuses the feature representation tensors of each modality to form a multimodal fusion feature map, establishes skip connections between layers of the U-Net encoding path, and outputs a fused semantic representation tensor.

[0067] S3. Construct a stable diffusion generation model, take the fused semantic representation tensor as the conditional control vector as input, perform multi-step denoising and reconstruction through the reverse diffusion process, and generate multimodal joint output content;

[0068] S4, verifying the semantic consistency of the multimodal joint output content and the multimodal input sample set, calculating the semantic matching score, and if the semantic matching score is lower than the set threshold, adjusting the attention mechanism parameters in the U-Net fusion network, and returning to step S2 to step S4 until the semantic matching score meets the set conditions;

[0069] S5. Based on the semantic consistency verification results, combined with the structured Internet information associated with the multimodal input sample set, a comprehensive analysis of the multimodal joint output content is performed, event expression units and analysis labels are constructed, and a structured analysis report of the Internet information is generated and output to the visualization terminal.

[0070] The present invention realizes the efficient collection, synchronous alignment and deep fusion of multimodal raw data of Internet platforms, giving full play to the complementary advantages of images, text, audio and structured information in different dimensions. The U-Net fusion network is used to extract and fuse multimodal features in a jump connection manner, which can fully retain spatial details and semantic information, and effectively improve the integrity and diversity of feature expression. Combined with the stable diffusion generation model, the fused semantic representation tensor is used as the conditional control vector, and after multi-step reverse denoising and reconstruction, it is ensured that the generated content is highly consistent with the multimodal input. The introduction of semantic consistency verification and feedback optimization mechanism enables the model to dynamically adjust the attention parameters and continuously improve the semantic fit and robustness of the output content. Finally, by integrating structured Internet information and multimodal joint output, event expression units and analysis tags are automatically constructed, and structured analysis reports are generated, which improves the intelligence and automation level of Internet information processing. The present invention not only improves the accuracy and depth of multimodal data processing, but also achieves high-quality structured output, providing strong support for the understanding and decision-making of complex Internet information.

[0071] In this embodiment, the multimodal original data specifically includes image data, text data, audio data and web page structure data.

[0072] In this embodiment, the synchronous alignment of multimodal raw data refers to size normalization of image data, word segmentation and embedding vector conversion of text data, extraction of Mel-spectrogram features of audio data, label hierarchical analysis of web page structure data, and synchronous alignment of all modal data according to timestamps.

[0073] In this embodiment, S2 specifically includes:

[0074] S21. Construct a U-Net fusion network, where the U-Net fusion network includes an image encoder, a text encoder, an audio encoder, and a structured modality encoder;

[0075] S22. Input the multimodal input sample set into the image encoder, text encoder, audio encoder, and structured modality encoder in the U-Net fusion network respectively. The image encoder adopts a multi-layer convolutional neural network structure, the text encoder adopts a Transformer network structure, the audio encoder adopts a hybrid structure of convolutional and recurrent neural networks, and the structured modality encoder adopts a joint structure of multi-layer perceptron and graph neural network.

[0076] S23, the image encoder extracts features from the image data input to obtain the image feature representation tensor T I ; The text encoder extracts features from the text data input and obtains the text feature representation tensor T T ; The audio encoder extracts features from the audio data input and obtains the audio feature representation tensor T A ; The structured modal encoder extracts features from the web page structure data input and obtains the structured modal feature representation tensor T S ;

[0077] S24. Setting a structurally shared sub-encoder module in the intermediate layer of each encoding path of the U-Net fusion network to apply a shared encoding operation to the intermediate feature tensors of each modality, where the intermediate feature tensors of each modality refer to the output feature tensors of each modality encoder in the non-final output stage and each intermediate layer in the U-Net fusion network;

[0078] S25. Introduce a cross-modal jump connection mechanism in the U-Net fusion network, perform cross-modal jump connections between the intermediate feature tensors of the text encoder and the same-layer feature tensors of the image encoder, and perform cross-modal jump connections between the intermediate feature tensors of the audio encoder and the structured modality encoder, to achieve alignment and information collaboration between the intermediate layer features of different modalities;

[0079] S26. Establish the downsampling structure of each encoding path in the U-Net fusion network, and represent the image feature tensor T respectively. I , text feature representation tensor T T , audio feature representation tensor T A , structured modal feature representation tensor T S Perform downsampling processing;

[0080] S27. Establish a corresponding upsampling structure in the U-Net fusion network, and connect the feature tensors between each layer of the encoding path to the corresponding positions of the decoding path through the skip connection mechanism to ensure the synchronous transmission of context and fine-grained information;

[0081] S28, performing feature fusion on the downsampled feature tensors of each modality processed by the structure-sharing sub-encoder module and the cross-modal jump connection mechanism to form a multimodal fusion feature map;

[0082] S29. In the decoding path of the U-Net fusion network, the spatial dimension is restored layer by layer and the skip connection features are fused, and finally the fused semantic representation tensor is output.

[0083] The present invention realizes deep collaborative modeling and efficient feature fusion of Internet multimodal data. Through the collaborative design of image encoder, text encoder, audio encoder and structured modality encoder, the feature advantages of different data types are fully exploited. The introduction of structure-sharing sub-encoder modules in the middle layer not only improves the semantic consistency of the intermediate features of each modality, but also effectively reduces parameter redundancy, improves the training efficiency and generalization ability of the model. The cross-modal jump connection mechanism realizes the precise alignment and complementarity of the intermediate layer features between different modalities, significantly enhancing the feature collaborative expression ability of the fusion network. Combining the downsampling and upsampling structure and jump connection, it ensures the full recovery of spatial details and global semantic information in the decoding stage. Finally, through the deep fusion and progressive decoding of multimodal features, a high-quality, semantically rich and structurally unified fusion semantic representation tensor is generated. Overall, the present invention significantly improves the integrity of multimodal feature expression, the accuracy of fusion and the intelligent level of analysis, providing a solid feature foundation and structural guarantee for subsequent generative modeling and in-depth analysis of Internet information.

[0084] In this embodiment, S3 specifically includes:

[0085] S31. Construct a stable diffusion generation model, wherein the stable diffusion generation model adopts a U-Net backbone structure and is provided with a multimodal condition control mechanism, a multi-level semantic condition adaptive injection module, a cross-modal attention guidance module, and an adaptive diffusion step size and dynamic convergence scheduling module;

[0086] S32, set the total number of steps in the diffusion process to N, and the diffusion state of each step to x t , where t=N,N-1,…,1,0;

[0087] S33, the initial diffusion state x N Set to a noise tensor that obeys a Gaussian distribution, i.e. Where I is the unit covariance matrix;

[0088] S34. Through the multimodal conditional control mechanism, the output fused semantic representation tensor is used as the multimodal conditional control vector, which is input into the stable diffusion generation model and serves as a guiding signal for generating content in the entire diffusion reverse denoising process;

[0089] S35. The multi-level semantic condition adaptive injection module hierarchically injects different semantic feature components of the fused semantic representation tensor into different layers of the U-Net backbone structure of the stable diffusion generative model. It injects spatial feature information into the shallow layers of the encoder and high-order semantic feature information into the deep layers. Through adaptive weight control, the adaptive injection of multi-level semantic conditions is completed.

[0090] S36, through the cross-modal attention guidance module, in each step of the reverse denoising process, the current noise state x is used t Together with the multimodal conditional control vector, multi-head cross-modal attention is performed on the spatial and semantic information of the fused semantic representation tensor and the current generated tensor;

[0091] S37, in each step of the reverse denoising process, through the multimodal conditional control mechanism and the cross-modal attention guidance module, based on the current generated state x t , fuse the semantic representation tensor, dynamically calculated adaptive weights of each modality, and adaptive gating units, fuse the feature flows of each modality, and use the semantic consistency deviation between the tensor generated at the previous moment and the fused semantic representation tensor at the current moment as the residual signal to participate in denoising prediction, and output the denoising prediction tensor

[0092]

[0093] Among them, m represents each mode category, λ m is the dynamically calculated adaptive weight of the mth modality, is the denoising prediction under the mth mode condition, R t is the semantic residual feedback term, γ t is the gating weight of the residual signal, α t is the noise scaling factor at step t, Represent tensors for fusion semantics;

[0094] The practical significance of the denoising prediction calculation formula lies in the dynamic injection of multimodal fusion semantic information into each step of the generation process. Through adaptive weighting and gating mechanisms, it coordinates the contributions of various modalities to content generation. Combined with semantic residual feedback from previous and subsequent moments, it effectively improves the consistency of the generated content with the target semantics. Specifically, it not only utilizes the adaptive weighting of each modal feature, but also enhances semantic coordination and detail completion between modalities through a multi-head cross-modal attention mechanism. At the same time, by introducing dynamic deviations between the generated content and the target semantics through residual signals, the model can continuously correct these deviations at each denoising step, achieving high-fidelity reconstruction of complex internet information. Unlike traditional single-condition or static weight generation methods, this method organically integrates multimodal information, semantic alignment, and dynamic generation optimization, ensuring that the generated content is highly consistent with the input target at the spatial, semantic, and structural levels. This innovative calculation method greatly improves the intelligent performance and robustness of multimodal generation systems in multiple scenarios and complex information, providing a solid technical foundation for high-quality internet information generation and analysis.

[0095] S38. In each step of the reverse denoising process, the multi-level semantic condition adaptive injection module dynamically adjusts the injection weight of the multimodal condition control vector according to the semantic consistency between the fused semantic representation tensor and the current generation state, thereby optimizing the contribution ratio of each modality to the generation result;

[0096] S39. In each step of the reverse denoising process, the adaptive diffusion step size and dynamic convergence scheduling module, through the semantic discrimination method, calculates the semantic matching score between the current generation state and the multimodal condition control vector in real time to judge the quality of the current generation state;

[0097] S310: During the reverse denoising iteration, when the adaptive diffusion step and dynamic convergence scheduling module determines that the semantic matching score is higher than the set threshold, the current generation state is output, and the adaptive diffusion step and dynamic convergence scheduling are completed; if the semantic matching score does not reach the set threshold, steps S36 to S39 are continued to continue the reverse denoising process;

[0098] S311, iteratively executing steps S36 to S310 in sequence, gradually reconstructing the noise tensor into the final multimodal joint output content generation tensor x0;

[0099] S312. The resulting generated tensor x0 is restored to multimodal joint output content through a decoding process, and the multimodal joint output content is output.

[0100] Through the process of the aforementioned stable diffusion generation model, the present invention achieves high-quality content generation and intelligent dynamic optimization guided by multimodal conditions. By adopting a U-Net backbone structure and combining it with a multimodal conditional control mechanism, the fused semantic representation tensor can be used as a guiding signal throughout the generation process, fully leveraging the synergistic advantages of multi-source data. The multi-level semantic conditional adaptive injection module enables dynamic injection and weighting of semantic information at different levels, significantly enhancing the generative model's ability to express spatial details and high-level semantics. The cross-modal attention guidance mechanism ensures deep alignment and information complementarity of features across modalities during denoising and reconstruction, improving the overall consistency and richness of multimodal output content. The adaptive diffusion step size and dynamic convergence scheduling strategy dynamically adjust the sampling process based on real-time semantic discrimination results, not only improving generation efficiency but also effectively preventing overfitting and inefficient computation. Through the innovative design of residual signals and adaptive gating units, the model possesses continuous error correction and robust optimization capabilities during denoising prediction. Overall, the present invention significantly improves the quality, flexibility, and adaptability of multimodal content generation, providing strong technical support for intelligent generation of internet information and multimodal analysis in complex scenarios.

[0101] In this embodiment, the S4 specifically includes:

[0102] S41, establishing a correspondence between the generated multimodal joint output content and the multimodal input sample set, using a unified sample identifier to form a one-to-one corresponding input and output sample pair;

[0103] S42. For each pair of multimodal joint output content and the corresponding multimodal input sample set, extract the fusion semantic representation tensor and As input features for semantic consistency verification;

[0104] S43. For each pair of fused semantic representation tensors and Calculate the semantic matching score S match :

[0105]

[0106] in, represents the vector inner product of the two, and They represent the L2 norm of the two respectively, the first term is the cosine similarity, the second term is the square of the Euclidean distance, β is the balance factor, S match Reflects the comprehensive consistency of the fused semantic representation tensor in terms of direction and distribution;

[0107] Semantic matching score S matchThe practical significance of the formula is that it provides a scientific, comprehensive and quantifiable semantic matching calculation standard for the semantic consistency verification link in the multimodal Internet information analysis method. The formula compares the fused semantic representation tensor of the generated content with the fused semantic representation tensor of the original input, and comprehensively adopts two underlying measurement methods, cosine similarity and Euclidean distance, to measure the directional similarity of semantic expression and reflect the absolute distance of feature distribution, effectively avoiding the bias that is prone to occur in single indicator judgment. Cosine similarity emphasizes the semantic consistency of two sets of features in high-dimensional space, while Euclidean distance examines the degree of distribution proximity between features. The weighted combination of the two enables the formula to accurately capture the dual consistency of multimodal information in expression content and feature structure. The introduction of the balance factor provides the algorithm with adaptive adjustment capabilities in different scenarios. The obtained semantic matching score not only provides an objective basis for subsequent consistency judgment, network parameter feedback adjustment and other steps, but also significantly improves the controllability, accuracy and robustness of multimodal generated content in complex Internet application scenarios. It is the key foundation for realizing full-process intelligent analysis and optimization.

[0108] S44, the semantic matching score S of each pair of input and output samples match Compare with the preset threshold δ, if S match <δ, the semantic consistency of the generated content is judged to be insufficient, and the inconsistent sample index is recorded;

[0109] S45. For samples whose semantic consistency does not meet the requirements, adjust the attention mechanism parameter α in the U-Net fusion network, use the feedback optimization method to correct the attention allocation weight, and obtain a new attention parameter set α′;

[0110] S46. Based on the modified attention mechanism parameter set α′, return to step S2 to step S4 and repeat the above semantic consistency verification and feedback optimization process until the semantic matching scores of all samples reach the preset threshold;

[0111] S47. When the semantic matching scores of all input and output samples are higher than the preset threshold δ, it is confirmed that the current combination of the U-Net fusion network and the stable diffusion generation model has achieved full-process semantic consistency between the multimodal output content and the input samples, and the process enters the next step of analysis.

[0112] The present invention effectively improves the fit and reliability of multimodal generated content and the original input samples at the deep semantic level. By using a unified sample identifier and one-to-one corresponding input and output sample pairs, accurate matching of generated content and input data is achieved, providing a solid foundation for subsequent consistency evaluation. By combining the innovative semantic matching score calculation of cosine similarity and Euclidean distance, the comprehensive consistency of the fused semantic representation tensor in direction and distribution can be fully reflected, which improves the scientific nature and discrimination ability of semantic evaluation. The feedback optimization mechanism enables the attention allocation weights of the U-Net fusion network to be adaptively adjusted according to the consistency results, and the model has the ability of continuous self-learning and dynamic correction. The closed-loop mechanism of multiple rounds of consistency verification and parameter updating not only significantly improves the semantic accuracy of the generated content, but also enhances the generalization and robustness of the system in complex Internet scenarios. Overall, the present invention provides a high-standard semantic quality control method for multimodal information analysis, which effectively guarantees the reliability and practical value of the analysis results.

[0113] In this embodiment, the S5 specifically includes:

[0114] S51. Based on the obtained semantic consistency verification results, mark the matching relationship between each multimodal joint output content and the multimodal input sample set, and screen out content pairs whose semantic consistency meets a set threshold;

[0115] S52. For multimodal joint output content that meets the semantic consistency requirements, combine it with structured Internet information associated with the multimodal input sample set, and use data fusion methods to achieve joint analysis of different information sources to form a multimodal analysis input unit;

[0116] S53, the fusion semantic representation tensor of the multimodal joint output content and the feature representation of structured Internet information S I Perform feature cascade;

[0117] S54, based on the fused multimodal analysis feature tensor T FA ,Through the event detection algorithm, the Internet event expression unit E is extracted, and each event expression unit is jointly represented by relevant text, image, audio and structured information;

[0118] S55. For each event expression unit, a label generation algorithm is used to generate a corresponding analysis label, where the analysis label includes structured analysis information such as event type, key information points, sentiment attributes, or source identification;

[0119] S56. Summarize and construct a structured analysis report of Internet information based on all event expression units and their analysis tags. The structured analysis report includes multimodal semantic descriptions, event links, statistical features, and visual analysis elements.

[0120] S57. Output the structured analysis report to the visualization terminal to realize the result display and decision support of Internet multimodal information.

[0121] This invention enables structured and intelligent processing of complex internet information. First, content is rigorously screened based on semantic consistency verification results to ensure that data entering the analysis process possesses high reliability and deep semantic alignment. Combining structured internet information associated with a multimodal input sample set, a data fusion strategy is employed to effectively integrate multimodal semantic features with structured attributes, significantly improving the comprehensiveness and granularity of the analysis. An event detection algorithm automatically extracts internet event expression units and leverages multi-source information to collaboratively represent event content, enhancing the richness and accuracy of the analysis results. A label generation algorithm automatically assigns structured analysis labels to each event, enabling intelligent labeling across multiple dimensions, including event type, key elements, sentiment, and source information. Finally, all event units and labels are aggregated into a structured analysis report, which is output via a visualization terminal, enabling users to efficiently gain in-depth insights from multimodal information and aid decision-making. Overall, this invention not only enhances the intelligent and automated level of internet information analysis but also provides efficient and reliable technical support for large-scale, diverse, and complex internet data analysis scenarios.

[0122] Example 1:

[0123] In order to verify the feasibility of the present invention in implementation, the present invention was applied to a large news portal platform, which adds more than 100,000 multimodal data including social news, comments, photo reports, short videos and user audio messages every day.

[0124] For example, regarding a traffic incident, the platform simultaneously collected news flashes, on-site photos, short videos of passersby, audio reports, and a structured public opinion index within minutes. Previously, the platform relied solely on text-based analysis algorithms, which often resulted in inaccurate event attribution, sentiment misjudgments, and missed detection of key public opinion nodes due to data fragmentation. For example, the label inconsistency rate between text news and short video comments on the same incident was as high as 18%, resulting in delays exceeding 40 minutes in responding to public opinion and a significant omission of key information points.

[0125] In the actual deployment of the system of the present invention, the platform collects all multimodal raw data, aligns them through unified timestamps and event IDs, and then inputs them into the multimodal U-Net fusion network. The system automatically encodes text, pictures, audio, structured data, etc. separately, and fuses a high-dimensional unified semantic feature tensor in the middle layer through structural sharing and cross-modal jump mechanisms. This tensor serves as a conditional input to drive the stable diffusion generation model to perform multi-step denoising reconstruction, and finally outputs multimodal joint content such as structured event summaries, emotional labels, key elements, etc. All newly generated content must be compared with the original data for semantic consistency, and the system automatically adjusts the model parameters until the output content fully matches the actual scenario.

[0126] After three months of operation, the platform's automatic attribution accuracy for hot events has significantly improved. Taking five typical hot events on the platform, including transportation, healthcare, and education, as examples, the platform's multimodal information analysis system outperformed traditional text analysis systems in terms of label consistency, event attribution accuracy, missed public opinion detection rate, automatic reporting timeliness, and user satisfaction. Specific results are shown in the table below.

[0127] Table 1 Performance comparison between the present invention and traditional algorithms

[0128]

[0129] As can be seen from Table 1, the Internet information analysis method based on multimodal data fusion of the present invention has significant advantages over traditional single-modal analysis systems in practical applications. First of all, whether in a variety of typical Internet hot events such as traffic emergencies, medical health, education reform, social livelihood or public safety, the label consistency rate of the system of the present invention remains above 97.6%, with the highest reaching 98.5%, while the label consistency rate of traditional systems is generally below 85%, and the gap between the two is obvious. This shows that the solution of the present invention can better integrate and understand multi-source heterogeneous information and achieve consistent induction of cross-modal content.

[0130] Secondly, in terms of event attribution accuracy, the system of the present invention has achieved over 98.7% for all event types, and even close to 100% for some individual events, while traditional systems mostly hover between 88% and 91%. This fully demonstrates that the model of the present invention has a higher level of intelligence in event understanding and automatic attribution, greatly reducing the burden of manual verification and the risk of misjudgment. In addition, in terms of the missed detection rate of public opinion, the missed detection rate of the system of the present invention is generally less than 0.6%, while the missed detection rate of traditional systems is as high as 4.5%-6.2%. This means that the multimodal fusion mechanism can discover and capture more key information, greatly improving the platform's coverage of complex public opinion.

[0131] In terms of the timeliness of automatic report generation, the system of the present invention can usually output a complete structured analysis report within 10 to 12 minutes, while the traditional system takes more than 40 minutes. Such efficiency improvement not only speeds up the response to hot events and public opinion warnings, but also provides a strong technical guarantee for the media and relevant departments to make timely decisions. Finally, in terms of the improvement rate of user satisfaction, the system of the present invention has achieved a satisfaction increase of more than 20% in all types of events, indicating that the experience improvement brought about by intelligent, automated, and multimodal in-depth analysis has been highly recognized by users and managers.

[0132] Overall, Table 1 fully demonstrates the innovation and practicality of the present invention in Internet multimodal information fusion, automatic event attribution, full-process tracking of public opinion, and intelligent decision support, bringing significant value enhancement to industry users.

[0133] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. An Internet information analysis method based on multimodal data fusion, characterized in that: The steps include: S1. Collect multimodal raw data from the Internet platform, synchronize and align the multimodal raw data, and construct a multimodal input sample set; S2. Input the multimodal input sample set into a U-Net fusion network. The U-Net fusion network includes an image encoder, a text encoder, an audio encoder, and a structured modality encoder. The U-Net fusion network outputs feature representation tensors of each modality respectively, fuses the feature representation tensors of each modality to form a multimodal fusion feature map, establishes skip connections between layers of the U-Net encoding path, and outputs a fused semantic representation tensor. S3. Construct a stable diffusion generation model, take the fused semantic representation tensor as the conditional control vector as input, perform multi-step denoising and reconstruction through the reverse diffusion process, and generate multimodal joint output content; S4, verifying the semantic consistency of the multimodal joint output content and the multimodal input sample set, calculating the semantic matching score, and if the semantic matching score is lower than the set threshold, adjusting the attention mechanism parameters in the U-Net fusion network, and returning to step S2 to step S4 until the semantic matching score meets the set conditions; S5. Based on the semantic consistency verification results, combined with the structured Internet information associated with the multimodal input sample set, a comprehensive analysis of the multimodal joint output content is performed, event expression units and analysis labels are constructed, and a structured analysis report of the Internet information is generated and output to the visualization terminal.

2. The Internet information analysis method based on multimodal data fusion according to claim 1 is characterized in that: The multimodal original data specifically includes image data, text data, audio data and web page structure data.

3. The Internet information analysis method based on multimodal data fusion according to claim 1 is characterized in that: The synchronous alignment of multimodal raw data refers to size normalization of image data, word segmentation and embedding vector conversion of text data, extraction of Mel-spectrogram features of audio data, label hierarchical analysis of web page structure data, and synchronous alignment of all modal data according to timestamps.

4. The Internet information analysis method based on multimodal data fusion according to claim 1 is characterized in that: The S2 specifically includes: S21. Construct a U-Net fusion network, where the U-Net fusion network includes an image encoder, a text encoder, an audio encoder, and a structured modality encoder; S22. Input the multimodal input sample set into the image encoder, text encoder, audio encoder, and structured modality encoder in the U-Net fusion network respectively. The image encoder adopts a multi-layer convolutional neural network structure, the text encoder adopts a Transformer network structure, the audio encoder adopts a hybrid structure of convolutional and recurrent neural networks, and the structured modality encoder adopts a joint structure of multi-layer perceptron and graph neural network. S23, the image encoder extracts features from the image data input to obtain the image feature representation tensor T I ; The text encoder extracts features from the text data input and obtains the text feature representation tensor T T ; The audio encoder extracts features from the audio data input and obtains the audio feature representation tensor T A ; The structured modal encoder extracts features from the web page structure data input and obtains the structured modal feature representation tensor T S ; S24. Setting a structurally shared sub-encoder module in the intermediate layer of each encoding path of the U-Net fusion network to apply a shared encoding operation to the intermediate feature tensors of each modality, where the intermediate feature tensors of each modality refer to the output feature tensors of each modality encoder in the non-final output stage and each intermediate layer in the U-Net fusion network; S25. Introduce a cross-modal jump connection mechanism in the U-Net fusion network, perform cross-modal jump connections between the intermediate feature tensors of the text encoder and the same-layer feature tensors of the image encoder, and perform cross-modal jump connections between the intermediate feature tensors of the audio encoder and the structured modality encoder, to achieve alignment and information collaboration between the intermediate layer features of different modalities; S26. Establish the downsampling structure of each encoding path in the U-Net fusion network, and represent the image feature tensor T respectively. I , text feature representation tensor T T , audio feature representation tensor T A , structured modal feature representation tensor T S Perform downsampling processing; S27. Establish a corresponding upsampling structure in the U-Net fusion network, and connect the feature tensors between each layer of the encoding path to the corresponding positions of the decoding path through the skip connection mechanism to ensure the synchronous transmission of context and fine-grained information; S28, performing feature fusion on the downsampled feature tensors of each modality processed by the structure-sharing sub-encoder module and the cross-modal jump connection mechanism to form a multimodal fusion feature map; S29. In the decoding path of the U-Net fusion network, the spatial dimension is restored layer by layer and the skip connection features are fused, and finally the fused semantic representation tensor is output.

5. The Internet information analysis method based on multimodal data fusion according to claim 1 is characterized in that: The S3 specifically includes: S31. Construct a stable diffusion generation model, wherein the stable diffusion generation model adopts a U-Net backbone structure and is provided with a multimodal condition control mechanism, a multi-level semantic condition adaptive injection module, a cross-modal attention guidance module, and an adaptive diffusion step size and dynamic convergence scheduling module; S32, set the total number of steps in the diffusion process to N, and the diffusion state of each step to x t , where t=N,N-1,…,1,0; S33, the initial diffusion state x N Set to a noise tensor that obeys a Gaussian distribution, i.e. Where I is the unit covariance matrix; S34. Through the multimodal conditional control mechanism, the output fused semantic representation tensor is used as the multimodal conditional control vector, which is input into the stable diffusion generation model and serves as a guiding signal for generating content in the entire diffusion reverse denoising process; S35. The multi-level semantic condition adaptive injection module hierarchically injects different semantic feature components of the fused semantic representation tensor into different layers of the U-Net backbone structure of the stable diffusion generative model. It injects spatial feature information into the shallow layers of the encoder and high-order semantic feature information into the deep layers. Through adaptive weight control, the adaptive injection of multi-level semantic conditions is completed. S36, through the cross-modal attention guidance module, in each step of the reverse denoising process, the current noise state x is used t Together with the multimodal conditional control vector, multi-head cross-modal attention is performed on the spatial and semantic information of the fused semantic representation tensor and the current generated tensor; S37, in each step of the reverse denoising process, through the multimodal conditional control mechanism and the cross-modal attention guidance module, based on the current generated state x t , fuse the semantic representation tensor, dynamically calculated adaptive weights of each modality, and adaptive gating units, fuse the feature flows of each modality, and use the semantic consistency deviation between the tensor generated at the previous moment and the fused semantic representation tensor at the current moment as the residual signal to participate in denoising prediction, and output the denoising prediction tensor Among them, m represents each mode category, λ m is the dynamically calculated adaptive weight of the mth modality, is the denoising prediction under the mth mode condition, R t is the semantic residual feedback term, γ t is the gating weight of the residual signal, α t is the noise scaling factor at step t, Represent tensors for fusion semantics; S38. In each step of the reverse denoising process, the multi-level semantic condition adaptive injection module dynamically adjusts the injection weight of the multimodal condition control vector according to the semantic consistency between the fused semantic representation tensor and the current generation state, thereby optimizing the contribution ratio of each modality to the generation result; S39. In each step of the reverse denoising process, the adaptive diffusion step size and dynamic convergence scheduling module, through the semantic discrimination method, calculates the semantic matching score between the current generation state and the multimodal condition control vector in real time to judge the quality of the current generation state; S310: During the reverse denoising iteration, when the adaptive diffusion step and dynamic convergence scheduling module determines that the semantic matching score is higher than the set threshold, the current generation state is output, and the adaptive diffusion step and dynamic convergence scheduling are completed; if the semantic matching score does not reach the set threshold, steps S36 to S39 are continued to continue the reverse denoising process; S311, iteratively executing steps S36 to S310 in sequence, gradually reconstructing the noise tensor into the final multimodal joint output content generation tensor x0; S312. The resulting generated tensor x0 is restored to multimodal joint output content through a decoding process, and the multimodal joint output content is output.

6. The Internet information analysis method based on multimodal data fusion according to claim 1 is characterized in that: The S4 specifically includes: S41, establishing a correspondence between the generated multimodal joint output content and the multimodal input sample set, using a unified sample identifier to form a one-to-one corresponding input and output sample pair; S42. For each pair of multimodal joint output content and the corresponding multimodal input sample set, extract the fusion semantic representation tensor and As input features for semantic consistency verification; S43. For each pair of fused semantic representation tensors and Calculate the semantic matching score S match : in, represents the vector inner product of the two, and They represent the L2 norm of the two respectively, the first term is the cosine similarity, the second term is the square of the Euclidean distance, β is the balance factor, S match Reflects the comprehensive consistency of the fused semantic representation tensor in terms of direction and distribution; S44, the semantic matching score S of each pair of input and output samples match Compare with the preset threshold δ, if S match <δ, the semantic consistency of the generated content is judged to be insufficient, and the inconsistent sample index is recorded; S45. For samples whose semantic consistency does not meet the requirements, adjust the attention mechanism parameter α in the U-Net fusion network, use the feedback optimization method to correct the attention allocation weight, and obtain a new attention parameter set α ′ ; S46, based on the modified attention mechanism parameter set α ′ , return to step S2 to step S4, and repeat the above semantic consistency verification and feedback optimization process until the semantic matching scores of all samples reach the preset threshold; S47. When the semantic matching scores of all input and output samples are higher than the preset threshold δ, it is confirmed that the current combination of the U-Net fusion network and the stable diffusion generation model has achieved full-process semantic consistency between the multimodal output content and the input samples, and the process enters the next step of analysis.

7. The Internet information analysis method based on multimodal data fusion according to claim 1 is characterized in that: The S5 specifically includes: S51. Based on the obtained semantic consistency verification results, mark the matching relationship between each multimodal joint output content and the multimodal input sample set, and screen out content pairs whose semantic consistency meets a set threshold; S52. For multimodal joint output content that meets the semantic consistency requirements, combine it with structured Internet information associated with the multimodal input sample set, and use data fusion methods to achieve joint analysis of different information sources to form a multimodal analysis input unit; S53, the fusion semantic representation tensor of the multimodal joint output content and the feature representation of structured Internet information S I Perform feature cascade; S54, based on the fused multimodal analysis feature tensor T FA ,Through the event detection algorithm, the Internet event expression unit E is extracted, and each event expression unit is jointly represented by relevant text, image, audio and structured information; S55. For each event expression unit, a label generation algorithm is used to generate a corresponding analysis label, where the analysis label includes structured analysis information such as event type, key information points, sentiment attributes, or source identification; S56. Summarize and construct a structured analysis report of Internet information based on all event expression units and their analysis tags. The structured analysis report includes multimodal semantic descriptions, event links, statistical features, and visual analysis elements. S57. Output the structured analysis report to the visualization terminal to realize the result display and decision support of Internet multimodal information.

Images