A network information service optimization system and method supporting multimodal data

Abstract

This invention belongs to the technical field of network information services, specifically referring to a network information service optimization system and method that supports multimodal data. The system integrates multimodal heterogeneous data, constructs a cross-modal unified semantic vector, introduces an improved cross-modal Transformer prediction engine, and designs a gradient manifold total loss function jointly constructed by cross-modal contrastive loss, inter-layer balance constraint terms, and low-rank regularization terms. Combined with a Hessian quadratic saddle point avoidance strategy and spectral normalization low-rank projection operation, it achieves deep fusion of multimodal temporal features and stable forward prediction of user service needs. The prediction results are combined with information service resource collaborative scheduling to dynamically generate instructions for content cache preheating, recommendation strategy adjustment, computing resource scaling, and network bandwidth configuration, realizing a technological transformation from user intent perception to end-to-end adaptive optimization of information services.

Description

Technical Field

[0001] This invention belongs to the technical field of network information services, specifically referring to a network information service optimization system and method that supports multimodal data. Background Technology

[0002] With the rapid development of mobile internet, social media, and IoT applications, network information service systems need to simultaneously process multimodal heterogeneous data such as text, images, audio, and video to achieve accurate content recommendation, intelligent search, and dynamic resource scheduling. However, existing multimodal service systems mostly adopt single-modal independent processing or simple feature concatenation, lacking effective mining of deep semantic relationships across modalities, making it difficult to form a unified view of user intent perception and scenario understanding. At the service decision-making level, current mainstream methods still rely on rule engines, threshold triggering, or linear weighting strategies, failing to perform forward-looking optimization based on real-time user behavior and environmental context, resulting in delayed service response and low resource utilization. In addition, multimodal data often suffers from modality loss, noise interference, and inconsistent feature spaces, making existing models prone to getting trapped in local optima during training, lacking generalization ability, and unable to adapt to dynamically changing service scenarios. Therefore, there is an urgent need for an intelligent system that can deeply integrate multimodal features, predict user service needs in advance, and collaboratively optimize the information service chain to improve service quality and system operating efficiency. Summary of the Invention

[0003] To address the aforementioned issues and overcome the shortcomings of existing technologies, this invention provides a network information service optimization system and method supporting multimodal data. This system solves the problems of insufficient multimodal data fusion, lagging demand forecasting, and rigid service resource scheduling in network information services. The proposed system acquires user interaction behavior and content modality information through a multimodal data acquisition module, and maps heterogeneous data to a unified semantic space using a multimodal feature extraction and alignment module. Based on this, an improved cross-modal Transformer prediction engine incorporating modality-aware gating and single-peak width constraints is constructed to learn the deep dynamic features of multimodal sequences and achieve joint optimization. The goal is to proactively predict users' future service needs. Based on the prediction results, the service optimization decision module generates control instructions such as content caching strategies, recommendation list adjustments, computing resource scaling, and network bandwidth configuration to achieve adaptive closed-loop optimization of information services. Simultaneously, intermodal energy balance constraints, spectral regularization, and saddle point avoidance strategies are introduced during model training to ensure the prediction engine possesses stable convergence characteristics and low-rank generalization capabilities in complex multimodal environments. Through these designs, this invention achieves early identification of user intent and proactive allocation of information service resources, enabling the system to possess cross-modal understanding, forward-looking decision-making capabilities, and efficient collaboration, significantly improving service accuracy and system throughput in scenarios such as personalized recommendations and intelligent search.

[0004] To solve the above technical problems, the technical solution adopted by the present invention is as follows: The proposed solution is a network information service optimization system that supports multimodal data, which is applied to network information service scenarios such as Internet content distribution, personalized recommendation, intelligent question answering or online advertising. The system includes: a service access gateway, a content storage node and a computing resource pool deployed on a cloud server or edge node. The multimodal data acquisition module includes a user behavior tracking unit, a content feature extraction unit, and an environmental sensor data interface. It is used to acquire multimodal raw data such as user interaction sequences, click logs, text comments, meta tags of browsed images / audio / video, and terminal device status. The multimodal feature extraction and alignment module, including a text encoder, a visual encoder, an audio encoder, and a cross-modal alignment network, is used to convert heterogeneous modal data into a unified semantic vector of fixed dimensions and eliminate the semantic gap between modalities through contrastive learning pre-training. The multimodal fusion prediction engine is deployed on a central inference server or edge intelligent node. It has an improved cross-modal Transformer model built in, which performs time series modeling through aligned multimodal semantic sequences to predict the probability distribution of user service demand within a preset time window in the future. The service optimization decision module dynamically generates cache preheating instructions for the content distribution network, recall and ranking strategy adjustment instructions for the recommendation system, elastic scaling instructions for computing instances, and priority scheduling configuration of network bandwidth based on the demand prediction results output by the multimodal fusion prediction engine, thereby achieving collaborative optimization of the entire information service chain.

[0005] Furthermore, the multimodal fusion prediction engine processes multimodal semantic sequences through an improved cross-modal Transformer model to predict future user service needs, specifically including the following: The temporal construction submodule concatenates the multimodal unified semantic vectors of continuous time steps into a multimodal temporal sample set according to time windows, and performs missing modality masking enhancement and sequence standardization on the samples; The predictive modeling submodule establishes an improved cross-modal Transformer model. This model introduces modality-aware gating units and channel width single-peak constraints between multi-head attention layers to construct a deep fusion structure with inter-layer energy balance characteristics. Using multimodal time-series samples as input, the model is trained through a joint loss function to output the service demand category distribution or interaction probability for multiple future time steps.

[0006] Furthermore, the process of establishing an improved cross-modal Transformer model, training it with multimodal time-series samples as input, and generating future service demand predictions specifically includes the following steps: Step S1: Arrange the multimodal unified semantic vectors according to the sampling timestamps to construct a multidimensional input tensor containing modality type labels, time position codes, and modality missing masks; perform sliding window truncation and batch normalization on the input tensor to form standardized multimodal temporal training samples; Step S2: Stack the feedforward weights and attention projection matrices of each layer of the cross-modal Transformer into a block shift parameter set, and constrain the number of feature channels of each layer to satisfy a single-peak distribution that first monotonically increases and then monotonically decreases, so as to ensure the stability of the spectral characteristics of deep mapping and sufficient information flow, and construct an improved cross-modal Transformer model with single-peak layer width and inter-layer consistency constraints. Step S3: Based on the cross-entropy loss for service demand classification, introduce a cross-modal contrastive loss term for aligning the joint distribution of multimodal distributions and marginal distributions, an inter-layer balance constraint term for constraining the feature energy of adjacent layers, and a low-rank regularization term for suppressing the spectral complexity of the weight matrix. Construct a Lyapunov-type total loss function on the gradient manifold to achieve stability control and implicit low-rank induction of model training. Step S4: Using multimodal time-series training samples as input, the improved cross-modal Transformer model is updated with weights by adopting a parameter update strategy that combines adaptive moment estimation and gradient flow discretization based on the Lyapunov-type total loss function; after each iteration, spectral normalization and low-rank projection operations are introduced to maintain the low-rank structure of the attention weight matrix and satisfy the stability boundary. Step S5: During the weight update process in step S4, multiple perturbation directions are sampled periodically from the random subspace of the parameter space. The Hessian quadratic approximation of the total loss function at the current weight is constructed using the finite difference method. When a negative curvature is detected corresponding to a certain perturbation direction, a controlled perturbation is applied along that direction to make the parameter state jump out of the saddle point region and avoid training from stagnating at a non-global optimal solution. Step S6: During the iteration process of steps S4 and S5, the decay rate of the total loss function and the feature spectrum gap of the interlayer weights are monitored in real time; by estimating the upper bound coefficient of the Lyapunov derivative online, when the coefficient exceeds the preset convergence threshold and the loss function shows an exponential decay trend, it is determined that the model has reached stable convergence in the Lyapunov sense, the network parameters are frozen and the training is ended, and the improved cross-modal Transformer model after training is obtained. Step S7: Use the trained model to infer the online multimodal data stream and generate predicted values ​​of user service demand within a future preset time window.

[0007] The present invention also provides a method for optimizing network information services that supports multimodal data, the method comprising the following steps: Step M1: Data Acquisition and Preprocessing; Acquire multimodal raw data such as user interaction sequences, click logs, text comments, meta tags of browsed images / audio / video, and terminal device status through the multimodal data acquisition module; Perform masking processing on missing modalities, and perform standardization and alignment operations on each modal data; Step M2: Multimodal feature extraction and alignment; using the text encoder, visual encoder and audio encoder in the multimodal feature extraction and alignment module, high-level semantic features of each modality are extracted respectively; through the cross-modal alignment network, the heterogeneous modal features are mapped to a unified joint semantic space based on contrastive learning to generate a fixed-dimensional multimodal unified semantic vector sequence; Step M3: Prospective prediction of user service demand; The multimodal unified semantic vector sequence is input into the trained multimodal fusion prediction engine, and the improved cross-modal Transformer model performs deep fusion and modeling of multimodal temporal features to predict the probability distribution of user service demand within a preset time window in the future. Step M4: Information service resource collaborative optimization decision; The service optimization decision module dynamically generates instructions for content cache preheating, recommendation strategy adjustment, computing resource scaling and network bandwidth scheduling based on the probability distribution of user service needs, and sends them to the corresponding execution units to achieve closed-loop adaptive optimization of the entire information service link.

[0008] The training process of the improved cross-modal Transformer model is as described in steps S1 to S7.

[0009] The beneficial effects achieved by the present invention using the above structure are as follows: First, this invention constructs an improved cross-modal Transformer prediction engine that incorporates modality-aware gating and single-peak width constraints. This engine enables deep temporal fusion of multimodal heterogeneous data and accurate anticipation of user needs. Traditional methods often employ independent processing of single modalities or simple feature splicing, making it difficult to capture deep semantic relationships across modalities, resulting in delayed perception of user intent. This invention maps multimodal data such as text, images, and audio to a unified semantic space and utilizes a deep network with a single-peak layer width structure for temporal modeling. This allows the system to anticipate user service requests before they explicitly express their needs. This cross-modal anticipatory perception capability fundamentally solves the problems of ambiguous intent understanding and response delays in high-noise and modality-deficient scenarios, enabling the system to upgrade from passive response to proactive service.

[0010] Secondly, this invention ensures the stability and consistency of the feature space structure during multimodal deep fusion by using interlayer energy balance constraints and modal distribution alignment mechanisms. In the layered mapping of cross-modal Transformers, the energy decay or expansion of different modal features often leads to one modality dominating prediction while ignoring the complementary information of other modalities. This invention introduces an interlayer balance constraint term into the loss function to explicitly constrain the difference in the feature Gram matrix of adjacent layers, and supplements it with distribution alignment regularization based on Wasserstein distance to ensure that the representation structure within each modality is not destroyed during deep transmission. The cross-modal attention mechanism can continuously and evenly aggregate multi-source information, which significantly improves the model's representation robustness and cross-modal collaborative reasoning ability under complex conditions such as modality missing and noise interference.

[0011] Finally, this invention employs a saddle point avoidance and low-rank convergence strategy based on gradient manifold stability during model training. This enables large-scale multimodal prediction models to reliably converge to a flat minimum region with good generalization performance. The saddle point problem in high-dimensional parameter space is a key factor leading to unstable training and poor generalization ability of multimodal models. This invention actively causes the parameter states to escape strict saddle points by online estimation of Hessian quadratic forms to detect negative curvature directions and applying controlled perturbations. At the same time, combined with spectral regularization and low-rank projection operations, the model is guided to converge to an implicit low-rank stable region. This training mechanism not only accelerates model convergence but also enables the model to maintain excellent adaptability and long-term operational stability in differentiated service scenarios and dynamic data streams, significantly improving the engineering usability of this invention in practical businesses such as Internet content distribution and personalized recommendation. Attached Figure Description

[0012] The accompanying drawings are provided to further understand the present invention and form part of the specification. They are used together with the embodiments of the present invention to explain the invention and do not constitute a limitation thereof.

[0013] Figure 1 This is a schematic diagram of the network information service optimization system and method supporting multimodal data proposed in this invention. The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0014] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0015] Example 1: This solution proposes a network information service optimization system that supports multimodal data and is applied to Internet content distribution and personalized recommendation scenarios. The system includes: a service access gateway deployed in the cloud, distributed content storage nodes, and an elastic computing instance resource pool.

[0016] The multimodal data acquisition module includes a front-end event tracking SDK, a server-side log collector, and a content analysis interface. It is used to acquire user browsing, clicking, liking, and sharing behavior sequences, as well as multimodal features such as title text, cover image, and descriptive audio clips corresponding to the interacted content. The multimodal feature extraction and alignment module includes a BERT-based text encoder, a ViT-based visual encoder, a VGGish-based audio encoder, and a multimodal contrast alignment layer. It uses cross-modal contrast loss to embed and map text, images, and audio into a shared semantic space. The multimodal fusion prediction engine is deployed on a GPU inference server and incorporates an improved cross-modal Transformer model. This model consists of a 6-layer encoder and a 2-layer decoder, with 8 attention heads per layer. The feature channel dimension follows a unimodal structure of 256→512→1024→512→256. The model takes the multimodal semantic sequence of the past 10 time steps as input and predicts the probability distribution of user clicks on content categories within the next 5 time steps. The service optimization decision module includes a cache scheduler, a recommendation reorderer, and a resource orchestrator. Based on the predicted demand distribution, it pre-caches high-probability content to edge nodes, adjusts the recall weight of the recommendation candidate pool, and coordinates the scaling of the number of microservice instances to ensure service quality.

[0017] Example 2, based on Example 1, details the training process of the improved cross-modal Transformer model in the multimodal fusion prediction engine. It describes the construction of an improved cross-modal Transformer model with single-peak layer width and inter-layer consistency constraints, and the training process using multimodal time-series training samples. Specifically, it includes the following steps: Step S1: The multimodal unified semantic vectors aggregated every 30 seconds in the past 30 minutes are used to form a time sequence. Learnable modality type encoding and time step position encoding are added. For missing modality positions, all-zero vectors are generated and masked to 1. The sequence is cut into segments using a sliding window of length 20 and Z-score normalization is performed to form a training sample set. Step S2: Define an improved cross-modal Transformer model, in which the feature dimensions of each layer's attention module and feedforward network follow a single-peak structure of expansion followed by contraction, that is, starting from d_model=256, increasing layer by layer to 512, 1024 and then decreasing to 512, 256; at the same time, organize the weight matrices of each layer in a block-shift form, so that the entire network mapping can be represented as a constrained family of adjacency operators, and constraining the feature energy between layers to maintain approximate conservation; Step S3: Construct the Lyapunov-type total loss function, using the following formula:

[0018] in, Total loss This is the focus loss for multi-label classification, used to measure the prediction error of the user's future demand category; For cross-modal contrastive loss, positive sample pairs (simultaneous image, text, and audio) are forced to be close in semantic space, while negative sample pairs are forced to be far apart. The interlayer balance constraint term is defined as the sum of the squares of the Frobenius norms of the differences between the Gram matrices of the output features of each layer, i.e. , , This is the output feature matrix of the l-th layer; For the spectral regularization term, calculate the sum of squared singular values ​​of all attention weight matrices; , and To balance hyperparameters; Step S4: The AdamW optimizer is used in combination with gradient flow discretization to update the weights. After each training step, singular value decomposition is performed on the weight matrix, and singular values ​​smaller than the threshold are truncated and reconstructed to maintain the low-rank property. Step S5: Every 200 training steps, randomly sample 5 orthogonal perturbation directions from the parameter space and use finite difference... Estimate the Hessian vector product; if a negative value is detected, update along that direction. This helps to escape the saddle point; Step S6: Track the changes in the total loss sequence and inter-layer balance term, estimate the upper bound of the Lyapunov derivative using exponential moving average, and trigger early stopping and freeze parameters when the upper bound is below 0.01 for 50 consecutive cycles and the loss decrease rate is stably below 1e-4, thus obtaining the trained model. Step S7: Deploy the trained model online, consume multimodal data streams in real time, output the distribution of user content click intents within the next 5 minutes, and drive the service optimization decision module to execute corresponding instructions.

[0019] Example 3: Based on Example 2, in step S3, the inter-layer balance constraint term L_bal further introduces modality-aware weights, calculates the inter-layer energy difference for the feature subspaces corresponding to different modal branches, so as to maintain the stability of the internal representation structure of each modality in deep transmission, and adds a distribution alignment regularization term based on Wasserstein distance in the cross-modal attention layer to further enhance the consistency of multimodal fusion.

[0020] Example 4, based on Example 3, further introduces a dynamic adaptive mechanism. In the inter-layer balance constraint term of step S3, optimization of the cross-modal contrastive loss function for multimodal time series samples is added: During model training, the decay rate of the total loss function and the feature spectrum gap of the interlayer weights are monitored in real time. When the online estimated upper bound coefficient of the Lyapunov derivative exceeds the preset convergence threshold and the gradient manifold shows an exponential decay trend, the model is determined to have reached a stable convergence state, the network parameters are frozen, and training ends.

[0021] Specifically, the steps are as follows: Step S4: After each weight update, spectral pruning and low-rank projection operations are introduced to maintain the singular value decomposition of the attention weight matrix.

[0022] Example 5: Based on Example 4, this example further samples the perturbation direction in step S5 and uses the finite difference method to construct a Hessian quadratic approximation of the total loss function at the current weight. When a negative curvature corresponding to a certain perturbation direction is detected, a controlled perturbation is applied along that direction to make the parameter state jump out of the saddle point region and avoid training from stagnating at a non-global optimal solution.

[0023] Step S6: Track the feature spectrum gap between the total loss sequence and the interlayer weights; estimate the upper bound coefficient of the Lyapunov derivative online. When the coefficient exceeds the preset convergence threshold and the loss function shows an exponential decay trend, determine that the model has reached stable convergence, freeze the network parameters and end the training. Step S7: Use the trained model to infer the online multimodal data stream and generate predicted values ​​of user service demand within a future preset time window.

[0024] Example 6: Based on Example 5, when the perturbation direction in step S5 corresponds to negative curvature, a controlled perturbation is applied along that direction to make the parameter state jump out of the saddle point region, thus avoiding training from stagnating at a non-global optimal solution.

[0025] Step S6: Track the feature spectrum gap between the total loss sequence and the interlayer weights; estimate the upper bound coefficient of the Lyapunov derivative online. When the coefficient exceeds the preset convergence threshold and the loss function shows an exponential decay trend, determine that the model has reached stable convergence, freeze the network parameters and end the training. Step S7: Use the trained model to infer the online multimodal data stream and generate predicted values ​​of user service demand within a future preset time window.

[0026] Example 7: This example is based on Example 6. When the perturbation direction in step S6 corresponds to negative curvature, a controlled perturbation is applied along that direction to make the parameter state jump out of the saddle point region, thus avoiding training from stagnating at a non-global optimal solution.

[0027] Step S7: Use the trained model to infer the online multimodal data stream and generate predicted values ​​of user service demand within a future preset time window.

[0028] Example 8 provides a method for optimizing network information services that supports multimodal data, specifically applied to internet content distribution and personalized recommendation scenarios. The method is executed based on the system described in Example 1 and includes the following steps: Step M1: By integrating the front-end event tracking SDK and the server-side log data collection module, obtain the user's browsing, clicking, and sharing behavior sequences within the past 30 minutes, as well as the corresponding content's title text, cover image, and audio clips; for missing image modalities, fill in all-zero vectors and set the mask to 1; segment all text and map it to word vectors, scale images to 224x224 pixels and normalize them, and convert audio to Mel spectrograms to complete the raw data preprocessing; Step M2: The preprocessed text, image, and audio data are fed into the BERT text encoder, ViT visual encoder, and VGGish audio encoder, respectively, to extract 768-dimensional single-modal feature vectors; then, through an alignment network trained based on cross-modal contrastive loss, all features are mapped to a 256-dimensional unified semantic space to form a multimodal semantic vector sequence arranged with time steps. Step M3: Input the sequence of length 20 time steps into the already trained improved cross-modal Transformer prediction engine. This engine uses a 6-layer encoder with a unimodal structure of feature channel dimensions: 256→512→1024→512→256. The model performs temporal modeling on the sequence and outputs the probability distribution of the user's click intent for various types of content in the next 5 minutes. For example, it predicts that the user's click probability for the "Technology News" category is 0.85, and for "Sports Videos" it is 0.23. Step M4: The service optimization decision module receives the prediction results. When the probability of a certain content category is higher than the threshold of 0.7, the cache scheduler immediately preheats the popular content of that category from the central node to the edge CDN nodes; the recommendation re-ranking module increases the recall weight and ranking score of the content of that category when the user refreshes the page next time; at the same time, the resource orchestrator pre-expands the computing instances of the recommendation algorithm microservice by 20% to ensure that the peak response latency is less than 50 milliseconds. Thus, a complete closed-loop process from perception and prediction to service optimization is completed.

[0029] The method in this embodiment, by performing the above steps, achieves early perception of users' potential intentions and proactive adaptation of service resources at the millisecond level, significantly improving click-through rate and user session duration.

Claims

1. A network information service optimization system supporting multimodal data, characterized in that: The system includes a service access gateway, content storage nodes, and a computing resource pool deployed on cloud servers or edge nodes. The multimodal data acquisition module is used to acquire multimodal raw data such as user interaction sequences, click logs, text comments, meta tags of browsed images / audio / video, and terminal device status; The multimodal feature extraction and alignment module, including a text encoder, a visual encoder, an audio encoder, and a cross-modal alignment network, is used to convert heterogeneous modal data into a unified semantic vector of fixed dimensions and eliminate the semantic gap between modalities through contrastive learning pre-training. The multimodal fusion prediction engine is deployed on a central inference server or edge intelligent node. It has an improved cross-modal Transformer model built in, which performs time series modeling through aligned multimodal semantic sequences to predict the probability distribution of user service demand within a preset time window in the future. The service optimization decision module dynamically generates cache preheating instructions for the content distribution network, recall and ranking strategy adjustment instructions for the recommendation system, elastic scaling instructions for computing instances, and priority scheduling configuration of network bandwidth based on the demand prediction results output by the multimodal fusion prediction engine, thereby achieving collaborative optimization of the entire information service chain.

2. The network information service optimization system supporting multimodal data according to claim 1, characterized in that: The multimodal raw data includes a combination of at least two modalities among text data, image data, audio data, and video data.

3. The network information service optimization system supporting multimodal data according to claim 2, characterized in that: The multimodal fusion prediction engine processes multimodal semantic sequences through an improved cross-modal Transformer model to predict future user service needs. This process specifically includes the following: The temporal construction submodule concatenates the multimodal unified semantic vectors of continuous time steps into a multimodal temporal sample set according to time windows, and performs missing modality masking enhancement and sequence standardization on the samples; The predictive modeling submodule establishes an improved cross-modal Transformer model. This model introduces modality-aware gating units and channel width single-peak constraints between multi-head attention layers to construct a deep fusion structure with inter-layer energy balance characteristics. Using multimodal time-series samples as input, the model is trained through a joint loss function to output the service demand category distribution or interaction probability for multiple future time steps.

4. The network information service optimization system supporting multimodal data according to claim 3, characterized in that: The process of establishing an improved cross-modal Transformer model, training it with multimodal time-series samples as input, and generating predictions of future service demand specifically includes the following steps: Step S1: Arrange the multimodal unified semantic vectors according to the sampling timestamps to construct a multidimensional input tensor containing modality type labels, time position codes, and modality missing masks; perform sliding window truncation and batch normalization on the input tensor to form standardized multimodal temporal training samples; Step S2: Stack the feedforward weights and attention projection matrices of each layer of the cross-modal Transformer into a block shift parameter set, and constrain the number of feature channels of each layer to satisfy a single-peak distribution that first monotonically increases and then monotonically decreases, so as to ensure the stability of the spectral characteristics of deep mapping and sufficient information flow, and construct an improved cross-modal Transformer model with single-peak layer width and inter-layer consistency constraints. Step S3: Based on the service demand classification cross-entropy loss, introduce a cross-modal contrastive loss term for aligning the multimodal joint distribution and the marginal distribution, an inter-layer balance constraint term for constraining the feature energy of adjacent layers, and a low-rank regularization term for suppressing the spectral complexity of the weight matrix. Weight the three terms with the classification cross-entropy loss to construct the total loss function on the gradient manifold; obtain the output feature matrix of each layer, calculate the Gram matrix of the output feature matrix of each layer, sum the squared Frobenius norm of the difference between the Gram matrices of adjacent layers over all layers, obtain the singular values ​​of all attention weight matrices, and calculate the sum of squares of all singular values. Step S4: Using multimodal time-series training samples as input, and based on the total loss function, perform the improved cross-modal Transformer model using a parameter update strategy that combines adaptive moment estimation and gradient flow discretization, while maintaining the low-rank structure of the attention weight matrix and satisfying the stability boundary. Step S5: During the weight update process in step S4, multiple perturbation directions are sampled periodically from the random subspace of the parameter space. The Hessian quadratic approximation of the total loss function at the current weight is constructed using the finite difference method. When a negative curvature corresponding to a certain perturbation direction is detected, a controlled perturbation is applied along that direction to make the parameter state jump out of the saddle point region. Step S6: During the iteration process of steps S4 and S5, the decay rate of the total loss function and the feature spectrum gap of the interlayer weights are monitored in real time; by estimating the upper bound coefficient of the Lyapunov derivative online, when the upper bound coefficient exceeds the preset convergence threshold and the total loss function shows an exponential decay trend, it is determined that the model has reached a stable convergence state, the network parameters are frozen and the training is ended, and the improved cross-modal Transformer model after training is obtained. Step S7: Use the trained model to infer the online multimodal data stream and generate predicted values ​​of user service demand within a future preset time window; In step S3, the inter-layer balance constraint term also introduces modality-aware weights to calculate the inter-layer energy difference for the feature subspaces corresponding to different modal branches.

5. A method for optimizing network information services supporting multimodal data according to any one of claims 1-5, characterized in that: The method includes the following steps: Step M1: Acquire raw multimodal data through the multimodal data acquisition module; perform masking processing on missing modalities, and perform standardization and alignment operations on each modal data; Step M2: Utilize the text encoder, visual encoder, and audio encoder in the multimodal feature extraction and alignment module to extract high-level semantic features of each modality; through the cross-modal alignment network, map the heterogeneous modal features to a unified joint semantic space based on contrastive learning to generate a fixed-dimensional multimodal unified semantic vector sequence; Step M3: Input the multimodal unified semantic vector sequence into the trained multimodal fusion prediction engine. The improved cross-modal Transformer model performs deep fusion and modeling of multimodal temporal features to predict the probability distribution of user service demand within a preset time window in the future. Step M4: The service optimization decision module dynamically generates instructions for content cache preheating, recommendation strategy adjustment, computing resource scaling and network bandwidth scheduling based on the probability distribution of user service needs, and sends them to the corresponding execution units to achieve closed-loop adaptive optimization of the entire information service chain.

Images