Big data resource service method and system based on multi-modal data fusion

Abstract

The invention discloses a big data resource service method and system based on multi-modal data fusion, belongs to the technical field of big data processing, and solves the problems that a scheduling scheme is formulated only by calculating an energy consumption value and a resource utilization rate of a server in an existing method, the adaptability to dynamic changes is poor, and the overall performance and the resource utilization rate of the system are reduced. The method comprises the steps that a multi-modal data sequence is preprocessed, a shared database is created, multi-modal data are encrypted and distributed to a distributed hash table of distributed nodes, a resource retrieval instruction is recognized based on a resource scheduling model, and the resource scheduling model dynamically deploys a data transmission task based on target data attributes and real-time transmission path information; according to the method, the resource scheduling model considering the multi-modal data distribution and transmission characteristics in the distributed environment is constructed, collaborative learning and optimization among distributed nodes are realized, the transmission path is dynamically optimized, the delay and energy consumption of data transmission are reduced, and the intelligent level of scheduling is improved.

Description

Technical Field

[0001] The present invention belongs to the field of big data processing technology, and specifically relates to a big data resource service method and system based on multimodal data fusion. Background Art

[0002] With the rapid development of information technology, big data has been widely used in various fields. However, the scheduling and optimal allocation of big data resources face numerous challenges. Traditional resource scheduling methods often struggle to meet the complex and ever-changing demands of big data processing, especially in scenarios involving multimodal data fusion.

[0003] Multimodal data fusion involves integrating and analyzing data from different modalities (such as text, images, video, and audio) to uncover more comprehensive and in-depth information. In practical applications, multimodal data fusion faces challenges such as diverse data sources, complex data formats, and enormous data volumes. Traditional resource scheduling methods often allocate resources based on the characteristics of single-modal data, making it difficult to adapt to the complex demands of multimodal data fusion. For example, when processing large-scale image and text data, allocating resources based solely on data volume or the number of computational tasks may result in insufficient resources for processing tasks in certain modalities, thereby impacting overall processing efficiency.

[0004] Chinese patent CN113204429B discloses a resource scheduling method and system, scheduling equipment, and medium for a data center, including: obtaining energy consumption data of each server in the data center; wherein the energy consumption data includes the resource utilization rate of the server and the performance data of each component of the server; calculating the energy consumption value of each server based on the energy consumption data and a preset energy consumption calculation model; and, using a genetic algorithm to calculate and find an optimal resource scheduling scheme based on the resource utilization rate and energy consumption value of each server, and scheduling or shutting down each server in the data center according to the optimal resource scheduling scheme. However, existing methods only formulate scheduling schemes by calculating the energy consumption value and resource utilization rate of the server, have poor adaptability to dynamic changes, do not consider the data distribution and transmission characteristics in a distributed environment, and reduce the overall performance and resource utilization of the system. To address the above problems, we propose a big data resource service method and system based on multimodal data fusion. Summary of the Invention

[0005] The purpose of the present invention is to address the shortcomings of the existing technology and provide a big data resource service method and system based on multimodal data fusion, which solves the problem that the existing method only formulates scheduling plans by calculating the energy consumption value and resource utilization of the server, has poor adaptability to dynamic changes, does not consider the data distribution and transmission characteristics in a distributed environment, and reduces the overall performance and resource utilization of the system.

[0006] The present invention is implemented as follows: a big data resource service method based on multimodal data fusion, characterized in that it includes:

[0007] Distributed acquisition of multimodal data sequences collected by data sources, preprocessing of multimodal data sequences based on data fusion technology to obtain multimodal data represented by low-dimensional feature vectors, and creation of a shared database using the preprocessed low-dimensional feature vectors;

[0008] Based on peer-to-peer technology, the multimodal data in the shared database is encrypted and distributed to the distributed hash table of the distributed nodes, and the data location of the multimodal data in the distributed nodes is determined by the hash function;

[0009] In response to a resource retrieval instruction for a subject, identifying the resource retrieval instruction based on a pre-built resource scheduling model, and triggering a resource access control request;

[0010] In response to a resource access control request, traverse a distributed hash table of at least one group of distributed nodes based on cosine similarity to extract target data to be scheduled and target data attributes;

[0011] The resource scheduling model dynamically deploys data transmission tasks based on target data attributes and real-time transmission path information, completes point-to-point encrypted transmission of target data to be scheduled based on data transmission tasks, and generates retrieval results in combination with the large language model interaction.

[0012] Preferably, the method for preprocessing a multimodal data sequence based on data fusion technology includes:

[0013] Load multimodal data sequences, delete duplicate data and abnormal data in the multimodal data sequences, and perform numerical quantification of type labeling, attribute labeling, dimension labeling, and importance labeling on the multimodal data based on data type, data attribute, data dimension, and data importance to obtain the labeled multimodal data sequences;

[0014] Obtain the labeled multimodal data sequence, and define the preprocessing importance of the multimodal data in the multimodal data sequence based on the probability density function combined with the weighted average algorithm;

[0015] Determine whether the preprocessing importance of the multimodal data exceeds a preset importance threshold;

[0016] If the preprocessing importance of the multimodal data exceeds a preset importance threshold, the multimodal data is preprocessed based on a pre-built joint adversarial model, and the multimodal data is represented by fine-grained vectorization;

[0017] If the preprocessing importance of the multimodal data does not exceed a preset importance threshold, the multimodal data is preprocessed based on the data type of the multimodal data, and the multimodal data is represented by coarse vectorization;

[0018] Load multimodal data represented by fine vectorization and coarse vectorization, and concatenate the features of the multimodal data represented by fine vectorization and coarse vectorization based on a multi-Transformer architecture that combines self-attention and cross-attention data fusion technology to obtain high-dimensional fused features. Use PCA to reduce the dimensionality of the high-dimensional fused features, reduce the feature dimensions, and output a normalized low-dimensional feature vector;

[0019] Obtain a low-dimensional feature vector, and predict the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database based on the conditional random field (CRF). Determine whether the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database exceeds a preset probability threshold. If so, establish a mapping relationship between the low-dimensional feature vector and the storage layer, map the low-dimensional feature vector to the storage layer, and complete the creation of the shared database.

[0020] Preferably, the method for preprocessing multimodal data based on the pre-built joint adversarial model includes:

[0021] Pre-constructing a joint adversarial model for filtering and extracting features from multimodal data, wherein the joint adversarial model includes an input layer, a type discriminator, an adaptive filter, a joint adversarial network, and an output layer. The input layer is connected to the type discriminator, the type discriminator is connected to the adaptive filter, the adaptive filter is connected to the joint adversarial network, and the joint adversarial network is connected to the output layer. The type discriminator identifies multimodal data types based on a decision tree algorithm. The adaptive filter consists of multiple heterogeneous filter layers and a Kalman filter layer. The joint adversarial network includes an upsampling layer, a full-dimensional dynamic convolution module, a BILSTM layer, and a pooling layer.

[0022] Load multimodal data. The type discriminator identifies the data type of the multimodal data based on the decision tree algorithm and determines the data type of the multimodal data.

[0023] After obtaining multimodal data with data type identification, the adaptive filter performs collaborative filtering on the multimodal data based on the multi-heterogeneous filter layer and the Kalman filter layer, and combines the three-frame differential detection algorithm and the Kalman filter algorithm to mark and remove noise data in the multimodal data;

[0024] The joint adversarial network upsamples the filtered multimodal data and feeds the upsampling results back to the full-dimensional dynamic convolution module. The full-dimensional dynamic convolution module uses dynamic convolution to adjust the size and shape of the convolution kernel according to the data importance of the multimodal data. It uses the attention calculation of the data type, data attribute, data dimension, and data importance dimension to derive the attention weight of each dimension, and multiply the attention weight by the convolution kernel to obtain the dynamic weight of the multimodal data. The dynamic weight of the multimodal data is convolved with the upsampling result to generate an adaptive dynamic convolution.

[0025] After loading the adaptive dynamic convolution, the BILSTM layer performs forward and reverse LSTM operations on the adaptive dynamic convolution to obtain forward and reverse hidden state vectors. The forward and reverse hidden state vectors and the adaptive dynamic convolution are then merged to obtain a fine-grained vectorized representation of multimodal data.

[0026] Preferably, the preprocessing importance of the multimodal data is calculated by the following formula:

[0027]

[0028] Among them, A(x t ) represents the importance of preprocessing of multimodal data, σ(q avg ) is the standard deviation of type label, attribute label, dimension label, and importance label, q avg is the weighted average of type tag, attribute tag, dimension tag, and importance tag. are the input value and mean of multimodal data, q1, q2, q3, and q4 are the type mark, attribute mark, dimension mark, and importance mark of numerical quantization, respectively;

[0029] When predicting the probability that a low-dimensional feature vector corresponds to a storage layer in a shared database based on the conditional random field CRF, the probability that the low-dimensional feature vector corresponds to a storage layer in the shared database is calculated using the following formula:

[0030]

[0031] Among them, γ(D x ,C y ) represents the low-dimensional feature vector D x Corresponding to the storage layer C in the shared database y The probability of, C, D are the number of low-dimensional feature vectors and the number of storage layers, are the eigenvector matrix, covariance, mean of the low-dimensional eigenvector, sim(D x ,C y ) represents the low-dimensional feature vector D x Corresponding to the storage layer C in the shared database y The similarity of WC Represents storage layer C y The attribute weight matrix, exp[·] represents the Gaussian kernel function, D x ×C y Represents the low-dimensional feature vector D x With storage layer C y The adjacency similarity matrix, ω C Represents the length scale parameter of the Gaussian kernel function.

[0032] Preferably, the resource scheduling model uses the federated learning model as the initial model. The federated learning model consists of a distributed server, a central virtual server, and a communication scheduling layer. The distributed server is the distributed node of the federated learning model. The distributed server and the central virtual server interact with data through the transmission path. The distributed server consists of an input layer, an ALBERT module, and a Transformer encoder. The ALBERT module introduces a flow control strategy and a quality of service mechanism. The central virtual server includes a content distribution network architecture CDN, a homomorphic encryption layer, and a differential privacy layer. The communication scheduling layer includes an adaptive multi-source multi-path congestion control module, a BLEST scheduler, and a fully connected model. The fully connected model introduces an ant colony optimization algorithm, and the fully connected model includes Layer-1, Layer-2, and Layer-3. The number of nodes in Layer-1, Layer-2, and Layer-3 are 250, 150, and 300, respectively.

[0033] Preferably, the resource scheduling model training method includes:

[0034] Traverse the multimodal data in the distributed hash table, divide the multimodal data into training and test sets, and preset the initial parameters, iteration rounds, maximum iteration rounds, and loss function of the resource scheduling model;

[0035] Load the pre-built resource scheduling model, use the training set to pre-train the distributed server, and obtain the server parameters of the distributed server;

[0036] Based on the content delivery network architecture CDN, the server parameters of at least one group of distributed servers are imported. The central virtual server dynamically aggregates the server parameters based on the adaptive multi-source multi-path congestion control algorithm combined with the ant colony optimization algorithm, and calculates the server parameter path delay rate and congestion window rate;

[0037] The server parameters whose sum of path delay rate and congestion window rate is less than the preset cumulative threshold are retained, and the server parameters whose sum of path delay rate and congestion window rate is greater than or equal to the preset cumulative threshold are weighted and pruned;

[0038] Load the server parameters after weight pruning, set them as global parameters, and deploy the global parameters in distributed servers to obtain a converged resource scheduling model;

[0039] Obtain a test set, input the test set into a converged resource scheduling model, and dynamically deploy test data transmission tasks based on test set attributes and real-time transmission path information, and calculate the test set transmission response time.

[0040] Determine whether the test set transmission response time meets the preset time threshold. If the test set transmission response time meets the preset time threshold, output a converged resource scheduling model.

[0041] Preferably, the resource scheduling model dynamically deploys data transmission tasks based on target data attributes and real-time transmission path information, including:

[0042] Obtaining target data attributes, indexing at least one set of distributed hash tables based on the target data attributes, creating at least one set of transmission paths using an adaptive multi-source multi-path congestion control algorithm, employing a multi-path transmission protocol to poll and allocate the target data for the transmission paths, and identifying real-time transmission path information corresponding to the transmission paths, the real-time transmission path information including bandwidth, delay, and packet loss rate;

[0043] Taking the path with the lowest delay and the congestion window rate as constraints, at least one set of transmission path combinations is generated by combining the ant colony optimization algorithm.

[0044] The path delay rate and congestion window rate of the transmission path combination are calculated based on the adaptive multi-source multi-path congestion control algorithm, and the optimal transmission path combination is the one with the smallest sum of the path delay rate and the congestion window rate.

[0045] In response to the optimal transmission path combination, data transmission tasks are dynamically deployed based on the optimal transmission path combination.

[0046] On the other hand, the present invention also provides a big data resource service system based on multimodal data fusion, the big data resource service system based on multimodal data fusion comprising:

[0047] The database creation module is used to obtain multimodal data sequences collected by data sources in a distributed manner, preprocess the multimodal data sequences based on data fusion technology, obtain multimodal data represented by low-dimensional feature vectors, and create a shared database with the preprocessed low-dimensional feature vectors;

[0048] Distributed encryption module, which encrypts and distributes multimodal data in the shared database to the distributed hash table of distributed nodes based on peer-to-peer technology, and determines the data location of multimodal data in the distributed nodes through hash functions;

[0049] A resource access module, in response to a resource retrieval instruction for a subject, identifies the resource retrieval instruction based on a pre-built resource scheduling model and triggers a resource access control request;

[0050] A storage retrieval module, in response to a resource access control request, traverses a distributed hash table of at least one group of distributed nodes based on cosine similarity to extract target data to be scheduled and target data attributes;

[0051] The retrieval scheduling module dynamically deploys data transmission tasks based on target data attributes and real-time transmission path information, completes point-to-point encrypted transmission of target data to be scheduled based on data transmission tasks, and generates retrieval results in combination with the large language model interaction.

[0052] Preferably, the database creation module includes:

[0053] The data tagging unit is used to delete duplicate data and abnormal data in the multimodal data sequence, and to perform type tagging, attribute tagging, dimension tagging, and importance tagging on the multimodal data based on the data type, data attribute, data dimension, and data importance;

[0054] An importance calculation unit is used to obtain the marked multimodal data sequence and define the preprocessing importance of the multimodal data in the multimodal data sequence based on a probability density function combined with a weighted average algorithm;

[0055] An importance judgment unit, used to judge whether the preprocessing importance of the multimodal data exceeds a preset importance threshold;

[0056] The vector concatenation unit is used to load multimodal data represented by fine vectorization and coarse vectorization. Based on a multi-Transformer architecture that combines self-attention and cross-attention data fusion technology, the multimodal data features of fine vectorization and coarse vectorization are concatenated to obtain high-dimensional fused features. PCA is used to reduce the dimensionality of the high-dimensional fused features to reduce the feature dimension and output a normalized low-dimensional feature vector.

[0057] The data mapping storage unit is used to obtain low-dimensional feature vectors, predict the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database based on the conditional random field CRF, and determine whether the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database exceeds a preset probability threshold. If it exceeds the preset probability threshold, a mapping relationship is established between the low-dimensional feature vector and the storage layer, and the low-dimensional feature vector is mapped to the storage layer to complete the creation of the shared database.

[0058] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0059] In an embodiment of the present invention, point-to-point technology is used to encrypt and distribute multimodal data in a shared database to a distributed hash table of distributed nodes, and the data location is determined by a hash function, thereby achieving efficient distributed storage and access. In combination with distributed servers, central virtual servers, and a communication scheduling layer, a resource scheduling model is constructed that takes into account the distribution and transmission characteristics of multimodal data in a distributed environment, thereby achieving collaborative learning and optimization among distributed nodes, dynamically optimizing transmission paths, reducing data transmission delays and energy consumption, and improving the intelligence level of scheduling.

[0060] In an embodiment of the present invention, by judging the importance of preprocessing, the system can allocate more computing resources (such as time, storage, and computing power) to high-importance multimodal data, and simplify the processing of low-importance data, thereby avoiding the "one-size-fits-all" approach of using the same processing accuracy for all data, reducing unnecessary computing overhead, and improving the overall operating efficiency of the system. Especially in large-scale multimodal data processing scenarios, the optimization of resource allocation is particularly important, thereby ensuring that in dynamic scenarios, the system can quickly adjust the processing strategy and give priority to the most critical data in the current task. The high-importance data features of fine-grained processing provide depth information, enhancing the accuracy of the fusion results, and the low-importance data features of coarse-grained processing provide breadth information, enriching the coverage of the fusion results. This complementarity makes the fusion results more comprehensive and reliable.

[0061] In the embodiment of the present invention, when preprocessing multimodal data sequences based on data fusion technology, a differentiated processing strategy based on importance is used to optimize the allocation of computing resources and improve system efficiency. The multi-Transformer architecture is used to achieve efficient fusion and dimensionality reduction of multimodal data, thereby improving data availability. The introduction of probabilistic prediction and dynamic storage allocation mechanisms realizes intelligent and adaptable data processing.

[0062] In an embodiment of the present invention, a joint adversarial model for filtering and feature extraction of multimodal data is provided. The joint adversarial model combines multiple technologies such as decision trees, Kalman filtering, dynamic convolution, BILSTM, etc., and can handle complex multimodal data distributions. It is suitable for feature extraction of various data types (such as text, images, audio, etc.). The dynamic filtering, dynamic convolution and attention mechanism of the joint adversarial model have strong adaptability, can cope with the dynamic changes and complex distribution of multimodal data, and can adaptively extract deep-level features of multimodal data, thereby improving the accuracy and effect of feature extraction.

[0063] In an embodiment of the present invention, when training a resource scheduling model, redundant parameters that contribute little to the performance of the resource scheduling model are removed through weight pruning, thereby reducing the complexity of the resource scheduling model, making the pruned parameter set more streamlined, and reducing the computational overhead of subsequent training and inference. At the same time, key parameters are retained, avoiding the interference of redundant parameters on model optimization, and improving the accuracy and reliability of the model. In addition, the pruning operation reduces the computing resource consumption of distributed servers and central virtual servers, and improves the resource utilization of the system.

[0064] In an embodiment of the present invention, when the resource scheduling model dynamically deploys data transmission tasks based on target data attributes and real-time transmission path information, it significantly reduces the delay of data transmission by giving priority to the path combination with the lowest delay, meeting scenarios with high real-time requirements (such as video streaming, online games, etc.). By utilizing the multi-path transmission protocol and the ant colony optimization algorithm, data can be transmitted in parallel on multiple paths, improving bandwidth utilization and transmission speed. By constraining the congestion window rate, network congestion can be avoided, ensuring the stability and reliability of transmission. The ant colony optimization algorithm has a strong global search capability and can find a path combination close to the optimal solution, avoiding local optimal solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a schematic diagram of the implementation flow of the big data resource service method based on multimodal data fusion provided by the present invention.

[0066] Figure 2 The figure shows a flow chart of the implementation of the multimodal data sequence preprocessing method based on data fusion technology.

[0067] Figure 3 A schematic diagram of the implementation process of the multimodal data preprocessing method based on the pre-built joint adversarial model is shown.

[0068] Figure 4 A schematic diagram of the implementation process of the resource scheduling model training method is shown.

[0069] Figure 5 The present invention shows a schematic diagram of the implementation process of a resource scheduling model for dynamically deploying data transmission tasks based on target data attributes and real-time transmission path information.

[0070] Figure 6 A structural diagram of a big data resource service system based on multimodal data fusion is shown. DETAILED DESCRIPTION

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0072] Existing methods formulate scheduling plans solely by calculating server energy consumption and resource utilization. These methods lack adaptability to dynamic changes and fail to consider the data distribution and transmission characteristics in a distributed environment, reducing the overall performance and resource utilization of the system. To address these issues, we propose a big data resource service method and system based on multimodal data fusion. Briefly, the method first obtains multimodal data sequences collected by data sources in a distributed manner. These sequences are preprocessed using data fusion techniques, and a shared database is created using the preprocessed low-dimensional feature vectors. The multimodal data in the shared database is encrypted and distributed to the distributed hash tables of distributed nodes using a point-to-point approach. The location of the multimodal data in the distributed nodes is determined using a hash function. During the service phase, in response to resource retrieval instructions for a topic, a pre-built resource scheduling model identifies the resource retrieval instructions. The distributed hash tables of at least one set of distributed nodes are traversed based on cosine similarity to extract the target data to be scheduled and its attributes. The resource scheduling model dynamically deploys data transmission tasks based on the target data attributes and real-time transmission path information. Point-to-point encrypted transmission of the target data to be scheduled is completed based on the data transmission tasks, and retrieval results are generated interactively using a large language model. In an embodiment of the present invention, point-to-point technology is used to encrypt and distribute multimodal data in a shared database to a distributed hash table of distributed nodes, and the data location is determined by a hash function, thereby achieving efficient distributed storage and access. In combination with distributed servers, central virtual servers, and a communication scheduling layer, a resource scheduling model is constructed that takes into account the distribution and transmission characteristics of multimodal data in a distributed environment, thereby achieving collaborative learning and optimization among distributed nodes, dynamically optimizing transmission paths, reducing data transmission delays and energy consumption, and improving the intelligence level of scheduling.

[0073] The embodiment of the present invention provides a big data resource service method based on multimodal data fusion. Figure 1 The following is a schematic diagram of a big data resource service method based on multimodal data fusion, which specifically includes:

[0074] Step S10: Distributedly acquire multimodal data sequences collected by data sources, preprocess the multimodal data sequences based on data fusion technology to obtain multimodal data represented by low-dimensional feature vectors, and create a shared database using the preprocessed low-dimensional feature vectors;

[0075] Step S20: encrypting and distributing the multimodal data in the shared database to the distributed hash table of the distributed nodes based on peer-to-peer technology, and determining the data location of the multimodal data in the distributed nodes through a hash function;

[0076] It should be noted that when encrypting multimodal data in a shared database and distributing it to a distributed hash table of distributed nodes based on peer-to-peer technology, a symmetric encryption algorithm (such as AES) or an asymmetric encryption algorithm (such as RSA) can be selected to encrypt the multimodal data. The selected encryption algorithm is used to encrypt the multimodal data. For each piece of data, a unique cryptographic identifier (such as a hash value) is generated and stored along with the encrypted data. When constructing the distributed hash table, DHT algorithms can be selected, including Kademlia and Chord. The Kademlia algorithm organizes nodes based on the XOR distance of node IDs, offering efficient routing performance. The Chord algorithm distributes data through consistent hashing, ensuring uniform data distribution. A bootstrap node is selected, and new nodes join the network by communicating with it. The bootstrap node helps the new node establish a routing table and introduces it to the DHT network. For each piece of encrypted multimodal data, a hash value is calculated using a hash function (such as SHA-1). This hash value serves as the data's key. Based on the DHT algorithm, the data's hash value is mapped to the DHT's virtual node space. The node responsible for storing the data is found, and the encrypted multimodal data is sent to the corresponding node for storage. After receiving the data, the node stores it in a local database or file system.

[0077] In an embodiment of the present invention, a decentralized shared database can store data in a dispersed manner on multiple distributed nodes based on peer-to-peer technology, and each distributed node has a complete copy of the data. This approach significantly reduces the risk of single point failures. Even if a node fails, other nodes can continue to provide services to ensure data security. Through distributed hash tables (DHT) and encryption technology, data is always kept encrypted during transmission and storage, further enhancing data privacy protection. The decentralized shared database uses data redundancy storage to ensure data integrity even if some nodes are damaged. Distributed nodes ensure data consistency and integrity through consensus algorithms, giving the system higher transparency and trustworthiness, and distributed nodes based on peer-to-peer technology can dynamically deploy data transmission tasks based on real-time transmission path information, further improving the flexibility and efficiency of resource scheduling.

[0078] Step S30, in response to a resource retrieval instruction for the subject, identifying the resource retrieval instruction based on a pre-built resource scheduling model, and triggering a resource access control request;

[0079] In step S40, in response to the resource access control request, the distributed hash tables of at least one group of distributed nodes are traversed based on cosine similarity to extract the target data to be scheduled and its attributes. When extracting the target data to be scheduled, the feature vector of the data stored at each distributed node is calculated. The feature vector of the target data to be scheduled is also calculated using the cosine similarity formula. Starting from a node in the DHT, the node most similar to the target data is searched. Then, using the Kademlia node search algorithm, the K nodes closest to the target data are recursively found. Cosine similarity is calculated for the data stored at these K nodes, and the data with the highest similarity is selected as the target data to be scheduled.

[0080] In step S50, the resource scheduling model dynamically deploys data transmission tasks based on target data attributes and real-time transmission path information, completes point-to-point encrypted transmission of the target data to be scheduled based on the data transmission tasks, and generates retrieval results in combination with the large language model interaction.

[0081] In this embodiment, when generating retrieval results in combination with a large language model, the large language model may be ChatGLM-6B, Stanford Alpaca, or Nemotron-4 340B. Large language models (LLMs) are an important advancement in the field of natural language processing (NLP) in recent years. Through large-scale pre-training and fine-tuning, they can handle various natural language tasks such as text generation, dialogue systems, and machine translation.

[0082] In an embodiment of the present invention, point-to-point technology is used to encrypt and distribute multimodal data in a shared database to a distributed hash table of distributed nodes, and the data location is determined by a hash function, thereby achieving efficient distributed storage and access. In combination with distributed servers, central virtual servers, and a communication scheduling layer, a resource scheduling model is constructed that takes into account the distribution and transmission characteristics of multimodal data in a distributed environment, thereby achieving collaborative learning and optimization among distributed nodes, dynamically optimizing transmission paths, reducing data transmission delays and energy consumption, and improving the intelligence level of scheduling.

[0083] The embodiment of the present invention provides a method for preprocessing multimodal data sequences based on data fusion technology. Figure 2 The figure shows a flow chart of a method for preprocessing a multimodal data sequence based on data fusion technology. The method for preprocessing a multimodal data sequence based on data fusion technology specifically includes:

[0084] Step S101: Load a multimodal data sequence, delete duplicate data and abnormal data in the multimodal data sequence, and numerically quantify the multimodal data based on data type, data attribute, data dimension, and data importance, such as type labeling, attribute labeling, dimension labeling, and importance labeling, to obtain a labeled multimodal data sequence;

[0085] In an embodiment of the present invention, multimodal data sequences include but are not limited to text, images, audio, video, sensor data, and geospatial data. When type labeling, attribute labeling, dimension labeling, and importance labeling are performed on multimodal data based on data type, data attribute, data dimension, and data importance, the label values of data type, data attribute, data dimension, and data importance can be determined based on principal component analysis or voting. Through type labeling, attribute labeling, dimension labeling, and importance labeling, the characteristics of multimodal data are clarified, providing a clear structured foundation for subsequent preprocessing and fusion. Importance labeling can distinguish the criticality of data, provide a basis for importance-based preprocessing in subsequent steps, and avoid wasting resources on low-value data.

[0086] Step S102: obtaining a labeled multimodal data sequence, and defining the preprocessing importance of the multimodal data in the multimodal data sequence based on a probability density function combined with a weighted average algorithm;

[0087] It's important to note that combining a probability density function (PDF) and a weighted average algorithm allows for a more accurate assessment of the preprocessing importance of each multimodal data point. This approach comprehensively considers the data's distribution characteristics and weights, more accurately reflecting the data's actual importance. By quantifying importance, it prioritizes high-importance data and avoids overprocessing low-importance data, thereby improving overall preprocessing efficiency. Given the high adaptability of the probability density function and weighted average algorithm, which can dynamically adjust the importance calculation method based on data distribution, this approach is applicable to a variety of scenarios.

[0088] In the embodiment of the present invention, the preprocessing importance of the multimodal data is calculated by the following formula:

[0089]

[0090] Among them, A(x t ) represents the importance of preprocessing of multimodal data, σ(q avg ) is the standard deviation of type label, attribute label, dimension label, and importance label, q avg is the weighted average of type label, attribute label, dimension label and importance label, x t , are the input value and mean of the multimodal data respectively, q1, q2, q3, q4 are the type tag, attribute tag, dimension tag, and importance tag of numerical quantization respectively, among which the type tag, attribute tag, dimension tag, and importance tag of numerical quantization can be 1-10.

[0091] Step S103, determining whether the preprocessing importance of the multimodal data exceeds a preset importance threshold;

[0092] It should be noted that the present invention reduces unnecessary computational overhead and improves the overall operating efficiency of the system by distinguishing between high-importance and low-importance data. The importance threshold may be 0.3-0.5.

[0093] Step S104: if the preprocessing importance of the multimodal data exceeds a preset importance threshold, the multimodal data is preprocessed based on the pre-built joint adversarial model, and the multimodal data is represented by fine-grained vectorization;

[0094] In an embodiment of the present invention, if the preprocessing importance of multimodal data exceeds a preset importance threshold, the multimodal data is judged to be relatively important (the preprocessing importance of multimodal data can be greater than 0.3) and requires fine-grained vectorization processing. The pre-built joint adversarial model can extract deep-level features of multimodal data through adversarial training, providing richer information for subsequent fusion and scheduling. The joint adversarial model also has strong modeling capabilities and can handle complex multimodal data distributions, making it suitable for a variety of complex scenarios.

[0095] Step S105 , if the preprocessing importance of the multimodal data does not exceed a preset importance threshold, preprocess the multimodal data based on the data type of the multimodal data, and coarsely vectorize the multimodal data;

[0096] In an embodiment of the present invention, if the preprocessing importance of multimodal data does not exceed a preset importance threshold, the current multimodal data is judged to be of general importance, and mature existing technologies can be used to preprocess the multimodal data based on the data type of the multimodal data. For example, if the data type is text, the BERT model can be used for recognition. If the data type is picture or video, a convolutional neural network can be used for preprocessing, and high-importance data is processed in a fine-grained manner to ensure that its deep features are fully extracted. Low-importance data is processed in a coarse-grained manner to ensure that its basic features are retained. This differentiated processing strategy achieves a balance of data features and improves the comprehensiveness and consistency of the fusion results.

[0097] In an embodiment of the present invention, by judging the importance of preprocessing, the system can allocate more computing resources (such as time, storage, and computing power) to high-importance multimodal data, and simplify the processing of low-importance data, thereby avoiding the "one-size-fits-all" approach of using the same processing accuracy for all data, reducing unnecessary computing overhead, and improving the overall operating efficiency of the system. Especially in large-scale multimodal data processing scenarios, the optimization of resource allocation is particularly important, thereby ensuring that in dynamic scenarios, the system can quickly adjust the processing strategy and give priority to the most critical data in the current task. The high-importance data features of fine-grained processing provide depth information, enhancing the accuracy of the fusion results, and the low-importance data features of coarse-grained processing provide breadth information, enriching the coverage of the fusion results. This complementarity makes the fusion results more comprehensive and reliable.

[0098] Step S106: Load the multimodal data represented by fine vectorization and coarse vectorization, and concatenate the multimodal data features represented by fine vectorization and coarse vectorization based on a multi-Transformer architecture that combines self-attention and cross-attention data fusion technology to obtain high-dimensional fusion features. The multi-Transformer architecture combines self-attention and cross-attention mechanisms to efficiently fuse fine vectors and coarse vectors, capturing the complex relationship between multimodal data. PCA is used to reduce the dimensionality of the high-dimensional fusion features, reduce the feature dimensions, and output a normalized low-dimensional feature vector. The Transformer architecture has strong modeling capabilities and can adaptively learn the features of multimodal data, thereby improving the intelligent level of fusion.

[0099] Step S107, obtain a low-dimensional feature vector, predict the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database based on the conditional random field CRF, and determine whether the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database exceeds a preset probability threshold. If it exceeds the preset probability threshold, establish a mapping relationship between the low-dimensional feature vector and the storage layer, map the low-dimensional feature vector to the storage layer, and complete the creation of the shared database.

[0100] In this embodiment, when predicting the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database based on the conditional random field CRF, the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database is calculated by the following formula:

[0101]

[0102] Among them, γ(D x ,C y ) represents the low-dimensional feature vector D x Corresponding to the storage layer C in the shared database y The probability of, C, D are the number of low-dimensional feature vectors and the number of storage layers, are the eigenvector matrix, covariance, mean of the low-dimensional eigenvector, sim(D x ,C y ) represents the low-dimensional feature vector D x Corresponding to the storage layer C in the shared database y The similarity of W C Represents storage layer C y The attribute weight matrix, exp[·] represents the Gaussian kernel function, D x ×C y Represents the low-dimensional feature vector D x With storage layer C y The adjacency similarity matrix, ω C Represents the length scale parameter of the Gaussian kernel function.

[0103] In this embodiment of the present invention, a conditional random field (CRF) accurately predicts the storage probability of low-dimensional feature vectors based on contextual information, ensuring the rationality of data storage. By determining whether to store data based on a probability threshold, dynamic storage allocation is achieved, avoiding the waste of storage resources. Low-dimensional feature vectors are mapped to a storage layer, completing the creation of a shared database, facilitating subsequent data retrieval and management.

[0104] In the embodiment of the present invention, when preprocessing multimodal data sequences based on data fusion technology, a differentiated processing strategy based on importance is used to optimize the allocation of computing resources and improve system efficiency. The multi-Transformer architecture is used to achieve efficient fusion and dimensionality reduction of multimodal data, thereby improving data availability. The introduction of probabilistic prediction and dynamic storage allocation mechanisms realizes intelligent and adaptable data processing.

[0105] The embodiment of the present invention provides a method for preprocessing multimodal data based on a pre-built joint adversarial model. Figure 3 The figure shows a flow chart of a method for preprocessing multimodal data based on a pre-built joint adversarial model. The method for preprocessing multimodal data based on a pre-built joint adversarial model specifically includes:

[0106] Step S1041: Pre-constructing a joint adversarial model for filtering and extracting features from multimodal data, wherein the joint adversarial model includes an input layer, a type discriminator, an adaptive filter, a joint adversarial network, and an output layer. The input layer is connected to the type discriminator, the type discriminator is connected to the adaptive filter, the adaptive filter is connected to the joint adversarial network, and the joint adversarial network is connected to the output layer. The type discriminator identifies multimodal data types based on a decision tree algorithm. The adaptive filter is composed of multiple heterogeneous filter layers and a Kalman filter layer. The joint adversarial network includes an upsampling layer, a full-dimensional dynamic convolution module, a BILSTM layer, and a pooling layer.

[0107] Step S1042: Load multimodal data. The type discriminator identifies the data type of the multimodal data based on the decision tree algorithm to determine the data type of the multimodal data. The decision tree algorithm can classify data based on its characteristics (such as data type and attributes) and is applicable to a variety of complex multimodal data scenarios. The pre-classification by the type discriminator reduces the computational complexity of subsequent processing modules and improves overall processing efficiency.

[0108] Step S1043: Obtain multimodal data after data type identification. The adaptive filter performs collaborative filtering on the multimodal data based on the multi-heterogeneous filter layer and the Kalman filter layer. The adaptive filter combines the three-frame difference detection algorithm and the Kalman filter algorithm to mark and remove noise data from the multimodal data. The combination of the three-frame difference detection algorithm and the Kalman filter algorithm can effectively identify and remove noise data from the multimodal data, thereby improving the purity of the data.

[0109] Step S1044: The joint adversarial network upsamples the filtered multimodal data and feeds the upsampling result back to the full-dimensional dynamic convolution module. The full-dimensional dynamic convolution module uses dynamic convolution to adjust the size and shape of the convolution kernel according to the data importance of the multimodal data. The full-dimensional dynamic convolution module uses attention calculation based on the data type, data attribute, data dimension, and data importance dimension to obtain the attention weight of each dimension, and multiplies the attention weight by the convolution kernel to obtain the dynamic weight of the multimodal data. The dynamic weight of the multimodal data is convolved with the upsampling result to generate an adaptive dynamic convolution.

[0110] In step S1045, the adaptive dynamic convolution is loaded, and the BILSTM layer performs forward and reverse LSTM operations on the adaptive dynamic convolution to obtain forward and reverse hidden state vectors. The forward and reverse hidden state vectors and the adaptive dynamic convolution are merged to obtain a fine-grained vector representation of multimodal data. The combination of forward and reverse LSTM operations enables the model to simultaneously capture the forward and backward dependencies of the data, thereby improving the comprehensiveness of feature extraction.

[0111] In this embodiment, a joint adversarial model for filtering and feature extraction of multimodal data is provided. The joint adversarial model combines multiple technologies such as decision trees, Kalman filtering, dynamic convolution, BILSTM, etc., and can handle complex multimodal data distributions. It is suitable for feature extraction of various data types (such as text, images, audio, etc.). The dynamic filtering, dynamic convolution and attention mechanism of the joint adversarial model have strong adaptability, can cope with the dynamic changes and complex distribution of multimodal data, and can adaptively extract deep features of multimodal data, thereby improving the accuracy and effect of feature extraction.

[0112] The embodiment of the present invention provides a resource scheduling model training method, Figure 4The following is a schematic diagram of a resource scheduling model training method implementation process, wherein the resource scheduling model training method specifically includes:

[0113] Step S201: traverse the multimodal data in the distributed hash table, divide the multimodal data into a training set and a test set, and preset the initial parameters, iteration rounds, maximum iteration rounds, and loss function of the resource scheduling model;

[0114] In an embodiment of the present invention, the ratio of the training set to the test set can be 5:1, the number of iterations can be 50-200 epochs, the maximum number of iterations can be 250, the loss function can be a cross-entropy loss function, and the initial parameter setting can be Xavier initialization or He initialization. The initial parameters include but are not limited to learning rate, batch size, and loss function weight.

[0115] Step S202: Load the pre-built resource scheduling model and pre-train the distributed server using the training set to obtain the server parameters of the distributed server. By pre-training on the distributed server, the distributed computing resources are fully utilized and the training efficiency is improved. The pre-training can generate optimized server parameters for the distributed server, laying the foundation for subsequent global parameter aggregation. The distributed server consists of an input layer, an ALBERT module, and a Transformer encoder. Combined with the flow control strategy and service quality mechanism, it can better capture the characteristics of multimodal data and improve the expression ability of the model.

[0116] Step S203: Import server parameters of at least one group of distributed servers based on the content delivery network architecture CDN. The central virtual server dynamically aggregates the server parameters based on the adaptive multi-source multi-path congestion control algorithm combined with the ant colony optimization algorithm, and calculates the server parameter path delay rate and congestion window rate. The adaptive multi-source multi-path congestion control algorithm can adjust parameters in real time to adapt to network congestion, improving the robustness and reliability of the system. The introduction of the ant colony optimization algorithm makes the parameter aggregation process more intelligent, can quickly find the optimal solution, and reduce computational complexity. The CDN-based architecture design can effectively reduce data transmission delay and improve the overall performance of the system.

[0117] Step S204: retaining server parameters whose sum of path delay rate and congestion window rate is less than a preset cumulative threshold, and performing weight pruning on server parameters whose sum of path delay rate and congestion window rate is greater than or equal to the preset cumulative threshold;

[0118] It should be noted that the preset accumulation threshold can be 0.1-0.15. The path delay rate refers to the ratio of the data transmission delay from the source node to the destination node to the ideal transmission delay. It is generally used to measure the transmission efficiency of a path. The transmission delay under ideal conditions (no congestion, no packet loss, and optimal path) can usually be estimated based on network topology and historical data. For each transmission path, the node delay and link delay are accumulated to obtain the actual path delay. The congestion window rate refers to the ratio of the congestion window size of the current transmission path to the maximum congestion window size. It is used to measure the degree of network congestion. The ant colony optimization algorithm can be used to optimize the path selection and parameter aggregation process. The path delay rate and congestion window rate of each path are used as the initial pheromone values of the ants. The ants select a path based on the combined performance indicators of the path delay rate and congestion window rate. The better the performance of the path, the higher the pheromone concentration. After multiple rounds of iteration, the ant colony algorithm can converge to the optimal path, ensuring the optimal balance between the path delay rate and the congestion window rate.

[0119] In an embodiment of the present invention, when training a resource scheduling model, redundant parameters that contribute little to the performance of the resource scheduling model are removed through weight pruning, thereby reducing the complexity of the resource scheduling model, making the pruned parameter set more streamlined, and reducing the computational overhead of subsequent training and inference. At the same time, key parameters are retained, avoiding the interference of redundant parameters on model optimization, and improving the accuracy and reliability of the model. In addition, the pruning operation reduces the computing resource consumption of distributed servers and central virtual servers, and improves the resource utilization of the system.

[0120] Step S205: Load the server parameters after weight pruning, set the server parameters after weight pruning as global parameters, and deploy the global parameters in the distributed servers to obtain a converged resource scheduling model;

[0121] Step S206: Obtain a test set, input the test set into a converged resource scheduling model, and dynamically deploy test data transmission tasks based on test set attributes and real-time transmission path information, and calculate the test set transmission response time.

[0122] Step S207: determining whether the test set transmission response time meets a preset time threshold, which may be 50-150ms;

[0123] Step S208: If the test set transmission response time meets the preset time threshold, a converged resource scheduling model is output.

[0124] Returning to step S202, if the test set transmission response time does not meet the preset time threshold, the resource scheduling model continues to be iteratively trained.

[0125] In this embodiment, the resource scheduling model uses the federated learning model as the initial model. Federated learning avoids the leakage of original data by training the model locally and exchanging only model parameters, effectively protects user privacy, reduces the possibility of data being eavesdropped or tampered with during transmission, and improves overall data security. The federated learning model consists of a distributed server, a central virtual server, and a communication scheduling layer. The distributed server is used as the distributed node of the federated learning model, and the computing power of edge devices (such as mobile phones and IoT devices) is used to reduce the load of the central server and improve the scalability of the system. The distributed server consists of an input layer, an ALBERT module, and a Transformer encoder. It can efficiently process multimodal data and transmit model parameters instead of large-scale original data, which significantly reduces communication overhead and storage costs. The distributed server, the central virtual server, and the communication scheduling layer are connected. Data is exchanged between virtual servers through transmission paths. The distributed server consists of an input layer, an ALBERT module, and a Transformer encoder. The ALBERT module introduces flow control strategies and quality of service mechanisms. Through the flow control strategy, data packet traffic under specified conditions can be counted, released, blocked, and other operations can be performed to optimize the use of network resources. The central virtual server includes a content distribution network architecture CDN, a homomorphic encryption layer, and a differential privacy layer. The communication scheduling layer includes an adaptive multi-source multi-path congestion control module, a BLEST scheduler, and a fully connected model. The fully connected model introduces an ant colony optimization algorithm, and the fully connected model includes Layer-1, Layer-2, and Layer-3. The number of nodes in Layer-1, Layer-2, and Layer-3 are 250, 150, and 300, respectively.

[0126] The embodiment of the present invention provides a method for dynamically deploying data transmission tasks based on target data attributes and real-time transmission path information in a resource scheduling model. Figure 5 The following is a schematic diagram of a method for dynamically deploying data transmission tasks based on target data attributes and real-time transmission path information using a resource scheduling model. The method specifically includes:

[0127] Step S301: Obtain target data attributes, index at least one set of distributed hash tables based on the target data attributes, and index the distributed hash tables by target data attributes (such as data size, priority, type, etc.). This allows for rapid location of suitable transmission paths, improving the efficiency and accuracy of path selection. An adaptive multi-source multi-path congestion control algorithm creates at least one set of transmission paths, employs a multi-path transmission protocol to poll and allocate target data for the transmission paths, and identifies real-time transmission path information corresponding to the transmission paths. The real-time transmission path information includes bandwidth, delay, and packet loss rate. Transmission path information such as bandwidth, delay, and packet loss rate is obtained in real time, which can dynamically reflect network status and ensure that transmission path selection is always based on the latest network conditions. The adaptive multi-source multi-path congestion control algorithm can dynamically adjust transmission paths according to network congestion conditions to avoid overloading a single path and improve system robustness.

[0128] In step S302, at least one transmission path combination is generated using the path with the lowest latency and the congestion window rate as constraints, in combination with the ant colony optimization algorithm. The ant colony optimization algorithm is an optimization algorithm based on swarm intelligence that can simulate the foraging behavior of ants and gradually find the optimal path combination. It is suitable for solving complex multi-path selection problems. The ant colony optimization algorithm has strong global search capabilities and can avoid falling into local optimal solutions, ensuring that the generated path combination is close to the global optimal solution.

[0129] Step S303: Calculate the path delay rate and congestion window rate of each transmission path combination based on the adaptive multi-source multi-path congestion control algorithm. The optimal transmission path combination is the one with the smallest sum of the path delay rate and the congestion window rate, which is less than 0.01%. By comprehensively considering the path delay rate and the congestion window rate, the overall performance of the transmission path can be comprehensively evaluated, avoiding the bias caused by a single indicator. Using the weighted sum of the path delay rate and the congestion window rate as the decision basis is simple and easy to implement, and can quickly find the optimal path combination.

[0130] Step S304 : In response to the optimal transmission path combination, dynamically deploy data transmission tasks based on the optimal transmission path combination.

[0131] In an embodiment of the present invention, when the resource scheduling model dynamically deploys data transmission tasks based on target data attributes and real-time transmission path information, it significantly reduces the delay of data transmission by giving priority to the path combination with the lowest delay, meeting scenarios with high real-time requirements (such as video streaming, online games, etc.). By utilizing the multi-path transmission protocol and the ant colony optimization algorithm, data can be transmitted in parallel on multiple paths, improving bandwidth utilization and transmission speed. By constraining the congestion window rate, network congestion can be avoided, ensuring the stability and reliability of transmission. The ant colony optimization algorithm has a strong global search capability and can find a path combination close to the optimal solution, avoiding local optimal solutions.

[0132] The embodiment of the present invention provides a big data resource service system based on multimodal data fusion. Figure 6 The structure diagram of a big data resource service system based on multimodal data fusion is shown. The big data resource service system based on multimodal data fusion specifically includes:

[0133] The database creation module 100 is used to obtain multimodal data sequences collected by data sources in a distributed manner, preprocess the multimodal data sequences based on data fusion technology to obtain multimodal data represented by low-dimensional feature vectors, and create a shared database with the preprocessed low-dimensional feature vectors;

[0134] The distributed encryption module 200 encrypts and distributes the multimodal data in the shared database to the distributed hash table of the distributed nodes based on the peer-to-peer technology, and determines the data location of the multimodal data in the distributed nodes through the hash function;

[0135] The resource access module 300, in response to a resource retrieval instruction for a subject, identifies the resource retrieval instruction based on a pre-built resource scheduling model and triggers a resource access control request;

[0136] The storage retrieval module 400 , in response to a resource access control request, traverses a distributed hash table of at least one group of distributed nodes based on cosine similarity to extract target data to be scheduled and target data attributes;

[0137] The retrieval scheduling module 500 dynamically deploys data transmission tasks based on target data attributes and real-time transmission path information, completes point-to-point encrypted transmission of target data to be scheduled based on the data transmission tasks, and generates retrieval results in combination with the large language model interaction.

[0138] In this embodiment, the database creation module 100 includes:

[0139] The data labeling unit 110 is used to delete duplicate data and abnormal data in the multimodal data sequence, and to perform type labeling, attribute labeling, dimension labeling, and importance labeling on the multimodal data based on data type, data attribute, data dimension, and data importance;

[0140] The importance calculation unit 120 is used to obtain the marked multimodal data sequence and define the preprocessing importance of the multimodal data in the multimodal data sequence based on the probability density function combined with the weighted average algorithm;

[0141] The importance determination unit 130 is used to determine whether the preprocessing importance of the multimodal data exceeds a preset importance threshold;

[0142] The vector concatenation unit 140 is used to load multimodal data represented by fine vectorization and coarse vectorization, concatenate the features of the multimodal data represented by fine vectorization and coarse vectorization based on a multi-Transformer architecture that combines self-attention and cross-attention data fusion technology to obtain high-dimensional fused features, perform dimensionality reduction processing on the high-dimensional fused features using PCA to reduce the feature dimensionality, and output a normalized low-dimensional feature vector;

[0143] The data mapping storage unit 150 is used to obtain a low-dimensional feature vector, predict the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database based on the conditional random field CRF, and determine whether the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database exceeds a preset probability threshold. If it exceeds the preset probability threshold, a mapping relationship is established between the low-dimensional feature vector and the storage layer, and the low-dimensional feature vector is mapped to the storage layer to complete the creation of the shared database.

[0144] In summary, the present invention provides a big data resource service method and system based on multimodal data fusion. In an embodiment of the present invention, point-to-point technology is used to encrypt and distribute the multimodal data in the shared database to the distributed hash table of the distributed nodes, and the data location is determined by the hash function, thereby achieving efficient distributed storage and access. In addition, a resource scheduling model is constructed that takes into account the distribution and transmission characteristics of multimodal data in a distributed environment, combining distributed servers, central virtual servers and communication scheduling layers, realizing collaborative learning and optimization among distributed nodes, dynamically optimizing the transmission path, reducing data transmission delays and energy consumption, and improving the intelligence level of scheduling.

[0145] It should be noted that for the aforementioned embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0146] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the scope of protection of the invention. Obviously, the embodiments described are only some embodiments of the present invention, rather than all embodiments. Based on these embodiments, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in this field can still combine, add, delete or make other adjustments to the features in the various embodiments of the present invention according to the circumstances without conflict, without making creative work, so as to obtain different other technical solutions that do not deviate from the concept of the present invention in essence, and these technical solutions also fall within the scope of protection of the present invention.

Claims

1. A big data resource service method based on multimodal data fusion, characterized in that: include: Distributed acquisition of multimodal data sequences collected by data sources, preprocessing of multimodal data sequences based on data fusion technology to obtain multimodal data represented by low-dimensional feature vectors, and creation of a shared database using the preprocessed low-dimensional feature vectors; Based on peer-to-peer technology, the multimodal data in the shared database is encrypted and distributed to the distributed hash table of the distributed nodes, and the data location of the multimodal data in the distributed nodes is determined by the hash function; In response to a resource retrieval instruction for a subject, identifying the resource retrieval instruction based on a pre-built resource scheduling model, and triggering a resource access control request; In response to a resource access control request, traverse a distributed hash table of at least one group of distributed nodes based on cosine similarity to extract target data to be scheduled and target data attributes; The resource scheduling model dynamically deploys data transmission tasks based on target data attributes and real-time transmission path information, completes point-to-point encrypted transmission of target data to be scheduled based on data transmission tasks, and generates retrieval results in combination with the large language model interaction.

2. The big data resource service method based on multimodal data fusion according to claim 1, characterized in that: The method for preprocessing a multimodal data sequence based on data fusion technology includes: Load multimodal data sequences, delete duplicate data and abnormal data in the multimodal data sequences, and perform numerical quantification of type labeling, attribute labeling, dimension labeling, and importance labeling on the multimodal data based on data type, data attribute, data dimension, and data importance to obtain the labeled multimodal data sequences; Obtain the labeled multimodal data sequence, and define the preprocessing importance of the multimodal data in the multimodal data sequence based on the probability density function combined with the weighted average algorithm; Determine whether the preprocessing importance of the multimodal data exceeds a preset importance threshold; If the preprocessing importance of the multimodal data exceeds a preset importance threshold, the multimodal data is preprocessed based on a pre-built joint adversarial model, and the multimodal data is represented by fine-grained vectorization; If the preprocessing importance of the multimodal data does not exceed a preset importance threshold, the multimodal data is preprocessed based on the data type of the multimodal data, and the multimodal data is represented by coarse vectorization; Load multimodal data represented by fine vectorization and coarse vectorization, and concatenate the features of the multimodal data represented by fine vectorization and coarse vectorization based on a multi-Transformer architecture that combines self-attention and cross-attention data fusion technology to obtain high-dimensional fused features. Use PCA to reduce the dimensionality of the high-dimensional fused features, reduce the feature dimensions, and output a normalized low-dimensional feature vector; Obtain a low-dimensional feature vector, and predict the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database based on the conditional random field (CRF). Determine whether the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database exceeds a preset probability threshold. If so, establish a mapping relationship between the low-dimensional feature vector and the storage layer, map the low-dimensional feature vector to the storage layer, and complete the creation of the shared database.

3. The big data resource service method based on multimodal data fusion according to claim 2, characterized in that: The method for preprocessing multimodal data based on the pre-built joint adversarial model includes: Pre-constructing a joint adversarial model for filtering and extracting features from multimodal data, wherein the joint adversarial model includes an input layer, a type discriminator, an adaptive filter, a joint adversarial network, and an output layer. The input layer is connected to the type discriminator, the type discriminator is connected to the adaptive filter, the adaptive filter is connected to the joint adversarial network, and the joint adversarial network is connected to the output layer. The type discriminator identifies multimodal data types based on a decision tree algorithm. The adaptive filter consists of multiple heterogeneous filter layers and a Kalman filter layer. The joint adversarial network includes an upsampling layer, a full-dimensional dynamic convolution module, a BILSTM layer, and a pooling layer. Load multimodal data. The type discriminator identifies the data type of the multimodal data based on the decision tree algorithm and determines the data type of the multimodal data. After obtaining multimodal data with data type identification, the adaptive filter performs collaborative filtering on the multimodal data based on the multi-heterogeneous filter layer and the Kalman filter layer, and combines the three-frame differential detection algorithm and the Kalman filter algorithm to mark and remove noise data in the multimodal data; The joint adversarial network upsamples the filtered multimodal data and feeds the upsampling results back to the full-dimensional dynamic convolution module. The full-dimensional dynamic convolution module uses dynamic convolution to adjust the size and shape of the convolution kernel according to the data importance of the multimodal data. It uses the attention calculation of the data type, data attribute, data dimension, and data importance dimension to derive the attention weight of each dimension, and multiply the attention weight by the convolution kernel to obtain the dynamic weight of the multimodal data. The dynamic weight of the multimodal data is convolved with the upsampling result to generate an adaptive dynamic convolution. After loading the adaptive dynamic convolution, the BILSTM layer performs forward and reverse LSTM operations on the adaptive dynamic convolution to obtain forward and reverse hidden state vectors. The forward and reverse hidden state vectors and the adaptive dynamic convolution are then merged to obtain a fine-grained vectorized representation of multimodal data.

4. The big data resource service method based on multimodal data fusion according to claim 3, characterized in that: The preprocessing importance of the multimodal data is calculated by the following formula: Among them, A(x t ) represents the importance of preprocessing of multimodal data, σ(q avg ) is the standard deviation of type label, attribute label, dimension label, and importance label, q avg is the weighted average of type label, attribute label, dimension label and importance label, x t , are the input value and mean of multimodal data, q1, q2, q3, and q4 are the type mark, attribute mark, dimension mark, and importance mark of numerical quantization, respectively; When predicting the probability that a low-dimensional feature vector corresponds to a storage layer in a shared database based on the conditional random field CRF, the probability that the low-dimensional feature vector corresponds to a storage layer in the shared database is calculated using the following formula: Among them, γ(D x ,C y ) represents the low-dimensional feature vector D x Corresponding to the storage layer C in the shared database y The probability of C, D are the number of low-dimensional feature vectors and the number of storage layers, respectively. D ,∑, are the eigenvector matrix, covariance, mean of the low-dimensional eigenvector, sim(D x ,C y ) represents the low-dimensional feature vector D x Corresponding to the storage layer C in the shared database y The similarity of W C Represents storage layer C y The attribute weight matrix, exp[·] represents the Gaussian kernel function, D x ×C y Represents the low-dimensional feature vector D x With storage layer C y The adjacency similarity matrix, ω C Represents the length scale parameter of the Gaussian kernel function.

5. The big data resource service method based on multimodal data fusion according to claim 1, characterized in that: The resource scheduling model uses the federated learning model as the initial model. The federated learning model consists of a distributed server, a central virtual server, and a communication scheduling layer. The distributed server is the distributed node of the federated learning model. The distributed server and the central virtual server interact with each other through a transmission path. The distributed server consists of an input layer, an ALBERT module, and a Transformer encoder. The ALBERT module introduces a flow control strategy and a service quality mechanism. The central virtual server includes a content distribution network architecture CDN, a homomorphic encryption layer, and a differential privacy layer. The communication scheduling layer includes an adaptive multi-source multi-path congestion control module, a BLEST scheduler, and a fully connected model. The fully connected model introduces an ant colony optimization algorithm, and the fully connected model includes Layer-1, Layer-2, and Layer-3. The number of nodes in Layer-1, Layer-2, and Layer-3 are 250, 150, and 300, respectively.

6. The big data resource service method based on multimodal data fusion according to claim 5, characterized in that: The resource scheduling model training method includes: Traverse the multimodal data in the distributed hash table, divide the multimodal data into training and test sets, and preset the initial parameters, iteration rounds, maximum iteration rounds, and loss function of the resource scheduling model; Load the pre-built resource scheduling model, use the training set to pre-train the distributed server, and obtain the server parameters of the distributed server; Based on the content delivery network architecture CDN, the server parameters of at least one group of distributed servers are imported. The central virtual server dynamically aggregates the server parameters based on the adaptive multi-source multi-path congestion control algorithm combined with the ant colony optimization algorithm, and calculates the server parameter path delay rate and congestion window rate; The server parameters whose sum of path delay rate and congestion window rate is less than the preset cumulative threshold are retained, and the server parameters whose sum of path delay rate and congestion window rate is greater than or equal to the preset cumulative threshold are weighted and pruned; Load the server parameters after weight pruning, set them as global parameters, and deploy the global parameters in distributed servers to obtain a converged resource scheduling model; Obtain a test set, input the test set into a converged resource scheduling model, and dynamically deploy test data transmission tasks based on test set attributes and real-time transmission path information, and calculate the test set transmission response time. Determine whether the test set transmission response time meets the preset time threshold. If the test set transmission response time meets the preset time threshold, output a converged resource scheduling model.

7. The big data resource service method based on multimodal data fusion according to claim 5, characterized in that: The resource scheduling model dynamically deploys data transmission tasks based on target data attributes and real-time transmission path information, including: Obtaining target data attributes, indexing at least one set of distributed hash tables based on the target data attributes, creating at least one set of transmission paths using an adaptive multi-source multi-path congestion control algorithm, employing a multi-path transmission protocol to poll and allocate the target data for the transmission paths, and identifying real-time transmission path information corresponding to the transmission paths, the real-time transmission path information including bandwidth, delay, and packet loss rate; Taking the path with the lowest delay and the congestion window rate as constraints, at least one set of transmission path combinations is generated by combining the ant colony optimization algorithm.

8. The big data resource service method based on multimodal data fusion according to claim 7, characterized in that: The method for dynamically deploying data transmission tasks based on target data attributes and real-time transmission path information in the resource scheduling model further includes: The path delay rate and congestion window rate of the transmission path combination are calculated based on the adaptive multi-source multi-path congestion control algorithm, and the optimal transmission path combination is the one with the smallest sum of the path delay rate and the congestion window rate. In response to the optimal transmission path combination, data transmission tasks are dynamically deployed based on the optimal transmission path combination.

9. A big data resource service system based on multimodal data fusion, for implementing the big data resource service method based on multimodal data fusion according to any one of claims 1 to 8, characterized in that: The big data resource service system based on multimodal data fusion includes: The database creation module is used to obtain multimodal data sequences collected by data sources in a distributed manner, preprocess the multimodal data sequences based on data fusion technology, obtain multimodal data represented by low-dimensional feature vectors, and create a shared database with the preprocessed low-dimensional feature vectors; Distributed encryption module, which encrypts and distributes multimodal data in the shared database to the distributed hash table of distributed nodes based on peer-to-peer technology, and determines the data location of multimodal data in the distributed nodes through hash functions; A resource access module, in response to a resource retrieval instruction for a subject, identifies the resource retrieval instruction based on a pre-built resource scheduling model and triggers a resource access control request; A storage retrieval module, in response to a resource access control request, traverses a distributed hash table of at least one group of distributed nodes based on cosine similarity to extract target data to be scheduled and target data attributes; The retrieval scheduling module dynamically deploys data transmission tasks based on target data attributes and real-time transmission path information, completes point-to-point encrypted transmission of target data to be scheduled based on data transmission tasks, and generates retrieval results in combination with the large language model interaction.

10. The big data resource service system based on multimodal data fusion according to claim 9, characterized in that: The database creation module includes: The data tagging unit is used to delete duplicate data and abnormal data in the multimodal data sequence, and to perform type tagging, attribute tagging, dimension tagging, and importance tagging on the multimodal data based on the data type, data attribute, data dimension, and data importance; An importance calculation unit is used to obtain the marked multimodal data sequence and define the preprocessing importance of the multimodal data in the multimodal data sequence based on a probability density function combined with a weighted average algorithm; An importance judgment unit, used to judge whether the preprocessing importance of the multimodal data exceeds a preset importance threshold; The vector concatenation unit is used to load multimodal data represented by fine vectorization and coarse vectorization. Based on a multi-Transformer architecture that combines self-attention and cross-attention data fusion technology, the multimodal data features of fine vectorization and coarse vectorization are concatenated to obtain high-dimensional fused features. PCA is used to reduce the dimensionality of the high-dimensional fused features to reduce the feature dimension and output a normalized low-dimensional feature vector. The data mapping storage unit is used to obtain low-dimensional feature vectors, predict the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database based on the conditional random field CRF, and determine whether the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database exceeds a preset probability threshold. If it exceeds the preset probability threshold, a mapping relationship is established between the low-dimensional feature vector and the storage layer, and the low-dimensional feature vector is mapped to the storage layer to complete the creation of the shared database.

Images