Big data resource service method and system based on multi-modal data fusion

Abstract

The application discloses a big data resource service method and system based on multi-modal data fusion, belongs to the technical field of big data processing, and solves the problem that the existing method only formulates a scheduling scheme by calculating the energy consumption value and resource utilization of a server, has poor adaptability to dynamic changes, and reduces the overall performance and resource utilization of the system, the method comprises the following steps: preprocessing a multi-modal data sequence, creating a shared database, encrypting and distributing the multi-modal data into a distributed hash table of distributed nodes, identifying a resource retrieval instruction based on a resource scheduling model, and dynamically deploying a data transmission task based on target data attributes and real-time transmission path information; in the application, a resource scheduling model considering the multi-modal data distribution and transmission characteristics in a distributed environment is constructed, collaborative learning and optimization among the distributed nodes are realized, the transmission path is dynamically optimized, the delay and energy consumption of data transmission are reduced, and the intelligent level of scheduling is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of big data processing, and specifically relates to a big data resource service method and system based on multi-modal data fusion. BACKGROUND

[0002] With the rapid development of information technology, big data has been widely applied in various fields. However, the scheduling and optimal allocation of big data resources face many challenges. Traditional resource scheduling methods often fail to meet the complex and changing needs of big data processing, especially in the context of multi-modal data fusion, where these problems are more prominent.

[0003] Multi-modal data fusion refers to the integration and analysis of data from different modalities (such as text, images, video, audio, etc.) to extract more comprehensive and in-depth information. In practical applications, multi-modal data fusion faces problems such as diverse data sources, complex data formats, and large data volumes. Traditional resource scheduling methods often allocate resources based on the characteristics of single-modal data, making it difficult to adapt to the complex needs of multi-modal data fusion. For example, when processing large-scale image and text data, allocating resources based only on data volume or the number of computing tasks may result in insufficient resource support for processing tasks of certain modalities, thereby affecting overall processing efficiency.

[0004] Chinese patent CN113204429B discloses a resource scheduling method and system for a data center, a scheduling device, and a medium, which includes obtaining energy consumption data of each server in the data center; wherein the energy consumption data includes resource utilization rate of the server and performance data of each component of the server; calculating the energy consumption value of each server according to the energy consumption data and a preset energy consumption calculation model; and using a genetic algorithm to calculate the optimal resource scheduling scheme according to the resource utilization rate and energy consumption value of each server, and performing scheduling operation or shutdown operation on each server in the data center according to the optimal resource scheduling scheme. However, the existing method only formulates a scheduling scheme by calculating the energy consumption value and resource utilization rate of the server, which has poor adaptability to dynamic changes and does not consider data distribution and transmission characteristics in a distributed environment, thereby reducing the overall performance and resource utilization rate of the system. To address the above problems, we propose a big data resource service method and system based on multi-modal data fusion. SUMMARY

[0005] The present application aims to address the shortcomings of the prior art by providing a big data resource service method and system based on multi-modal data fusion, which solves the problem of poor adaptability to dynamic changes and does not consider data distribution and transmission characteristics in a distributed environment, thereby reducing the overall performance and resource utilization rate of the system.

[0006] The application is implemented based on a big data resource service method based on multi-modal data fusion, and the big data resource service method based on multi-modal data fusion comprises the following steps.

[0007] Distributedly acquire multi-modal data sequences collected by a data source, pre-process the multi-modal data sequences based on a data fusion technology, obtain multi-modal data represented by low-dimensional feature vectors, and create a shared database based on the pre-processed low-dimensional feature vectors.

[0008] Based on a point-to-point technology, the multi-modal data in the shared database is encrypted and distributed to a distributed hash table of distributed nodes, and the position of the multi-modal data in the distributed nodes is determined through a hash function.

[0009] In response to a resource retrieval instruction for a subject, the resource retrieval instruction is identified based on a pre-constructed resource scheduling model, and a resource access control request is triggered.

[0010] In response to the resource access control request, the distributed hash table of at least one group of distributed nodes is traversed based on cosine similarity, and target data to be scheduled and target data attributes are extracted.

[0011] The resource scheduling model dynamically deploys a data transmission task based on the target data attributes and real-time transmission path information, performs point-to-point encrypted transmission of the target data to be scheduled based on the data transmission task, and generates a retrieval result in combination with a large language model interaction.

[0012] Preferably, the method for pre-processing the multi-modal data sequences based on the data fusion technology comprises the following steps.

[0013] Load the multi-modal data sequences, delete repeated data and abnormal data in the multi-modal data sequences, and perform numerical quantization type marking, attribute marking, dimension marking, and importance marking on the multi-modal data based on data types, data attributes, data dimensions, and data importance, to obtain the marked multi-modal data sequences.

[0014] Obtain the marked multi-modal data sequences, and define the pre-processing importance of the multi-modal data in the multi-modal data sequences based on a probability density function in combination with a weighted average algorithm.

[0015] Determine whether the pre-processing importance of the multi-modal data exceeds a preset importance threshold.

[0016] If the pre-processing importance of the multi-modal data exceeds the preset importance threshold, pre-process the multi-modal data based on a pre-constructed joint adversarial model, and represent the multi-modal data by a fine-dimensional vector.

[0017] If the importance of the pre-processing of the multi-modal data does not exceed the preset importance threshold, the multi-modal data is pre-processed based on the data type of the multi-modal data, and the multi-modal data is represented by coarse vectorization;

[0018] The multi-modal data represented by the fine vectorization and the coarse vectorization is loaded, the multi-modal data represented by the fine vectorization and the coarse vectorization is spliced based on a multi-Transformer architecture combining self-attention and cross-attention data fusion technology, high-dimensional fusion features are obtained, the high-dimensional fusion features are processed by PCA for dimension reduction to reduce the feature dimension, and a normalized low-dimensional feature vector is output;

[0019] The low-dimensional feature vector is obtained, the probability that the low-dimensional feature vector corresponds to a storage layer in the shared database is predicted based on a conditional random field (CRF), and it is determined whether the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database exceeds a preset probability threshold. If the preset probability threshold is exceeded, the low-dimensional feature vector and the storage layer establish a mapping relationship, the low-dimensional feature vector is mapped to the storage layer, and the creation of the shared database is completed.

[0020] Preferably, the method for pre-processing multi-modal data based on a pre-constructed joint adversarial model comprises:

[0021] The joint adversarial model is pre-constructed for filtering and feature extraction of multi-modal data, wherein the joint adversarial model comprises an input layer, a type discriminator, an adaptive filter, a joint adversarial network, and an output layer. The input layer is connected to the type discriminator, the type discriminator is connected to the adaptive filter, the adaptive filter is connected to the joint adversarial network, and the joint adversarial network is connected to the output layer. The type discriminator identifies the type of multi-modal data based on a decision tree algorithm. The adaptive filter is composed of a multi-heterogeneous filter layer and a Kalman filter layer. The joint adversarial network comprises an up-sampling layer, a full-dimensional dynamic convolution module, a BILSTM layer, and a pooling layer.

[0022] The multi-modal data is loaded, and the type discriminator identifies the type of the multi-modal data based on a decision tree algorithm.

[0023] The multi-modal data after type discrimination is obtained, and the adaptive filter cooperatively filters the multi-modal data based on the multi-heterogeneous filter layer and the Kalman filter layer. Noise data in the multi-modal data is marked and removed by combining a three-frame difference detection algorithm and a Kalman filtering algorithm.

[0024] The joint adversarial network performs up-sampling processing on the multi-modal data after filtering processing, and feeds back the up-sampling result to the full-dimensional dynamic convolution module. The full-dimensional dynamic convolution module uses dynamic convolution to adjust the size and shape of the convolution kernel according to the data importance of the multi-modal data, calculates the attention weight of each dimension using the data type, data attribute, data dimension, and data importance dimension, and obtains the multi-modal data dynamic weight by multiplying the attention weight and the convolution kernel. The multi-modal data dynamic weight is convolved with the up-sampling result to generate adaptive dynamic convolution.

[0025] The adaptive dynamic convolution is loaded, and the BILSTM layer performs forward and reverse LSTM operations on the adaptive dynamic convolution to obtain forward and reverse hidden state vectors. The forward and reverse hidden state vectors and the adaptive dynamic convolution are combined to obtain a fine-grained vector representation of the multi-modal data.

[0026] Preferably, the preprocessing importance of the multi-modal data is calculated by the following formula:

[0027]

[0028] wherein A(x t ) represents the preprocessing importance of the multi-modal data, σ(q avg ) is the standard deviation of the type label, attribute label, dimension label, and importance label, q avg is the weighted average value of the type label, attribute label, dimension label, and importance label, are the input value and mean value of the multi-modal data, respectively, and q1, q2, q3, and q4 are the type label, attribute label, dimension label, and importance label quantized by numerical values, respectively.

[0029] When the conditional random field (CRF) is used to predict the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database, the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database is calculated by the following formula:

[0030]

[0031] wherein γ(D x ,C y ) represents the probability that the low-dimensional feature vector D x corresponds to the storage layer C y in the shared database, C and D are the number of low-dimensional feature vectors and the number of storage layers, respectively, are the feature vector matrix, covariance, and mean value of the low-dimensional feature vector, respectively, and sim(D x ,C y ) represents the similarity of the low-dimensional feature vector D x corresponding to the storage layer C y in the shared database, and WC represents the attribute weight matrix of the storage layer C y , exp[·] represents a Gaussian kernel function, D x xC y represents a low-dimensional feature vector D x and the adjacency similarity matrix of the storage layer C y , ω C represents a length scale parameter of the Gaussian kernel function.

[0032] Preferably, the resource scheduling model takes a federated learning model as an initial model, the federated learning model is composed of a distributed server, a central virtual server and a communication scheduling layer, the distributed server is a distributed node of the federated learning model, the distributed server and the central virtual server interact data through a transmission path, the distributed server is composed of an input layer, an ALBERT module and a Transformer encoder, a flow control strategy and a quality of service mechanism are introduced into the ALBERT module, the central virtual server includes a content distribution network architecture CDN, a homomorphic encryption layer and a differential privacy layer, the communication scheduling layer includes an adaptive multi-source multi-path congestion control module, a BLEST scheduler and a full connection model, the full connection model introduces an ant colony optimization algorithm, and the full connection model includes Layer-1, Layer-2 and Layer-3, the node numbers of Layer-1, Layer-2 and Layer-3 are 250, 150 and 300 respectively.

[0033] Preferably, the resource scheduling model training method comprises:

[0034] Traverse the multi-modal data in the distributed hash table, divide the multi-modal data into a training set and a test set, and preset initial parameters, iteration rounds, maximum iteration rounds and a loss function of the resource scheduling model;

[0035] Load the pre-constructed resource scheduling model, pre-train the distributed server using the training set, and obtain server parameters of the distributed server;

[0036] Import at least one group of server parameters of the distributed server based on the content distribution network architecture CDN, and dynamically aggregate the server parameters based on an adaptive multi-source multi-path congestion control algorithm combined with an ant colony optimization algorithm, calculate the path delay rate and the congestion window rate of the server parameters;

[0037] Retain the server parameters whose path delay rate and congestion window rate addition value is less than a preset cumulative threshold value, and perform weight pruning on the server parameters whose path delay rate and congestion window rate addition value is greater than or equal to the preset cumulative threshold value;

[0038] Load the weight pruned server parameters, set the weight pruned server parameters as global parameters, deploy the global parameters in the distributed server, and obtain a converged resource scheduling model;

[0039] Obtain a test set, input the test set into the converged resource scheduling model, dynamically deploy a test data transmission task based on test set attributes and real-time transmission path information, and calculate the test set transmission response time;

[0040] Determine whether the test set transmission response time meets a preset time threshold, and if the test set transmission response time meets the preset time threshold, output the converged resource scheduling model.

[0041] Preferably, the method for dynamically deploying a data transmission task based on target data attributes and real-time transmission path information comprises:

[0042] Obtain target data attributes, index at least one group of distributed hash tables based on the target data attributes, create at least one transmission path using an adaptive multi-source multi-path congestion control algorithm, poll and allocate the target data to the transmission path using a multi-path transmission protocol, and identify real-time transmission path information corresponding to the transmission path, wherein the real-time transmission path information includes bandwidth, delay, and packet loss rate.

[0043] Generate at least one transmission path combination using an ant colony optimization algorithm, with the lowest delay path and congestion window rate as constraints.

[0044] Calculate the path delay rate and congestion window rate of the transmission path combination based on the adaptive multi-source multi-path congestion control algorithm, and determine the optimal transmission path combination based on the sum of the path delay rate and the congestion window rate being less than the minimum sum.

[0045] In response to the optimal transmission path combination, dynamically deploy a data transmission task based on the optimal transmission path combination.

[0046] In another aspect, the present application also provides a big data resource service system based on multi-modal data fusion, which comprises:

[0047] A database creation module is configured to obtain multi-modal data sequences collected by data sources in a distributed manner, pre-process the multi-modal data sequences based on data fusion technology, obtain multi-modal data represented by low-dimensional feature vectors, and create a shared database based on the pre-processed low-dimensional feature vectors.

[0048] A distributed encryption module is configured to encrypt and distribute the multi-modal data in the shared database to the distributed hash tables of the distributed nodes based on peer-to-peer technology, and determine the data location of the multi-modal data in the distributed nodes through a hash function.

[0049] The resource access module identifies the resource retrieval instruction based on a pre-constructed resource scheduling model in response to a resource retrieval instruction for a subject, and triggers a resource access control request;

[0050] The storage retrieval module extracts the target data to be scheduled and target data attributes based on the cosine similarity by traversing the distributed hash table of at least one group of distributed nodes in response to the resource access control request.

[0051] The retrieval scheduling module dynamically deploys a data transmission task based on the target data attributes and real-time transmission path information, performs point-to-point encryption transmission of the target data to be scheduled based on the data transmission task, and generates a retrieval result in combination with large language model interaction.

[0052] Preferably, the database creation module comprises:

[0053] The data marking unit is configured to delete repeated data and abnormal data in the multi-modal data sequence, and perform type marking, attribute marking, dimension marking, and importance marking on the multi-modal data based on data types, data attributes, data dimensions, and data importance.

[0054] The importance calculation unit is configured to obtain the marked multi-modal data sequence, and define a pre-processing importance of the multi-modal data in the multi-modal data sequence based on a probability density function and a weighted average algorithm.

[0055] The importance judgment unit is configured to judge whether the pre-processing importance of the multi-modal data exceeds a preset importance threshold.

[0056] The vector splicing unit is configured to load the multi-modal data represented by the fine-grained vectorization and the coarse-grained vectorization, splice the multi-modal data features represented by the fine-grained vectorization and the coarse-grained vectorization based on a multi-Transformer architecture combined with self-attention and cross-attention data fusion technology, obtain high-dimensional fusion features, perform dimension reduction processing on the high-dimensional fusion features using PCA to reduce the feature dimension, and output normalized low-dimensional feature vectors.

[0057] The data mapping storage unit is configured to obtain the low-dimensional feature vectors, predict the probability that the low-dimensional feature vectors correspond to a storage layer in the shared database based on a conditional random field (CRF), judge whether the probability that the low-dimensional feature vectors correspond to the storage layer in the shared database exceeds a preset probability threshold, and if so, establish a mapping relationship between the low-dimensional feature vectors and the storage layer, map the low-dimensional feature vectors to the storage layer, and complete the creation of the shared database.

[0058] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0059] In the embodiment of the present application, the multi-modal data in the shared database is encrypted and distributed to the distributed hash table of the distributed nodes by using point-to-point technology, and the data position is determined by a hash function, realizing efficient distributed storage and access, and combining a distributed server, a central virtual server and a communication scheduling layer, a resource scheduling model considering the distribution and transmission characteristics of multi-modal data in a distributed environment is constructed, realizing collaborative learning and optimization between distributed nodes, dynamically optimizing the transmission path, reducing the delay and energy consumption of data transmission, and improving the intelligent level of scheduling.

[0060] In the embodiment of the present application, through the judgment of the preprocessing importance, the system can allocate more computing resources (such as time, storage, computing power) to high importance multi-modal data, and simplify the processing of low importance data, thereby avoiding the "one-size-fits-all" approach of using the same processing precision for all data, reducing unnecessary computing overhead, improving the overall operation efficiency of the system, especially in large-scale multi-modal data processing scenarios, the optimization of resource allocation is particularly important, thereby ensuring that the system can quickly adjust the processing strategy in a dynamic scenario, and the most critical data in the current task is processed preferentially, and the high importance data features processed in fine granularity provide depth information, enhancing the accuracy of the fusion result, and the low importance data features processed in coarse granularity provide breadth information, enriching the coverage of the fusion result, and this complementarity makes the fusion result more comprehensive and reliable.

[0061] In the embodiment of the present application, based on the data fusion technology for preprocessing multi-modal data sequence, the differentiated processing strategy based on importance optimizes the allocation of computing resources, improves the system efficiency, realizes efficient fusion and dimension reduction of multi-modal data by using multi-Transformer architecture, improves the usability of data, and introduces probability prediction and dynamic storage allocation mechanism, realizes the intelligentization and adaptability of data processing.

[0062] In the embodiment of the present application, a joint adversarial model for filtering and feature extraction of multi-modal data is provided, which combines decision tree, Kalman filtering, dynamic convolution, BILSTM and other technologies, can process complex multi-modal data distribution, is suitable for feature extraction of various data types (such as text, image, audio, etc.), and the dynamic filtering, dynamic convolution and attention mechanism of the joint adversarial model have strong adaptability, can cope with the dynamic changes and complex distribution of multi-modal data, can adaptively extract deep features of multi-modal data, and improve the accuracy and effect of feature extraction.

[0063] In the embodiment of the present application, during the training of the resource scheduling model, the redundant parameters with less contribution to the performance of the resource scheduling model are removed through weight pruning, the complexity of the resource scheduling model is reduced, the parameter set after pruning is more concise, the calculation overhead of subsequent training and inference is reduced, the key parameters are retained, the interference of redundant parameters on model optimization is avoided, the accuracy and reliability of the model are improved, and the pruning operation reduces the calculation resource consumption of the distributed server and the central virtual server, and improves the resource utilization of the system.

[0064] In the embodiment of the present application, when the resource scheduling model dynamically deploys a data transmission task based on target data attributes and real-time transmission path information, the path combination with the lowest delay is preferentially selected, the delay of data transmission is significantly reduced, the scene with high real-time requirement (such as video streaming media, online games, etc.) is met, the multi-path transmission protocol and the ant colony optimization algorithm are used to transmit data in parallel on multiple paths, the bandwidth utilization and transmission speed are improved, the network congestion is avoided through the constraint of the congestion window rate, and the stability and reliability of transmission are ensured, the ant colony optimization algorithm has strong global search capability and can find a path combination close to the optimal solution, and local optimal solution is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 is the implementation flowchart of the big data resource service method based on multi-modal data fusion provided by the present application.

[0066] Figure 2 The multi-modal data sequence preprocessing method based on data fusion technology is shown.

[0067] Figure 3 The multi-modal data preprocessing method based on the pre-constructed joint adversarial model is shown.

[0068] Figure 4 The resource scheduling model training method implementation flowchart is shown.

[0069] Figure 5 The resource scheduling model based on target data attributes and real-time transmission path information dynamically deploys a data transmission task method implementation flowchart is shown.

[0070] Figure 6 The structure diagram of the big data resource service system based on multi-modal data fusion is shown. DETAILED DESCRIPTION

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application; the use herein of terms such as "comprise", "have" and "include" or any variation thereof, is intended to cover a non-exclusive inclusion, the use herein of terms such as "first", "second" and the like is intended to distinguish between similar objects unless the context indicates otherwise.

[0072] The existing method only formulates a scheduling scheme by calculating the energy consumption value and resource utilization of the server, and has poor adaptability to dynamic changes, and does not consider the data distribution and transmission characteristics in a distributed environment, thereby reducing the overall performance and resource utilization of the system. In view of the above problems, the present application provides a big data resource service method and system based on multi-modal data fusion. In short, when the method is implemented, first, the multi-modal data sequence collected by the data source is distributedly acquired, the multi-modal data sequence is preprocessed based on a data fusion technology, a shared database is created based on the low-dimensional feature vector after preprocessing, the multi-modal data in the shared database is encrypted and distributed to the distributed hash table of the distributed node based on point-to-point technology, and the position of the multi-modal data in the distributed node is determined through a hash function. In the service stage, in response to a resource retrieval instruction for a topic, the resource retrieval instruction is identified based on a pre-constructed resource scheduling model, the distributed hash table of at least one group of distributed nodes is traversed based on cosine similarity, the target data to be scheduled and the target data attribute are extracted, the resource scheduling model dynamically deploys a data transmission task based on the target data attribute and real-time transmission path information, the point-to-point encrypted transmission of the target data to be scheduled is performed based on the data transmission task, and a retrieval result is generated by combining a large language model interaction. In the embodiment of the application, the multi-modal data in the shared database is encrypted and distributed to the distributed hash table of the distributed node based on point-to-point technology, and the data position is determined through a hash function, thereby realizing efficient distributed storage and access. In combination with a distributed server, a central virtual server and a communication scheduling layer, a resource scheduling model considering the multi-modal data distribution and transmission characteristics in a distributed environment is constructed, collaborative learning and optimization between distributed nodes are realized, the transmission path is dynamically optimized, the delay and energy consumption of data transmission are reduced, and the intelligent level of scheduling is improved.

[0073] The embodiment of the application provides a big data resource service method based on multi-modal data fusion, Figure 1 A big data resource service method based on multi-modal data fusion is shown, and the big data resource service method based on multi-modal data fusion specifically comprises:

[0074] In step S10, the multi-modal data sequence collected by the distributed data source is acquired, the multi-modal data sequence is preprocessed based on a data fusion technology, and multi-modal data represented by a low-dimensional feature vector is obtained, and a shared database is created based on the preprocessed low-dimensional feature vector;

[0075] In step S20, the multi-modal data in the shared database is encrypted and distributed to the distributed hash table of the distributed node based on the peer-to-peer technology, and the position of the multi-modal data in the distributed node is determined by a hash function.

[0076] It should be noted that when the multi-modal data in the shared database is encrypted and distributed to the distributed hash table of the distributed node based on the peer-to-peer technology, a symmetric encryption algorithm (such as AES) or an asymmetric encryption algorithm (such as RSA) can be selected for encryption processing of the multi-modal data, and the selected encryption algorithm is used to encrypt the multi-modal data. For each piece of data, a unique encrypted identifier (such as a hash value) is generated and stored together with the encrypted data, and when the distributed hash table is constructed, a DHT algorithm can be selected, including Kademlia and Chord. The Kademlia algorithm organizes nodes by XOR distance of node ID, and has high routing performance; the Chord algorithm distributes data by consistent hashing, ensures uniform distribution of data, selects a bootstrap node, and the new node joins the network by communicating with the bootstrap node. The bootstrap node helps the new node to establish a routing table and introduces it into the DHT network. For each encrypted multi-modal data, a hash function (such as SHA-1) is used to calculate its hash value. This hash value will be used as the key of the data, and according to the DHT algorithm, the hash value of the data is mapped to the virtual node space of the DHT, and the node responsible for storing the data is found. The encrypted multi-modal data is sent to the corresponding node and stored on the node. After the node receives the data, it is stored in the local database or file system.

[0077] In the embodiment of the application, the decentralized shared database can store data on multiple distributed nodes based on peer-to-peer technology, and each distributed node has a complete data copy. This method significantly reduces the risk of single point failure, and even if a node fails, other nodes can still provide services to ensure data security. Through the distributed hash table (DHT) and encryption technology, the data is always in an encrypted state during transmission and storage, further enhancing the privacy protection of the data. The decentralized shared database uses data redundancy storage, so even if some nodes are damaged, the integrity of the data can be guaranteed. The distributed nodes ensure the consistency and integrity of the data through consensus algorithms, making the system have higher transparency and trustworthiness. Based on the peer-to-peer technology, the distributed nodes can dynamically deploy data transmission tasks according to real-time transmission path information, further improving the flexibility and efficiency of resource scheduling.

[0078] Step S30, in response to the resource retrieval instruction for the topic, identifying the resource retrieval instruction based on the pre-constructed resource scheduling model, triggering a resource access control request;

[0079] Step S40, in response to the resource access control request, traversing the distributed hash table of at least one group of distributed nodes based on the cosine similarity, extracting the target data to be scheduled and the target data attribute, extracting the data stored by each distributed node when scheduling the target data, calculating the feature vector thereof, and calculating the feature vector of the target data to be scheduled. The cosine similarity formula is used to calculate the similarity between the data of each distributed node and the target data. Starting from a node in the DHT, find the node most similar to the target data, and then use the node search algorithm of Kademlia to recursively find the K nodes closest to the target data. The cosine similarity of the data stored by the K nodes is calculated, and the data with the highest similarity is extracted as the target data to be scheduled.

[0080] Step S50, the resource scheduling model dynamically deploys data transmission tasks based on the target data attribute and real-time transmission path information, performs point-to-point encryption transmission of the target data to be scheduled based on the data transmission task, and generates a retrieval result in combination with a large language model interaction.

[0081] In this embodiment, when generating a retrieval result in combination with a large language model interaction, the large language model can be ChatGLM-6B, Stanford Alpaca, or Nemotron-4 340B. Large language models (LLMs) are an important development in the field of natural language processing (NLP) in recent years. They can handle various natural language tasks such as text generation, dialogue systems, and machine translation through large-scale pre-training and fine-tuning.

[0082] In the embodiment of the application, point-to-point technology is used to encrypt and distribute multi-modal data in a shared database to a distributed hash table of distributed nodes, and a hash function is used to determine the data location, achieving efficient distributed storage and access. In combination with a distributed server, a central virtual server, and a communication scheduling layer, a resource scheduling model is constructed that considers the distribution and transmission characteristics of multi-modal data in a distributed environment, enabling collaborative learning and optimization between distributed nodes, dynamically optimizing the transmission path, reducing the delay and energy consumption of data transmission, and improving the intelligence level of scheduling.

[0083] The embodiment of the application provides a method for preprocessing a multi-modal data sequence based on data fusion technology, Figure 2 A flowchart of a method for preprocessing a multi-modal data sequence based on data fusion technology is shown. The method for preprocessing a multi-modal data sequence based on data fusion technology specifically includes:

[0084] Step S101, load the multi-modal data sequence, delete the repeated data and abnormal data in the multi-modal data sequence, and perform type marking, attribute marking, dimension marking and importance marking on the multi-modal data based on data type, data attribute, data dimension and data importance to obtain the marked multi-modal data sequence;

[0085] In the embodiment of the application, the multi-modal data sequence includes but is not limited to text, image, audio, video, sensor data and geographic spatial data. When performing type marking, attribute marking, dimension marking and importance marking on the multi-modal data based on data type, data attribute, data dimension and data importance, the marking values of data type, data attribute, data dimension and data importance can be determined based on principal component analysis or voting method. Through type marking, attribute marking, dimension marking and importance marking, the characteristics of the multi-modal data are determined, which provides a clear structured basis for subsequent preprocessing and fusion. Through importance marking, the criticality of the data can be distinguished, which provides a basis for importance-based preprocessing in the subsequent steps and avoids wasting resources on low-value data.

[0086] Step S102, obtain the marked multi-modal data sequence, and define the preprocessing importance of the multi-modal data in the multi-modal data sequence based on a probability density function combined with a weighted average algorithm;

[0087] It should be noted that the preprocessing importance of each multi-modal data can be more accurately evaluated by combining the probability density function and the weighted average algorithm. This method can comprehensively consider the distribution characteristics and weights of the data, so as to more accurately reflect the actual importance of the data. By quantifying the importance, high importance data can be processed preferentially, and excessive processing of low importance data can be avoided, thereby improving the overall preprocessing efficiency. Considering that the probability density function and the weighted average algorithm have strong adaptability and can dynamically adjust the calculation method of the importance according to the data distribution, they are suitable for various scenarios.

[0088] In the embodiment of the application, the preprocessing importance of the multi-modal data is calculated by the following formula:

[0089]

[0090] wherein, A(x t ) represents the preprocessing importance of the multi-modal data, σ(q avg ) is the standard deviation of the type marking, attribute marking, dimension marking and importance marking, q avg is the weighted average value of the type marking, attribute marking, dimension marking and importance marking, and x t , The input value and the mean value of the multi-modal data are respectively, q1, q2, q3 and q4 are respectively a type label, an attribute label, a dimension label and an importance label quantified by a numerical value, wherein the type label, the attribute label, the dimension label and the importance label quantified by the numerical value can be 1-10.

[0091] In step S103, it is judged whether the pre-processing importance of the multi-modal data exceeds a preset importance threshold.

[0092] It should be noted that, by distinguishing high importance data and low importance data, the present application reduces unnecessary computing overhead and improves the overall operation efficiency of the system, and the importance threshold can be 0.3-0.5.

[0093] In step S104, if the pre-processing importance of the multi-modal data exceeds the preset importance threshold, the multi-modal data is pre-processed based on a pre-constructed joint adversarial model, and the multi-modal data is represented by fine-grained vectorization.

[0094] In the embodiment of the present application, if the pre-processing importance of the multi-modal data exceeds the preset importance threshold, it is judged that the multi-modal data is relatively important (the pre-processing importance of the multi-modal data can be greater than 0.3), and fine-grained vectorization processing is required. The pre-constructed joint adversarial model can extract deep features of the multi-modal data through adversarial training, providing more abundant information for subsequent fusion and scheduling. Moreover, the joint adversarial model has strong modeling ability and can handle complex multi-modal data distribution, which is suitable for various complex scenarios.

[0095] In step S105, if the pre-processing importance of the multi-modal data does not exceed the preset importance threshold, the multi-modal data is pre-processed based on the data type of the multi-modal data, and the multi-modal data is represented by coarse-grained vectorization.

[0096] In the embodiment of the present application, if the pre-processing importance of the multi-modal data does not exceed the preset importance threshold, it is judged that the importance of the current multi-modal data is general. The multi-modal data can be pre-processed based on the data type of the multi-modal data by using mature existing technology. For example, if the data type is text, BERT model recognition can be used, and if the data type is picture or video, convolutional neural network can be used for pre-processing. Fine-grained processing is used for high importance data to ensure that its deep features are fully extracted. Coarse-grained processing is used for low importance data to ensure that its basic features are retained. This differentiated processing strategy realizes the balance of data features and improves the comprehensiveness and consistency of the fusion result.

[0097] In the embodiment of the present application, through the judgment of the preprocessing importance, the system can allocate more computing resources (such as time, storage, and computing power) to high-importance multi-modal data, and simplify the processing of low-importance data, thereby avoiding the "one-size-fits-all" approach of using the same processing precision for all data, reducing unnecessary computing overhead, and improving the overall operation efficiency of the system. In particular, in a large-scale multi-modal data processing scenario, the optimization of resource allocation is particularly important, thereby ensuring that the system can quickly adjust the processing strategy in a dynamic scenario, and prioritize the processing of the most critical data in the current task. The high-importance data features processed in a fine-grained manner provide depth information, enhancing the accuracy of the fusion result. The low-importance data features processed in a coarse-grained manner provide breadth information, enriching the coverage of the fusion result. This complementarity makes the fusion result more comprehensive and reliable.

[0098] In step S106, the multi-modal data represented by fine-grained vectorization and coarse-grained vectorization is loaded, and the multi-modal data features represented by fine-grained vectorization and coarse-grained vectorization are spliced based on a multi-Transformer architecture combining self-attention and cross-attention data fusion technology to obtain high-dimensional fusion features. The multi-Transformer architecture combines self-attention and cross-attention mechanisms, which can efficiently fuse fine-grained vectors and coarse-grained vectors, capture complex relationships between multi-modal data, and use PCA to reduce the dimensionality of high-dimensional fusion features, reducing the feature dimension, and outputting normalized low-dimensional feature vectors. The Transformer architecture has strong modeling capability and can adaptively learn the features of multi-modal data, improving the intelligent level of fusion.

[0099] In step S107, the low-dimensional feature vector is obtained, and the conditional random field CRF is used to predict the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database. It is determined whether the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database exceeds the preset probability threshold. If it exceeds the preset probability threshold, the low-dimensional feature vector and the storage layer establish a mapping relationship, and the low-dimensional feature vector is mapped to the storage layer, completing the creation of the shared database.

[0100] In this embodiment, when predicting the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database based on the conditional random field CRF, the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database is calculated by the following formula:

[0101]

[0102] where γ(D x ,C y ) represents the probability that the low-dimensional feature vector D x corresponds to the storage layer C y in the shared database, C and D are the number of low-dimensional feature vectors and the number of storage layers, respectively. respectively, are a feature vector matrix, covariance, mean of low-dimensional feature vectors, sim(D x ,C y ) represents the similarity of the low-dimensional feature vector D x corresponding to the storage layer C y in the shared database, W C represents the attribute weight matrix of the storage layer C y , exp[·] represents the Gaussian kernel function, D x xC y represents the adjacency similarity matrix of the low-dimensional feature vector D x and the storage layer C y , and omega C represents the length scale parameter of the Gaussian kernel function.

[0103] In the embodiment of the application, the conditional random field CRF can accurately predict the storage probability of the low-dimensional feature vector based on the context information, ensure the rationality of data storage, judge whether to store data through a probability threshold, realize dynamic storage allocation, and avoid waste of storage resources. The low-dimensional feature vector is mapped to the storage layer, the creation of the shared database is completed, and subsequent data retrieval and management are facilitated.

[0104] In the embodiment of the application, when the multi-modal data sequence is preprocessed based on the data fusion technology, the importance-based differential processing strategy optimizes the allocation of computing resources, improves the system efficiency, realizes efficient fusion and dimension reduction of multi-modal data by using a multi-Transformer architecture, improves the usability of data, and introduces a probability prediction and dynamic storage allocation mechanism to realize the intelligentization and adaptability of data processing.

[0105] The embodiment of the application provides a method for preprocessing multi-modal data based on a pre-constructed joint adversarial model, Figure 3 Fig. 1 shows a flowchart of a method for preprocessing multi-modal data based on a pre-constructed joint adversarial model, and the method for preprocessing multi-modal data based on the pre-constructed joint adversarial model specifically comprises:

[0106] Step S1041, a joint adversarial model for filtering and feature extraction of multi-modal data is pre-constructed, wherein the joint adversarial model comprises an input layer, a type discriminator, an adaptive filter, a joint adversarial network, and an output layer, the input layer is connected with the type discriminator, the type discriminator is connected with the adaptive filter, the adaptive filter is connected with the joint adversarial network, the joint adversarial network is connected with the output layer, the type discriminator identifies the type of multi-modal data based on a decision tree algorithm, the adaptive filter is composed of a multi-heterogeneous filter layer and a Kalman filter layer, and the joint adversarial network comprises an up-sampling layer, a full-dimensional dynamic convolution module, a BILSTM layer, and a pooling layer.

[0107] In step S1042, the multi-modal data is loaded, and the type discriminator identifies the data type of the multi-modal data based on a decision tree algorithm, determines the data type of the multi-modal data, the decision tree algorithm can classify according to the characteristics of the data (such as data type, attribute, etc.), is suitable for a variety of complex multi-modal data scenes, and through the pre-classification of the type discriminator, the computational complexity of the subsequent processing module is reduced, and the overall processing efficiency is improved.

[0108] In step S1043, the multi-modal data after the data type discrimination is obtained, and the adaptive filter cooperatively filters the multi-modal data based on a multi-heterogeneous filter layer and a Kalman filter layer, and combines a three-frame difference detection algorithm and a Kalman filter algorithm to mark and remove noise data in the multi-modal data, so that the noise data in the multi-modal data can be effectively identified and removed through the combination of the three-frame difference detection algorithm and the Kalman filter algorithm, and the purity of the data is improved.

[0109] In step S1044, the joint adversarial network performs up-sampling processing on the multi-modal data after the filtering processing, and feeds back the up-sampling result to the full-dimensional dynamic convolution module, the full-dimensional dynamic convolution module uses dynamic convolution to adjust the size and shape of the convolution kernel according to the data importance of the multi-modal data, calculates the attention weight of each dimension by using the data type, data attribute, data dimension and data importance dimension, and multiplies the attention weight and the convolution kernel to obtain the multi-modal data dynamic weight, and the multi-modal data dynamic weight is convolved with the up-sampling result to generate adaptive dynamic convolution.

[0110] In step S1045, the adaptive dynamic convolution is loaded, the BILSTM layer performs forward and reverse LSTM operations on the adaptive dynamic convolution to obtain forward and reverse hidden state vectors, and the forward and reverse hidden state vectors and the adaptive dynamic convolution are combined to obtain a fineness vector representing the multi-modal data, so that the model can capture the forward and backward dependencies of the data through the combination of the forward and reverse LSTM operations, and the comprehensiveness of feature extraction is improved.

[0111] In the embodiment, the joint adversarial model for filtering and feature extraction of multi-modal data is provided, which combines decision tree, Kalman filter, dynamic convolution, BILSTM and other technologies, can process complex multi-modal data distribution, is suitable for feature extraction of various data types (such as text, image, audio, etc.), and the dynamic filtering, dynamic convolution and attention mechanism of the joint adversarial model have strong adaptability, can cope with the dynamic changes and complex distribution of multi-modal data, can adaptively extract deep features of multi-modal data, and improve the accuracy and effect of feature extraction.

[0112] The resource scheduling model training method provided in the embodiment of the application comprises the following steps: Figure 4A resource scheduling model training method implementation flowchart is shown, the resource scheduling model training method specifically comprises:

[0113] Step S201, traverse the multi-modal data in the distributed hash table, divide the multi-modal data into a training set and a test set, and preset initial parameters, iteration rounds, maximum iteration rounds and a loss function of the resource scheduling model;

[0114] In the embodiment of the application, the ratio of the training set and the test set can be 5:1, the iteration rounds can be 50-200 Epochs, the maximum iteration rounds can be 250 times, the loss function can be a cross-entropy loss function, the initial parameter setting can adopt Xavier initialization or He initialization setting, and the initial parameters include but are not limited to a learning rate, a batch size and a loss function weight.

[0115] Step S202, load the pre-constructed resource scheduling model, pre-train the distributed server using the training set, obtain server parameters of the distributed server, and pre-train on the distributed server to fully utilize distributed computing resources and improve training efficiency, the pre-training can generate optimized server parameters for the distributed server and lay a foundation for subsequent global parameter aggregation, the distributed server is composed of an input layer, an ALBERT module and a Transformer encoder, and can better capture the characteristics of multi-modal data and improve the expression ability of the model in combination with a flow control strategy and a quality of service mechanism;

[0116] Step S203, import the server parameters of at least one group of distributed servers based on a content distribution network architecture CDN, and dynamically aggregate the server parameters based on an adaptive multi-source multi-path congestion control algorithm combined with an ant colony optimization algorithm, calculate the path delay rate and the congestion window rate of the server parameters, the adaptive multi-source multi-path congestion control algorithm can adjust the parameters in real time to adapt to network congestion, improve the robustness and reliability of the system, the introduction of the ant colony optimization algorithm makes the parameter aggregation process more intelligent, can quickly find the optimal solution, reduces the computational complexity, and the architecture design based on the CDN can effectively reduce the data transmission delay and improve the overall performance of the system;

[0117] Step S204, retain the server parameters with a path delay rate and a congestion window rate addition value less than a preset cumulative threshold, and perform weight pruning on the server parameters with a path delay rate and a congestion window rate addition value greater than or equal to the preset cumulative threshold;

[0118] It should be noted that the preset cumulative threshold can be 0.1-0.15, the path delay rate refers to the ratio of the transmission delay of data from the source node to the target node to the ideal transmission delay, and is usually used to measure the transmission efficiency of the path, and the transmission delay under ideal conditions (no congestion, no packet loss, optimal path) can be estimated through network topology and historical data. For each transmission path, the cumulative node delay and the link delay are added to obtain the actual path delay. The congestion window rate refers to the ratio of the size of the congestion window of the current transmission path to the size of the maximum congestion window, and is used to measure the congestion degree of the network. The ant colony optimization algorithm can be used to optimize the path selection and parameter aggregation process. The path delay rate and the congestion window rate of each path are used as the initial pheromone value of the ant, and the ant selects the path according to the comprehensive performance index of the path delay rate and the congestion window rate. The path with higher performance has higher pheromone concentration. After multiple iterations, the ant colony algorithm can converge to the optimal path, ensuring the optimal balance of the path delay rate and the congestion window rate.

[0119] In the embodiment of the application, during training of the resource scheduling model, redundant parameters with small contribution to the performance of the resource scheduling model are removed through weight pruning, the complexity of the resource scheduling model is reduced, the parameter set after pruning is more compact, the calculation overhead of subsequent training and inference is reduced, and the precision and reliability of the model are improved. The pruning operation reduces the calculation resource consumption of the distributed server and the central virtual server, and improves the resource utilization of the system.

[0120] In step S205, the server parameters after weight pruning are loaded, the server parameters after weight pruning are set as global parameters, the global parameters are deployed in the distributed server, and a converged resource scheduling model is obtained.

[0121] In step S206, a test set is obtained, the test set is input into the converged resource scheduling model, the resource scheduling model dynamically deploys a test data transmission task based on the test set attribute and real-time transmission path information, and calculates the transmission response time of the test set.

[0122] In step S207, it is determined whether the transmission response time of the test set meets a preset time threshold, and the time threshold can be 50-150 ms.

[0123] In step S208, if the transmission response time of the test set meets the preset time threshold, the converged resource scheduling model is output.

[0124] Returning to step S202, if the transmission response time of the test set does not meet the preset time threshold, the iterative training of the resource scheduling model is continued.

[0125] In the embodiment, the resource scheduling model takes a federated learning model as an initial model. The federated learning avoids leakage of original data by training the model locally and exchanging only model parameters, effectively protects user privacy, reduces the possibility of eavesdropping or tampering of data during transmission, and improves overall data security. The federated learning model is composed of a distributed server, a central virtual server, and a communication scheduling layer. The distributed server is a distributed node of the federated learning model, and uses the computing power of edge devices (such as mobile phones and IoT devices) to reduce the load of the central server and improve system scalability. The distributed server is composed of an input layer, an ALBERT module, and a Transformer encoder, which can efficiently process multi-modal data and transmit model parameters instead of large-scale original data, significantly reducing communication overhead and storage costs. The distributed server and the central virtual server interact through a transmission path. The distributed server is composed of an input layer, an ALBERT module, and a Transformer encoder. The ALBERT module introduces a flow control strategy and a quality of service mechanism. Through the flow control strategy, the data packet flow under specified conditions can be counted, released, blocked, and other operations, optimizing the use of network resources. The central virtual server includes a content distribution network architecture CDN, a homomorphic encryption layer, and a differential privacy layer. The communication scheduling layer includes an adaptive multi-source multi-path congestion control module, a BLEST scheduler, and a fully connected model. The fully connected model introduces an ant colony optimization algorithm, and the fully connected model includes Layer-1, Layer-2, and Layer-3, with 250, 150, and 300 nodes respectively.

[0126] The embodiment of the present application provides a method for dynamically deploying a data transmission task based on target data attributes and real-time transmission path information, Figure 5 The resource scheduling model based on target data attributes and real-time transmission path information dynamically deploys a data transmission task method implementation flowchart is shown, the method for dynamically deploying a data transmission task based on target data attributes and real-time transmission path information of the resource scheduling model, specifically includes:

[0127] Step S301, obtain target data attributes, index at least one group of distributed hash tables based on the target data attributes, index the distributed hash table through the target data attributes (such as data size, priority, type, etc.), which can quickly locate the suitable transmission path, improve the efficiency and accuracy of path selection, and create at least one group of transmission paths by the adaptive multi-source multi-path congestion control algorithm, poll and allocate and schedule target data to the transmission path by using a multi-path transmission protocol, identify real-time transmission path information corresponding to the transmission path, the real-time transmission path information includes bandwidth, delay, packet loss rate, and real-time transmission path information such as bandwidth, delay, and packet loss rate, which can dynamically reflect the network state, ensure that the selection of the transmission path is always based on the latest network conditions, and the adaptive multi-source multi-path congestion control algorithm can dynamically adjust the transmission path according to the network congestion situation, avoid single path overload, and improve the robustness of the system.

[0128] Step S302, at least one group of transmission path combinations is generated by using the path with the lowest delay and the congestion window rate as the constraint, and the ant colony optimization algorithm is combined, the ant colony optimization algorithm is an optimization algorithm based on swarm intelligence, which can simulate the foraging behavior of ants and gradually find the optimal path combination, and is suitable for solving complex multi-path selection problems, the ant colony optimization algorithm has strong global search ability and can avoid falling into local optimal solution, and the generated path combination is close to the global optimum.

[0129] Step S303, the path delay rate and the congestion window rate of the transmission path combination are calculated based on the adaptive multi-source multi-path congestion control algorithm, the optimal transmission path combination is that the sum of the path delay rate and the congestion window rate is the minimum, by comprehensively considering the path delay rate and the congestion window rate, the comprehensive performance of the transmission path can be evaluated comprehensively, the one-sidedness caused by a single index is avoided, the weighted sum of the path delay rate and the congestion window rate is used as the decision basis, the calculation is simple and easy to implement, and the optimal path combination can be quickly found;

[0130] Step S304, in response to the optimal transmission path combination, the data transmission task is dynamically deployed based on the optimal transmission path combination.

[0131] In the embodiment of the application, when the resource scheduling model dynamically deploys the data transmission task based on the target data attributes and the real-time transmission path information, the path combination with the lowest delay is preferentially selected, the delay of data transmission is significantly reduced, the scenes with high real-time requirements (such as video streaming media, online games, etc.) are met, the multi-path transmission protocol and the ant colony optimization algorithm are used to transmit data in parallel on multiple paths, the bandwidth utilization and transmission speed are improved, the network congestion is avoided by the constraint of the congestion window rate, and the stability and reliability of the transmission are ensured, the ant colony optimization algorithm has strong global search ability and can find a path combination close to the optimal solution, and local optimal solution is avoided.

[0132] The embodiment of the application provides a big data resource service system based on multi-modal data fusion, Figure 6 A structural schematic diagram of the big data resource service system based on multi-modal data fusion is shown, and the big data resource service system based on multi-modal data fusion specifically comprises:

[0133] The database creation module 100 is used for distributedly acquiring multi-modal data sequences collected by a data source, pre-processing the multi-modal data sequences based on a data fusion technology, obtaining multi-modal data represented by low-dimensional feature vectors, and creating a shared database based on the pre-processed low-dimensional feature vectors;

[0134] The distributed encryption module 200 is used for encrypting and distributing the multi-modal data in the shared database to a distributed hash table of distributed nodes based on a peer-to-peer technology, and determining the data positions of the multi-modal data in the distributed nodes by using a hash function;

[0135] The resource access module 300 is used for identifying a resource retrieval instruction based on a pre-constructed resource scheduling model in response to the resource retrieval instruction for a subject, and triggering a resource access control request;

[0136] The storage calling module 400 is used for traversing the distributed hash table of at least one group of distributed nodes based on a cosine similarity in response to the resource access control request, extracting target data to be scheduled and target data attributes;

[0137] The retrieval scheduling module 500 is used for dynamically deploying a data transmission task based on the target data attributes and real-time transmission path information, performing point-to-point encryption transmission of the target data to be scheduled based on the data transmission task, and generating a retrieval result in combination with a large language model interaction.

[0138] In the embodiment, the database creation module 100 comprises:

[0139] The data marking unit 110 is used for deleting repeated data and abnormal data in the multi-modal data sequences, and marking the multi-modal data based on a data type, a data attribute, a data dimension and a data importance in a numerical quantity, an attribute, a dimension and an importance;

[0140] The importance calculation unit 120 is used for acquiring the marked multi-modal data sequences, and defining a pre-processing importance of the multi-modal data in the multi-modal data sequences based on a probability density function in combination with a weighted average algorithm;

[0141] The importance judgment unit 130 is used for judging whether the pre-processing importance of the multi-modal data exceeds a preset importance threshold value;

[0142] The vector splicing unit 140 is used for loading the multi-modal data represented by the fineness vectorization and the roughness vectorization, splicing the multi-modal data features represented by the fineness vectorization and the roughness vectorization based on the multi-Transformer architecture combining the self-attention and the cross-attention data fusion technology, obtaining high-dimensional fusion features, performing dimension reduction processing on the high-dimensional fusion features by using the PCA, reducing the feature dimension, and outputting the normalized low-dimensional feature vector;

[0143] The data mapping storage unit 150 is used for acquiring the low-dimensional feature vector, predicting the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database based on the conditional random field (CRF), judging whether the probability that the low-dimensional feature vector corresponds to the storage layer in the shared database exceeds a preset probability threshold, if the probability exceeds the preset probability threshold, establishing a mapping relationship between the low-dimensional feature vector and the storage layer, mapping the low-dimensional feature vector to the storage layer, and completing the creation of the shared database.

[0144] To sum up, the present application provides a big data resource service method and system based on multi-modal data fusion, in the embodiment of the present application, the multi-modal data in the shared database is encrypted and distributed to the distributed hash table of the distributed node by using the point-to-point technology, and the data position is determined by using the hash function, thereby realizing efficient distributed storage and access, and combining the distributed server, the central virtual server and the communication scheduling layer, a resource scheduling model considering the multi-modal data distribution and transmission characteristics in the distributed environment is constructed, the collaborative learning and optimization between the distributed nodes are realized, the transmission path is dynamically optimized, the delay and energy consumption of data transmission are reduced, and the intelligent level of scheduling is improved.

[0145] It should be noted that, for the foregoing embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0146] The above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the protection scope of the present application. Obviously, the described examples are only some of the embodiments of the present application, but not all the embodiments. Based on these examples, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application. Although the present application is described in detail with reference to the above examples, those of ordinary skill in the art can still combine, add or delete the features in the embodiments of the present application according to the circumstances without creative work, so as to obtain different other technical solutions which do not deviate from the concept of the present application in essence, and these technical solutions also fall within the scope of the present application.

Claims

1. A big data resource service method based on multi-modal data fusion, characterized in that, The method comprises the following steps: Distributed acquisition of multi-modal data sequences collected by a data source, pre-processing of the multi-modal data sequences based on data fusion technology to obtain low-dimensional feature vector representation of the multi-modal data, and creation of a shared database based on the pre-processed low-dimensional feature vector; Based on point-to-point technology, the multi-modal data in the shared database is encrypted and distributed to the distributed hash table of the distributed node, and the data location of the multi-modal data in the distributed node is determined by a hash function; In response to a resource retrieval instruction for a topic, the resource retrieval instruction is identified based on a pre-built resource scheduling model, and a resource access control request is triggered; In response to the resource access control request, the distributed hash table of at least one group of distributed nodes is traversed based on cosine similarity to extract target data to be scheduled and target data attributes; The resource scheduling model dynamically deploys data transmission tasks based on target data attributes and real-time transmission path information, performs point-to-point encrypted transmission of the target data to be scheduled based on the data transmission tasks, and generates a retrieval result in combination with a large language model interaction; The resource scheduling model uses a federated learning model as an initial model, and the federated learning model is composed of a distributed server, a central virtual server, and a communication scheduling layer. The distributed server is a distributed node of the federated learning model. The distributed server and the central virtual server interact with each other through a transmission path. The distributed server is composed of an input layer, an ALBERT module, and a Transformer encoder. The ALBERT module introduces a flow control strategy and a quality of service mechanism. The central virtual server includes a content distribution network architecture CDN, a homomorphic encryption layer, and a differential privacy layer. The communication scheduling layer includes an adaptive multi-source multi-path congestion control module, a BLEST scheduler, and a fully connected model. The fully connected model introduces an ant colony optimization algorithm. 2.The big data resource service method based on multi-modal data fusion of claim 1, wherein: The method for pre-processing the multi-modal data sequence based on data fusion technology comprises the following steps: Load the multi-modal data sequence, delete the repeated data and abnormal data in the multi-modal data sequence, and perform numerical quantization type marking, attribute marking, dimension marking, and importance marking on the multi-modal data based on data type, data attribute, data dimension, and data importance to obtain the marked multi-modal data sequence; Obtain the marked multi-modal data sequence, define the pre-processing importance of the multi-modal data in the multi-modal data sequence based on a probability density function and a weighted average algorithm; Determine whether the pre-processing importance of the multi-modal data exceeds a preset importance threshold; If the pre-processing importance of the multi-modal data exceeds the preset importance threshold, pre-process the multi-modal data based on a pre-built joint adversarial model to represent the multi-modal data in a fine vector form; If the pre-processing importance of the multi-modal data does not exceed the preset importance threshold, pre-process the multi-modal data based on the data type of the multi-modal data to represent the multi-modal data in a coarse vector form; Load the multi-modal data represented by the fineness vectorization and the coarseness vectorization, splice the multi-modal data features represented by the fineness vectorization and the coarseness vectorization based on a multi-Transformer architecture combining self-attention and cross-attention data fusion technology, obtain high-dimensional fusion features, perform dimension reduction processing on the high-dimensional fusion features using PCA, reduce the feature dimension, and output normalized low-dimensional feature vectors; Obtain the low-dimensional feature vectors, predict the probability that the low-dimensional feature vectors correspond to the storage layer in the shared database based on a conditional random field (CRF), determine whether the probability that the low-dimensional feature vectors correspond to the storage layer in the shared database exceeds a preset probability threshold, and if so, establish a mapping relationship between the low-dimensional feature vectors and the storage layer, map the low-dimensional feature vectors to the storage layer, and complete the creation of the shared database. 3.The big data resource service method based on multi-modal data fusion of claim 2, wherein: The method for pre-processing the multi-modal data based on the pre-constructed joint adversarial model comprises: The joint adversarial model comprises an input layer, a type discriminator, an adaptive filter, a joint adversarial network, and an output layer, the input layer is connected to the type discriminator, the type discriminator is connected to the adaptive filter, the adaptive filter is connected to the joint adversarial network, and the joint adversarial network is connected to the output layer, the type discriminator identifies the type of the multi-modal data based on a decision tree algorithm, the adaptive filter comprises a multi-heterogeneous filter layer and a Kalman filter layer, and the joint adversarial network comprises an up-sampling layer, a full-dimensional dynamic convolution module, a BILSTM layer, and a pooling layer; Load the multi-modal data, identify the type of the multi-modal data based on the decision tree algorithm, and determine the type of the multi-modal data; Obtain the multi-modal data after type identification, cooperatively filter the multi-modal data based on the multi-heterogeneous filter layer and the Kalman filter layer, and mark and remove noise data in the multi-modal data by combining a three-frame difference detection algorithm and a Kalman filtering algorithm; The joint adversarial network performs up-sampling processing on the filtered multi-modal data, feeds back the up-sampling result to the full-dimensional dynamic convolution module, adjusts the size and shape of a convolution kernel using dynamic convolution according to the importance of the multi-modal data, calculates the attention weight of each dimension by using the attention of the data type, data attribute, data dimension, and data importance dimension, multiplies the attention weight and the convolution kernel to obtain a multi-modal data dynamic weight, and performs convolution operation on the multi-modal data dynamic weight and the up-sampling result to generate adaptive dynamic convolution. Load the adaptive dynamic convolution, perform forward and reverse LSTM operations on the adaptive dynamic convolution by the BILSTM layer to obtain forward and reverse hidden state vectors, combine the forward and reverse hidden state vectors and the adaptive dynamic convolution to obtain multi-modal data represented by the fineness vectorization. 4.The big data resource service method based on multi-modal data fusion of claim 3, wherein: The preprocessing importance of the multi-modal data is calculated by the following formula: （1） （2） wherein, denotes a pre-processing importance of the multi-modal data, is a standard deviation of the type label, the attribute label, the dimension label, the importance label, is a weighted average of the type label, the attribute label, the dimension label, the importance label, are respectively an input value and a mean value of the multi-modal data, are respectively a numerical quantification of the type label, the attribute label, the dimension label, the importance label; When predicting the probability that the low-dimensional feature vectors correspond to the storage layer in the shared database based on a conditional random field (CRF), the probability that the low-dimensional feature vectors correspond to the storage layer in the shared database is calculated by the following formula: （3） （4） （5） wherein, denotes a low-dimensional feature vector corresponding to a storage layer in the shared database of probabilities, are the number of low-dimensional feature vectors, the number of storage layers, are the feature vector matrix, the covariance, the mean of the low-dimensional feature vectors, respectively, denotes a low-dimensional feature vector corresponding to a storage layer in the shared database of similarities, denotes a property weight matrix of the storage layer , denotes a Gaussian kernel function, denotes a low-dimensional feature vector adjacency similarity matrix of the low-dimensional feature vector and the storage layer denotes a length scale parameter of the Gaussian kernel function. 5.The big data resource service method based on multi-modal data fusion of claim 1, wherein: The full connection model comprises Layer-1, Layer-2 and Layer-3, and the number of nodes of Layer-1, Layer-2 and Layer-3 is 250, 150 and 300 respectively. 6.The big data resource service method based on multi-modal data fusion of claim 5, wherein: The resource scheduling model training method comprises: Traverse the multi-modal data in the distributed hash table, divide the multi-modal data into a training set and a test set, and preset initial parameters, iteration rounds, maximum iteration rounds and a loss function of the resource scheduling model; Load the pre-constructed resource scheduling model, pre-train the distributed server using the training set, and obtain server parameters of the distributed server; Based on the content distribution network architecture CDN, import the server parameters of at least one group of distributed servers, and based on the adaptive multi-source multi-path congestion control algorithm combined with the ant colony optimization algorithm, dynamically aggregate the server parameters, calculate the path delay rate and the congestion window rate of the server parameters, and reserve the server parameters with a path delay rate and a congestion window rate addition value less than a preset cumulative threshold value, and perform weight pruning on the server parameters with a path delay rate and a congestion window rate addition value greater than or equal to the preset cumulative threshold value; Load the server parameters after weight pruning, set the server parameters after weight pruning as global parameters, deploy the global parameters in the distributed server, and obtain a converged resource scheduling model; Obtain the test set, input the test set into the converged resource scheduling model, dynamically deploy the test data transmission task based on the test set attribute and real-time transmission path information, and calculate the test set transmission response time; Determine whether the test set transmission response time meets the preset time threshold value, and if the test set transmission response time meets the preset time threshold value, output the converged resource scheduling model. The method for dynamically deploying a data transmission task based on target data attributes and real-time transmission path information by the resource scheduling model comprises: 7.The big data resource service method based on multi-modal data fusion of claim 5, wherein: Obtain target data attributes, index at least one group of distributed hash tables based on the target data attributes, create at least one group of transmission paths based on the adaptive multi-source multi-path congestion control algorithm, poll and allocate the target data to the transmission paths using a multi-path transmission protocol, identify real-time transmission path information corresponding to the transmission paths, and the real-time transmission path information comprises bandwidth, delay and packet loss rate; Generate at least one group of transmission path combinations by using the path with the lowest delay and the congestion window rate as constraints and combining the ant colony optimization algorithm. The method for dynamically deploying a data transmission task based on target data attributes and real-time transmission path information by the resource scheduling model further comprises: 8.The big data resource service method based on multi-modal data fusion of claim 7, wherein: Calculate the path delay rate and the congestion window rate of the transmission path combinations based on the adaptive multi-source multi-path congestion control algorithm, and the optimal transmission path combination is determined based on the minimum addition value of the path delay rate and the congestion window rate; In response to the optimal transmission path combination, dynamically deploy the data transmission task based on the optimal transmission path combination. The big data resource service system based on multi-modal data fusion comprises:

9. A big data resource service system based on multi-modal data fusion, for implementing the big data resource service method based on multi-modal data fusion according to any one of claims 1-8, characterized in that: ​ The database creation module is configured to acquire multi-modal data sequences collected by a data source in a distributed manner, pre-process the multi-modal data sequences based on a data fusion technology, and obtain multi-modal data represented by low-dimensional feature vectors, and create a shared database based on the pre-processed low-dimensional feature vectors. The distributed encryption module is configured to encrypt and distribute the multi-modal data in the shared database to a distributed hash table of the distributed nodes based on a peer-to-peer technology, and determine the data positions of the multi-modal data in the distributed nodes by using a hash function. The resource access module is configured to identify a resource retrieval instruction based on a pre-constructed resource scheduling model in response to the resource retrieval instruction for a topic, and trigger a resource access control request. The storage calling module is configured to extract target data to be scheduled and target data attributes by traversing the distributed hash table of at least one group of distributed nodes based on a cosine similarity in response to the resource access control request. The retrieval scheduling module is configured to dynamically deploy a data transmission task based on the target data attributes and real-time transmission path information, perform point-to-point encrypted transmission of the target data to be scheduled based on the data transmission task, and generate a retrieval result in combination with large language model interaction.

10. The big data resource service system based on multi-modal data fusion of claim 9, wherein: The database creation module includes: The data labeling unit is configured to delete repeated data and abnormal data in the multi-modal data sequences, and perform type labeling, attribute labeling, dimension labeling, and importance labeling on the multi-modal data based on data types, data attributes, data dimensions, and data importance. The importance calculation unit is configured to acquire the labeled multi-modal data sequences, and define a pre-processing importance of the multi-modal data in the multi-modal data sequences based on a probability density function and a weighted average algorithm. The importance judgment unit is configured to judge whether the pre-processing importance of the multi-modal data exceeds a preset importance threshold. The vector splicing unit is configured to load the multi-modal data represented by fine-grained vectorization and coarse-grained vectorization, splice features of the multi-modal data represented by fine-grained vectorization and coarse-grained vectorization based on a multi-Transformer architecture combining self-attention and cross-attention data fusion technology, obtain high-dimensional fusion features, perform dimension reduction processing on the high-dimensional fusion features by using PCA to reduce feature dimensions, and output normalized low-dimensional feature vectors. The data mapping storage unit is configured to acquire the low-dimensional feature vectors, predict a probability that the low-dimensional feature vectors correspond to a storage layer in the shared database based on a conditional random field (CRF), judge whether the probability that the low-dimensional feature vectors correspond to the storage layer in the shared database exceeds a preset probability threshold, and if so, establish a mapping relationship between the low-dimensional feature vectors and the storage layer, map the low-dimensional feature vectors to the storage layer, and complete creation of the shared database.

Images