An artificial intelligence-based cloud data model information processing method and system

Abstract

The application discloses a kind of cloud data model information processing method and system based on artificial intelligence, and the present application relates to the cross technical field of cloud computing and artificial intelligence, including the execution between the storage node and the computing node in cloud environment, including the following steps: S1: data pre-screening step: at the storage node, utilize a pre-trained lightweight prediction model to infer the data batch to be transmitted.This cloud data model information processing method and system based on artificial intelligence realizes pre-screening and multi-round verification on the key path from data storage to computing, reduces network transmission load from the source;Then through hybrid coding and distributed consistency verification, the integrity and reliability of the transmission data are ensured;Finally, through context association analysis, the data input into the training model have high value and high novelty, and this mechanism effectively overcomes the bottleneck caused by data I / O delay in cloud data model training.

Description

Technical Field

[0001] This invention relates to the field of cloud computing and artificial intelligence, specifically to a cloud data model information processing method and system based on artificial intelligence. Background Technology

[0002] In the field of cloud data model information processing, existing technologies typically rely on distributed cloud storage and computing nodes to support the training and inference of artificial intelligence models. However, this architecture exposes a core flaw when processing large-scale data: data input and output operations become the main bottleneck for system performance. Because training data is distributed across multiple cloud storage nodes, the process of transferring data from storage to computing involves high-latency network communication and complex data serialization and deserialization operations, significantly extending the model training cycle. Data loading time accounts for the majority of the training process, making model iteration speed far below ideal levels, failing to meet the real-time requirements of applications such as online learning or dynamic model updates. Existing solutions attempt to alleviate this problem through caching mechanisms or data prefetching techniques, but maintaining cache consistency is difficult, prefetching accuracy is insufficient, and it increases network traffic and resource overhead, failing to fundamentally solve the problems of poor data locality and low resource utilization. Therefore, cloud data model information processing urgently needs to overcome the limitations caused by data access latency. To address this, this invention, a cloud data model information processing method and system based on artificial intelligence, aims to solve the problem of low training efficiency caused by data input and output bottlenecks. Summary of the Invention

[0003] The purpose of this invention is to provide a cloud data model information processing method and system based on artificial intelligence, so as to solve the problems mentioned in the background art.

[0004] To address the aforementioned technical problems, this invention provides the following technical solution: a cloud data model information processing method and system based on artificial intelligence, wherein the method is executed between storage nodes and computing nodes in a cloud environment, and includes the following steps:

[0005] S1: Data pre-screening step: At the storage node, a pre-trained lightweight prediction model is used to infer the batch of data to be transmitted. The lightweight prediction model is configured to evaluate the value of the data batch to the current training state of the artificial intelligence model and generate a corresponding value score. The value score is compared with a dynamically adjusted threshold, and only data batches with a value score higher than the threshold are allowed to be selected to enter the subsequent transmission process.

[0006] S2: Data transmission and verification steps: For data batches that pass the pre-screening, a hybrid encoding scheme is used for encapsulation before transmission from the storage node to the computing node; after the data batch is transmitted to the computing node, decoding and data integrity verification are performed first, and then a distributed consistency verification process is started, which verifies the integrity and consistency of the transmitted data by comparing the data fingerprint with the source storage node and other data replica nodes.

[0007] S3: Data Value Confirmation and Injection Steps: Before being input into the training process of the artificial intelligence model, the verified data batches further pass through a context association gateway; the context association gateway analyzes the metadata information of the data batch and compares it with the current model's memory state and historical training trajectory in real time; based on the comparison results, at least one of the following operations is performed on the data batch: if it is determined that its information novelty is insufficient, it is fused with another small batch of data; if it is determined that it conforms to the current training direction, it is directly injected into the training process; finally, only data that successfully passes all the above steps is used to update the parameters of the artificial intelligence model.

[0008] Preferably, in the data pre-screening step, the lightweight prediction model is constructed and run in the following manner: the structural complexity of the lightweight prediction model is lower than that of the artificial intelligence model undergoing main training; its training task is associated with the main training model and is configured to learn and predict the potential contribution of different data features to the decrease of the loss function of the main training model; the dynamically adjusted threshold is periodically adjusted based on one or more factors, including the utilization rate of currently available computing resources on the computing node side and the network bandwidth status.

[0009] Preferably, the update mechanism of the lightweight prediction model is as follows: the parameters of the lightweight prediction model are not fixed, but are updated asynchronously offline or online according to the actual learning effect fed back by the main training model during the training process, so that the value evaluation capability of the lightweight prediction model can adapt to the dynamic evolution of the training state of the main training model.

[0010] Preferably, in the data transmission and verification steps, the hybrid encoding scheme combines forward error correction codes with compression encoding based on a specific data pattern; the distributed consistency verification process specifically includes: the computing node generating a data fingerprint of the received data batch and sending the data fingerprint to the source storage node and one or more other nodes holding a copy of the data batch; the source storage node and other nodes returning their stored corresponding data fingerprints to the computing node for comparison, and only when all comparison results are consistent is the data batch confirmed to have passed verification.

[0011] Preferably, the metadata analysis performed by the context association gateway includes: parsing the source identifier, timestamp, and statistical characteristics of the data batch; the real-time comparison with the current model memory state and historical training trajectory includes: calculating the similarity between the features of the data batch and the features of recently learned data batches, and evaluating the degree of matching with the current gradient descent direction of the model.

[0012] Preferably, in the data value confirmation and injection step, when the novelty of the data batch is deemed insufficient, the data fusion operation performed is as follows: the verified data batch is weighted and combined with a high-value small batch of data dynamically selected according to the current training state of the model to generate a new, enhanced data batch, which is then input into the training process.

[0013] An artificial intelligence-based cloud data model information processing system for performing the method, the system being deployed in a cloud environment, comprising:

[0014] One or more storage nodes are used to store training data;

[0015] One or more computing nodes are used to run training tasks for artificial intelligence models;

[0016] The system also includes a hierarchical data processing pipeline, which is logically deployed between the storage node and the computing node, including:

[0017] A data pre-screening module, configured on the storage node side, is used to carry and run the lightweight prediction model and execute the data pre-screening steps;

[0018] The data verification module, configured on the computing node side, is used to perform decoding of the hybrid encoding scheme, data integrity verification, and coordination of the distributed consistency verification process;

[0019] The context association gateway module, configured on the computing node side, interacts with the training engine of the artificial intelligence model to perform the data value confirmation and injection steps.

[0020] Preferably, the data pre-screening module further includes a threshold management unit, which is configured to continuously monitor the resource indicators and network status of the computing node, and dynamically calculate and update the dynamic threshold used for data screening according to a preset strategy.

[0021] Preferably, the data verification module communicates with the storage node and other replica nodes through a dedicated communication proxy to complete the data fingerprint request and comparison operation.

[0022] Preferably, the context association gateway module integrates a metadata parser and a state comparator; the metadata parser is used to extract metadata information of the incoming data batch; the state comparator is used to access the training state cache of the artificial intelligence model and execute the real-time comparison logic.

[0023] This invention provides a cloud data model information processing method and system based on artificial intelligence. It has the following beneficial effects:

[0024] This AI-based cloud data model information processing method and system constructs a hierarchical intelligent data processing pipeline, achieving pre-screening and multi-round verification along the critical path from storage to computation. The system first uses a lightweight model on the storage side to filter low-value data, reducing network transmission load at the source. Then, through hybrid encoding and distributed consistency verification, it ensures the integrity and reliability of transmitted data. Finally, through contextual analysis, it guarantees that the data input into the training model possesses both high value and high novelty. This mechanism effectively overcomes the bottleneck caused by data I / O latency in cloud data model training.

[0025] This AI-based cloud data model information processing method and system, through a closed-loop processing flow, systematically reduces the idle waiting time of computing nodes, improves the overall utilization of hardware resources, and accelerates the iteration cycle of model training. Simultaneously, because the data quality injected into the training process is strictly controlled, the model can obtain more effective learning signals with the same amount of data, thereby helping to improve the model's convergence speed and final performance, providing an efficient and reliable data supply path for large-scale cloud data model training. Attached Figure Description

[0026] Figure 1 This invention provides a data flow diagram between modules of a cloud data model information processing system based on artificial intelligence.

[0027] Figure 2 This invention provides a data batch state transition diagram for a cloud data model information processing system based on artificial intelligence.

[0028] Figure 3 This is a flowchart illustrating a cloud data model information processing method based on artificial intelligence according to the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Please see Figures 1 to 3 This invention provides a technical solution: a cloud data model information processing method and system based on artificial intelligence. The method is executed between storage nodes and computing nodes in a cloud environment, and includes the following steps:

[0031] S1: Data pre-screening step: At the storage node, a pre-trained lightweight prediction model is used to infer the batch of data to be transmitted. The lightweight prediction model is configured to evaluate the value of the data batch to the current training state of the artificial intelligence model and generate a corresponding value score. The value score is compared with a dynamically adjusted threshold, and only data batches with a value score higher than the threshold are allowed to be selected to enter the subsequent transmission process.

[0032] S2: Data transmission and verification steps: For data batches that pass the pre-screening, a hybrid encoding scheme is used for encapsulation before transmission from the storage node to the compute node; after the data batch is transmitted to the compute node, decoding and data integrity verification are performed first, and then a distributed consistency verification process is started. The integrity and consistency of the transmitted data are verified by comparing the data fingerprint with the source storage node and other data replica nodes.

[0033] S3: Data Value Confirmation and Injection Steps: Before being input into the training process of the artificial intelligence model, the verified data batches further pass through a context association gateway. The context association gateway analyzes the metadata information of the data batch and compares it with the current memory state and historical training trajectory of the model in real time. Based on the comparison results, at least one of the following operations is performed on the data batch: if it is determined that its information novelty is insufficient, it is fused with another small batch of data; if it is determined that it conforms to the current training direction, it is directly injected into the training process. Finally, only data that successfully passes all the above steps is used to update the parameters of the artificial intelligence model.

[0034] It should be further explained that the specific implementation involves a hierarchical intelligent data processing pipeline established between storage nodes and compute nodes in a cloud environment. This pipeline begins by deploying a simplified, lightweight neural network model on the storage node side. This model is pre-trained using historical training data, and its learning objective is set to predict the contribution of different data features to the reduction of the loss function during the main training model's training task, i.e., evaluating the data value. During model runtime, forward inference is performed on each batch of data to be read, outputting a quantified value score. The system compares this score with a dynamic threshold, which is calculated by combining the current GPU utilization and available network bandwidth periodically fed back from the compute node side. This achieves the filtering of low-value data at the source, directly reducing network transmission load.

[0035] For the data that passes the screening, the system encapsulates it before transmission using a hybrid scheme that combines Reed-Solomon forward error correction codes with data pattern dictionary encoding to enhance transmission robustness.

[0036] It should be further explained that the hybrid coding scheme combines Reed-Solomon (RS) forward error correction codes with LZ77 compression coding, and the specific implementation is as follows:

[0037] The Reed-Solomon forward error correction code parameter configuration includes: using RS(n,k) code, where the symbol length m=8, that is, each symbol is 1 byte, the code length n=255, each segment of encoded data contains 255 symbols, the number of information symbols k=223, the number of actual valid data symbols in each segment of encoded data, the number of check symbols r=nk=32, and the number of redundant symbols used for error correction;

[0038] Error correction capability: It can correct up to 16 random error symbols or 32 consecutive error symbols in each segment of coded data, meeting the fault tolerance requirements of network transmission in cloud environments;

[0039] Encoding polynomial: using GF(2) 8 The primitive polynomial p(x) = x over a finite field 8 +x 4 +x 3 +x 2 +1 ensures the uniqueness and reproducibility of the encoding process.

[0040] The LZ77 compression encoding parameters are configured as follows:

[0041] Sliding window structure: includes a history buffer and a lookup buffer; the history buffer is 4096 bytes in size and stores processed data; the lookup buffer is 1024 bytes in size and stores data to be processed.

[0042] Matching rules: Find the longest matching sequence in the history buffer that matches the starting data of the lookahead buffer. The matching length ranges from [3, 258] bytes. When the matching length is ≥3, it is compressed; otherwise, a single byte is output directly.

[0043] Encoded output: Each matching sequence outputs a tuple of "(offset, length)", where the offset represents the position of the matching sequence in the history buffer, ranging from [0, 4095]; the length represents the number of bytes in the matching sequence, and unmatched data is output directly as raw bytes.

[0044] The hybrid encoding and decoding process includes the following:

[0045] Encoding process: For the pre-screened batches of data, the storage node first performs LZ77 compression, processing the data according to the parameters mentioned above to obtain compressed data; then, the compressed data is segmented into 223-byte segments, and RS(255, 223) encoding is performed on each segment, with a 32-byte checksum added to form an encoding unit of "compressed data segment + RS checksum segment"; finally, all encoding units are concatenated, and header information including data length and encoding parameter identifiers is added to complete the encapsulation; the encoding parameter identifiers include RS code parameters and LZ77 window size;

[0046] Decoding process: After receiving the encoded data, the computing node first parses the header information to confirm the encoding parameters; then it splits the encoded data into "compressed data segment + RS check segment" units, performs RS decoding on each unit, uses 32-byte check symbols to correct transmission errors, and recovers the 223-byte compressed data segment; finally, it performs LZ77 decoding on the recovered compressed data segment, reverses the parsing of the "(offset, length)" tuple according to the sliding window parameters, and restores the original data batch.

[0047] After the data arrives at the computing node, it is first decoded and cyclic redundancy check is performed. Then, a distributed data consistency verification process is triggered: the computing node uses the SHA-256 algorithm to generate a fingerprint of the received data and sends this fingerprint to the source storage node and at least one auxiliary node holding a copy of the data. Each party returns the original data fingerprint stored in its own storage for comparison. Only if all comparisons are consistent is the data batch considered valid and enters the next stage.

[0048] After that, the valid data is sent to a software module called the context association gateway.

[0049] It should be further explained that the metadata analysis and state comparison performed by the context association gateway are calculated using the following method:

[0050] The metadata analysis content is as follows:

[0051] Source identifier resolution: Extract the source ID from the header of the data batch, such as the storage node IP address + dataset number, in the format "192.168.1.100_DS001", for tracing the source of the data;

[0052] Timestamp parsing: Extract the data collection timestamp in the format "YYYY-MM-DDHH:MM:SS.ms", accurate to milliseconds. Calculate the time difference Δt between the data batch and the current training time to help determine the timeliness of the data. Data batches with Δt > 72 hours need to have their novelty criteria increased.

[0053] Data distribution statistical characteristics: Calculate the mean μ of the batch feature matrix of the data, with dimension F, where F is the feature dimension, using the formula: x i，j Let be the j-th eigenvalue and variance σ of the i-th sample. 2 ,formula: This serves as the basis for subsequent comparisons.

[0054] Information novelty comparison is performed using cosine similarity calculation as follows:

[0055] Historical Feature Cache: The training state cache of the main training model stores the mean values ​​of statistical features {μ1, μ2, ..., μ3} of the most recent 100 batches of trained data. 100}, each μ k The dimension is F;

[0056] Cosine similarity formula: Calculates the mean μ of the current batch of data. curr Compared with historical characteristic mean μ k cosine similarity sim k ,formula:

[0057] , where “·” represents the vector dot product and ||·||2 represents the L2 norm;

[0058] Novelty determination: Calculate the average similarity:

[0059] If sim avg <0.6, is considered "sufficiently novel"; if sim avg A score ≥0.6 is considered "insufficient novelty". 0.6 is a threshold verified through numerous experiments, balancing data novelty and training stability.

[0060] The training direction matching degree is calculated and compared using gradient dot product as follows:

[0061] Gradient vector acquisition: Obtain the parameter gradient vector G for the current round from the main training model training engine. The dimension is P, where P is the total number of parameters in the main training model, calculated through backpropagation of the main training model. Each element of G... p This represents the gradient value of the p-th parameter;

[0062] Hypothesis gradient calculation: Input the current data batch into the forward propagation process of the main training model, and calculate the hypothesis gradient vector G induced by this data batch. curr Dimension P, formula:

[0063] θ p Loss of the p-th parameter of the main training model main (x i ) is the sample x i Cross-entropy loss;

[0064] Gradient matching formula: Calculate G and G curr The cosine similarity, i.e., the gradient matching degree: ;

[0065] Match determination: If match > 0.5, it is determined to be "in line with the current training direction" and directly injected into the training process; if match ≤ 0.5, it needs to be combined with the novelty determination result. If the novelty is insufficient, data fusion is performed. If the novelty is sufficient, the learning rate is adjusted and reduced by 5% before being injected into the training process to avoid conflict with the current gradient direction and cause training oscillation.

[0066] This module first parses the data's metadata, including its source identifier, collection timestamp, and statistical moments of the feature vector distribution. Then, it compares this metadata with the state cache maintained by the main training model's training engine, which records feature summaries and model parameter update trajectories from recent training data batches. The gateway's core algorithm calculates the cosine similarity between the current data and recent data features and evaluates the matching degree between its feature distribution and the model's current gradient direction. If the data is deemed insufficiently novel, it is weighted and concatenated with a high-value mini-batch dynamically selected based on the model's current perplexity to generate an enhanced batch.

[0067] It should be further explained that when the context association gateway determines that the novelty of the data batch information is insufficient, the weighted combination operation performed is as follows:

[0068] The calculation basis and formula for the weighting coefficients include: defining the two data batches to be merged: master data batch D main High-value small batch D high Among them, master data batch D main The number of data batches that passed verification but lacked novelty was N. main High-value small batch Dhigh The number of samples is N, selected based on the current training state of the main training model. high And N high =1 / 5N main , ensure D high Do not dominate the fusion outcome;

[0069] The perplexity (PPL) of the current training model is introduced as the basis for adjusting the weighting coefficients. The perplexity is used to quantify the difficulty of the model in understanding the data. The lower the PPL value, the more fully the model understands the data. The value range is [10, 1000].

[0070] Master data batch weight w main Weight w for high-value small batches high The calculation formula is as follows:

[0071] w main =1-w high ,

[0072] Among them PPL min =10, which is the ideal perplexity in the later stages of model training, PPL max =1000, high perplexity in the early stage of model training, with weight values ​​ranging from [0.3, 0.7] to avoid information loss due to excessively low weights in a single batch;

[0073] How to calculate perplexity:

[0074] Where M is the number of samples in the current training batch of the main training model, and Loss is... main (x j The main training model is used for sample x j The cross-entropy loss value is output in real time by the main training model training engine.

[0075] The specific process of data fusion includes the following:

[0076] Feature alignment: Ensure D main With D high If the feature dimensions are consistent, then D is mapped using a linear mapping layer. high Feature dimensions mapped to D main The feature dimension and mapping matrix are initialized by the feature projection parameters of the previous training round of the main training model;

[0077] Weighted combination: For the corresponding feature vectors x in the two batches main，i With x high，i Calculate the fusion feature x according to the formula fusion，i =w main ×x main，i +whigh ×x high，i i = 1, 2, ..., N main , where the feature vector x main，i D main The i-th sample feature, x high，i D high The i-th sample feature;

[0078] Normalization: L2 normalization is performed on the fused feature matrix, with the L2 norm of each row of feature vectors set to 1, to avoid excessive differences in feature values ​​affecting model training; the fused feature matrix has a dimension of N. main ×F, where F is the feature dimension;

[0079] Tag inheritance: If the data batch contains tag information, the tags of the merged data are inherited from D. main The labels are used to ensure label consistency for the training task, and after fusion, augmented data batch D is generated. fusion Injected into the training process of the main training model.

[0080] If a good match is found, the data is directly injected into the training process. Ultimately, only data that successfully passes through this triple-checking, filtering, and enhancement mechanism is used for model weight updates, thereby fundamentally improving training efficiency by ensuring that each computation is applied to high-value, high-completeness data.

[0081] It should be further noted that the lightweight prediction model adopts a four-layer structure of "input layer - feature extraction layer - fully connected layer - output layer", with the specific parameter configuration as follows:

[0082] Input layer: Receives feature vectors of the batch of data to be filtered. The feature dimension is consistent with the input feature dimension of the main training model. If the main training model processes image data, the input layer dimension is 3×224×224, which corresponds to the flattened vector of a 224×224 pixel image with three RGB channels.

[0083] Feature extraction layer: A single 2D convolutional layer is used with 16 kernels, a kernel size of 3×3, a stride of 1, and padding of "same" to ensure that the output feature map size is consistent with the input. The activation function is ReLU, with the formula: f(x)=max(0,x), used to extract key local features of the data batch.

[0084] Fully connected layer: Contains 64 neurons, and the activation function is also the ReLU function. The feature map output by the convolutional layer is flattened and then non-linearly mapped to achieve global integration of features.

[0085] Output layer: Contains 1 neuron, using the sigmoid function as the activation function, formula: The output range is [0, 1], representing the value score of the data batch. The training process and loss function selection for the lightweight prediction model are as follows:

[0086] Training data: The training set consists of 10,000 historical data batches that have been labeled with the "contribution value of loss function decline" during the historical training process of the main training model. These batches are the decrease magnitude ΔLoss of the loss function of the main training model after the historical data batches are input into the main training model. The training set and validation set are divided into training set and validation set in an 8:2 ratio.

[0087] Loss function: The mean squared error loss function (MSE) formula is used.

[0088] , where y i This represents the normalized value of the actual ΔLoss for the historical data batch, normalized to [0, 1]. The value score output by the lightweight model, where N is the number of training batches;

[0089] Training optimizer: The Adam optimizer is used, with a learning rate of 1e-4, a batch size of 32, and 50 training epochs. Training stops when the validation set loss is stable. Loss stability is determined by the validation loss fluctuation being less than 1e-5 for 5 consecutive epochs.

[0090] Update mechanism: An online asynchronous update method is adopted. Every 10 training cycles of the main training model, that is, after the main training model completes 10 parameter updates, the latest 100 data batches of "actual ΔLoss" are obtained from the main training model training engine and used as incremental training data to fine-tune the parameters of the lightweight model. The fine-tuning rounds are 5, and the learning rate is reduced to 5e-5 to ensure that the value evaluation capability of the lightweight model evolves synchronously with the training state of the main training model.

[0091] It should be further explained that the dynamically adjusted threshold is calculated by the threshold management unit through the following strategy function, specifically implemented as follows:

[0092] Monitoring parameter definition: Let the GPU util The real-time utilization of the GPU on the compute node side, with a value ranging from [0, 1], is collected by the compute node operating system using the NVIDIA-SMI tool, with a sampling period of 5 seconds; let B... avail The real-time available network bandwidth between storage nodes and compute nodes is collected by the network monitoring module using the iperf tool, with a sampling period of 5 seconds, and then normalized. B min The preset minimum effective bandwidth is 50Mbps; B max The preset maximum ideal bandwidth is set to 500Mbps, B. normThe value range is [0, 1];

[0093] The threshold calculation formula includes: The dynamic threshold T is calculated using a weighted summation formula:

[0094] Where: α is the weighting coefficient of GPU utilization, with a value of 0.4, representing the degree of influence of GPU load on the threshold; β is the weighting coefficient of network bandwidth, with a value of 0.5, representing the degree of influence of bandwidth status on the threshold; T0 is the base threshold, with a value of 0.3, ensuring that even when resources are sufficient, extremely low-value data can be filtered out; the value range of the threshold T is limited to [0.2, 0.8], implemented through the clip function: T=clip(T, 0.2, 0.8), to avoid the threshold being too high, causing valid data to be filtered out, or too low, causing the filtering to fail;

[0095] The following is an example of adjustment: When GPU utilization is high... util =0.8, which means high load and available bandwidth B avail =80Mbps, B norm When the value is 0.067, T = 0.4 × 0.8 + 0.5 × (1 - 0.067) + 0.3 ≈ 0.32 + 0.4665 + 0.3 = 1.086. After clipping, T = 0.8, which means that the data is strictly screened and only high-value data passes through.

[0096] When GPU utilization util =0.2, meaning low load and available bandwidth B avail =400Mbps, B norm When the value is 0.778, T = 0.4 × 0.2 + 0.5 × (1 - 0.778) + 0.3 ≈ 0.08 + 0.111 + 0.3 = 0.491. No clipping is needed, meaning moderate filtering is used to retain more potentially valuable data.

[0097] In the data pre-screening step, the lightweight prediction model is built and run in the following way: the structural complexity of the lightweight prediction model is lower than that of the AI ​​model undergoing main training; its training task is associated with the main training model and is configured to learn and predict the potential contribution of different data features to the reduction of the loss function of the main training model; the dynamically adjusted threshold is periodically adjusted based on one or more factors, such as the utilization rate of currently available computing resources on the computing node side and the network bandwidth status.

[0098] It should be further explained that the lightweight prediction model cited is constructed and operated as follows: This model employs a simplified artificial neural network structure with significantly fewer layers and neurons per layer compared to the main training model. For example, it contains only one convolutional layer and one fully connected layer, while the main training model might be a deep residual network. The training task of this lightweight model is closely related to that of the main training model. Its training objective is to learn the potential contribution of different data features to the reduction of the loss function of the main training model. During the training phase, historical training data and the corresponding reduction in the loss of the main training model are used as labels. Through supervised learning, the lightweight model learns to output a score reflecting the potential training value of the data. The dynamically adjusted threshold is not a fixed value but is managed by an independent threshold management unit. This unit periodically collects data on the utilization rate of the current graphics processing unit and the available bandwidth of the network interface from the computing node. Based on a predefined policy function, it maps these two resource indicators to a specific threshold value, thereby dynamically optimizing the strictness of data filtering according to the real-time system load.

[0099] The update mechanism of the lightweight prediction model is as follows: the parameters of the lightweight prediction model are not fixed, but are updated asynchronously, either offline or online, based on the actual learning effect fed back by the main training model during the training process. This allows the value evaluation capability of the lightweight prediction model to adapt to the dynamic evolution of the training state of the main training model.

[0100] It should be further explained that the lightweight prediction model update mechanism involved is implemented in that the model's parameters have the ability to evolve dynamically, rather than remaining fixed after initial training. This update process is coordinated and executed through an independent model parameter management service, which periodically collects feedback signals from the training engine of the main training model. These signals specifically include the deviation between the actual decrease curve of the loss function of the main training model and the expected value after using selected batches of data in recent training cycles, as well as the magnitude and direction of the main training model's weight updates.

[0101] In offline update mode, the parameter management service accumulates a certain amount of feedback data and then initiates a retraining task for a lightweight model. Using the collected feedback information from the main training model as a supervisory signal, the weights of the lightweight model are fine-tuned to match its value evaluation criteria with the current learning stage of the main training model. In online update mode, an incremental learning algorithm is employed. The real-time feedback from the main training model is used as a continuous adjustment signal. By calculating the gradient of the value prediction error, small-scale, high-frequency adjustments are made to some layer parameters of the lightweight model. This two-way feedback and adaptation mechanism ensures that the decision-making basis for data pre-screening can self-optimize along with the dynamic progress of the core training task, forming a continuously improving intelligent data filtering closed loop.

[0102] In the data transmission and verification steps, the hybrid coding scheme combines forward error correction codes with compression coding based on a specific data pattern. The distributed consistency verification process specifically includes: the computing node generates a data fingerprint of the received data batch and sends the data fingerprint to the source storage node and one or more other nodes holding a copy of the data batch; the source storage node and other nodes return the corresponding data fingerprint stored by themselves to the computing node for comparison, and the data batch is confirmed to have passed the verification only when all comparison results are consistent.

[0103] It should be further explained that the specific implementation of the defined data transmission and verification steps involves a carefully designed hybrid encoding and distributed verification mechanism. The hybrid encoding scheme performs two operations in parallel during encapsulation: first, Reed-Solomon forward error correction codes are applied to generate redundant check blocks, enabling them to correct burst errors during transmission; simultaneously, based on the structured characteristics of the data content itself, a sliding window dictionary-based compression encoding algorithm is used to process the original data. This algorithm can identify and compress recurring byte patterns in the data. The data streams after these two encoding processes are multiplexed into a complete transmission unit, thereby enhancing reliability while reducing network load.

[0104] After data is successfully delivered to the compute nodes and decoded and preliminarily checked for cyclic redundancy, a distributed consistency verification process is triggered. This process is managed by a verification coordinator running on the compute nodes. The coordinator first uses the SHA-256 secure hash algorithm to calculate a unique digital fingerprint on the received complete data batch. Then, the coordinator sends this digital fingerprint simultaneously to the source storage node and at least one other node pre-configured in the system that holds a copy of the data batch via a remote procedure call. These nodes perform the same SHA-256 calculation on the stored original data locally and immediately return their respective digital fingerprints. The coordinator collects all returned fingerprints and compares them byte-by-byte. Only when the fingerprints returned by all participating nodes are completely consistent with the fingerprint generated by the compute nodes is the data batch finally confirmed to have not suffered any silent corruption or tampering during transmission and storage, and is allowed to proceed to subsequent processing stages. This rigorous multi-node cross-validation mechanism fundamentally guarantees the absolute integrity of the data input into the training process.

[0105] The metadata analysis performed by the context association gateway includes: parsing the source identifier, timestamp, and statistical characteristics of the data batch; and real-time comparison with the current model memory state and historical training trajectory, including: calculating the similarity between the features of the data batch and the features of recently learned data batches, and evaluating the degree of matching with the current gradient descent direction of the model.

[0106] It should be further explained that the specific implementation of the defined data value confirmation and injection steps relies on the metadata parsing and state comparison functions integrated within the context association gateway. For each incoming data batch, the metadata parser first extracts its source identifier to clarify the data source, parses its timestamp information to establish a temporal context, and calculates the statistical moments of its numerical feature vectors to quantify the data distribution characteristics. The state comparator maintains a real-time state cache that deeply interacts with the main training model. This cache continuously records and updates the core feature vector summaries of recently consumed data batches and the update trajectory of model parameters in recent iterations. The core algorithm of the comparison operation includes two parallel evaluation processes: first, calculating the cosine similarity between the feature vector of the current data batch and the feature vectors of recent data in the cache to quantify its information novelty; second, evaluating its matching degree with the model's most pressing learning needs by analyzing the gradient tensor direction of the current model's weight matrix and calculating the dot product between this gradient direction and the hypothetical gradient direction induced by the features of the current data batch. This dual evaluation mechanism provides a quantitative, direct basis based on the model's internal state for subsequent data routing decisions.

[0107] In the data value confirmation and injection step, when the information novelty of a data batch is deemed insufficient, the data fusion operation performed is as follows: the data batch that has passed verification is weighted and combined with a high-value small batch of data dynamically selected according to the current training state of the model to generate a new, enhanced data batch, which is then input into the training process.

[0108] It should be further explained that the specific implementation of the defined data fusion operation is as follows: When the context association gateway determines that the novelty of the current data batch is lower than a preset threshold based on a dual evaluation of cosine similarity and gradient direction matching, the system does not simply discard the batch, but instead initiates a data augmentation process. This process first dynamically selects a high-value small batch of data from the model training state cache. This selection is based on records where the small batch of data has caused a significant decrease in the model's loss function during recent training, or the characteristic that its feature distribution has a high degree of alignment with the current model's gradient direction.

[0109] Subsequently, the system performs a weighted combination of the currently verified but insufficiently novel data batch with this high-value small batch of data. This combination is not a simple splicing, but rather the data fusion unit calculates a set of adaptive weighting coefficients based on the feature statistics of each batch and the current training perplexity of the model. These coefficients are then applied to the corresponding feature vectors in the two batches, thereby fusing them to generate a new composite data batch with enhanced information density and training potential. This composite batch is then injected into the training process, aiming to improve the contribution of a single data input to model optimization by replacing quantity with quality.

[0110] An AI-based cloud data model information processing system for executing methods, the system being deployed in a cloud environment, includes:

[0111] One or more storage nodes are used to store training data;

[0112] One or more computing nodes are used to run training tasks for artificial intelligence models;

[0113] The system also includes a hierarchical data processing pipeline, which is logically deployed between storage nodes and compute nodes, including:

[0114] The data pre-screening module, configured on the storage node side, is used to host and run a lightweight prediction model and perform data pre-screening steps;

[0115] The data verification module, configured on the computing node side, is used to perform decoding of the hybrid encoding scheme, data integrity verification, and coordinate the distributed consistency verification process.

[0116] The context association gateway module, configured on the computing node side, interacts with the training engine of the artificial intelligence model to perform data value verification and injection steps.

[0117] It should be further explained that the protected cloud data model information processing system based on artificial intelligence has the following specific hardware architecture and software module deployment: The system physically includes one or more storage nodes and one or more computing nodes interconnected by a network, wherein the storage nodes are servers with large-capacity hard drives used to store the original training dataset; the computing nodes are equipped with graphics processing units used to carry out the training tasks of the main artificial intelligence model.

[0118] The hierarchical data processing pipeline serves as the core software framework, logically bridging the storage and compute nodes. On the storage node side, it is instantiated as a data pre-screening module. This module is integrated into the data access interface of the storage node as software middleware, directly calling and running a lightweight predictive model to perform localized data value assessment and screening.

[0119] On the compute node side, a data verification module and a context association gateway module are deployed. The data verification module, as an independent daemon process, is responsible for processing the data stream transmitted from the storage node, performing decoding, verification, and management of the distributed consistency verification process with the remote node. The context association gateway module is tightly integrated with the main training engine in the form of a kernel extension or library, directly accessing the memory state and parameter update history of the training engine, performing fine-grained metadata parsing and real-time state comparison, thereby controlling the final injection of data into the training engine.

[0120] These three modules communicate and exchange data through predefined network protocols and application programming interfaces, together forming a closed-loop intelligent data supply chain from the data source to the training core.

[0121] The data pre-screening module further includes a threshold management unit, which is configured to continuously monitor the resource metrics and network status of the computing nodes, and dynamically calculate and update the dynamic thresholds used for data screening according to a preset strategy. It should be further noted that the threshold management unit is integrated within the data pre-screening module and runs continuously as an independent control thread. This unit periodically collects the current utilization percentage of the graphics processing unit and the instantaneous available bandwidth data of the network interface controller by calling the performance monitoring interface provided by the computing node's operating system.

[0122] These raw resource metrics are fed into a built-in policy function that is pre-programmed with a weighted decision logic. For example, when GPU utilization is above a certain level and available bandwidth is below a certain level, the function outputs a higher threshold to implement stricter data filtering, and vice versa.

[0123] The specific implementation of this strategy function can be a set of conditional judgment rules or a small linear regression model. Its core function is to map the multi-dimensional and dynamically changing system resource status to a single threshold value for data value comparison, thereby realizing the precise linkage and adaptive adjustment between the data pre-screening strategy and the actual load of the underlying infrastructure.

[0124] The data verification module communicates with the storage nodes and other replica nodes through a dedicated communication proxy to complete the data fingerprint request and comparison operations. It should be further noted that the dedicated communication proxy is specifically implemented as a highly optimized network communication subsystem within the data verification module. This proxy is built on an asynchronous non-blocking input / output model and implemented using a high-performance network framework. Its core responsibility is to efficiently and reliably execute cross-node communication in the distributed consistency verification process.

[0125] Once the data verification module generates a digital fingerprint of the batch of data to be verified, the communication agent will simultaneously send remote call requests to the source storage node and at least one preset replica node; these requests carry the digital fingerprint information to be compared and set a reasonable network timeout period.

[0126] The communication broker maintains a lightweight request status table to track all requests that have been sent but for which no response has yet been received. Upon receiving a response from any node, the broker immediately compares the returned fingerprint with the locally calculated result and updates the status table.

[0127] The agent also incorporates basic fault-tolerance logic. For example, for nodes that do not respond within the timeout period, it will attempt a retry or temporarily mark them as unavailable according to a preset policy to ensure that the verification process is not completely blocked due to the temporary failure of a single node. Through this centralized and efficient communication management, the agent ensures that the distributed verification process can be completed smoothly with low overhead and latency.

[0128] The context association gateway module integrates a metadata parser and a state comparator; the metadata parser is used to extract metadata information of incoming data batches; the state comparator is used to access the training state cache of the artificial intelligence model and perform real-time comparison logic.

[0129] It should be further explained that the defined context-related gateway module is internally composed of two core components: a metadata parser and a state comparator. The metadata parser is implemented as a high-efficiency data stream processing unit. It first parses the format of each incoming batch of data, identifies and extracts the source identifier string embedded in the data header and the timestamp value accurate to milliseconds. Then, it performs online calculations on the numerical feature vector of the data body to obtain its distribution characteristics such as mean, variance, and higher-order statistical moments in real time.

[0130] The state comparator is designed as a dedicated processor with direct access to the training engine's memory space. It continuously retrieves the latest maintained array of recent data feature summaries and the model parameter update trajectory matrix from the training engine's state cache via a shared memory interface or direct memory access technology. In practice, the state comparator calls its embedded vector operation unit to first calculate the cosine similarity scalar value between the current data feature and the feature summary in the cache. Simultaneously, it evaluates the matching degree scalar value between the hypothesized gradient direction induced by the current data feature and the actual gradient tensor direction of the model through matrix multiplication and projection analysis. These two scalar values ​​together constitute the decision vector for data routing, driving subsequent data fusion or direct injection operations.

[0131] This technical solution constructs a hierarchical intelligent data processing pipeline, achieving pre-screening and multi-round verification along the critical path from storage to computation. The system first uses a lightweight model on the storage side to filter low-value data, reducing network transmission load at the source. Then, through hybrid encoding and distributed consistency verification, it ensures the integrity and reliability of transmitted data. Finally, through contextual analysis, it guarantees that the data input into the training model possesses both high value and high novelty. This mechanism effectively overcomes the bottleneck caused by data I / O latency in cloud data model training.

[0132] Through the closed-loop processing described above, this solution can systematically reduce the idle waiting time of computing nodes, improve the overall utilization of hardware resources, and accelerate the iteration cycle of model training. Simultaneously, because the data quality injected into the training process is strictly controlled, the model can obtain more effective learning signals with the same amount of data, thereby helping to improve the model's convergence speed and final performance, and providing an efficient and reliable data supply path for large-scale cloud data model training.

[0133] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0134] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An artificial intelligence-based cloud data model information processing method, characterized by, The method is performed between a storage node and a computing node in a cloud environment, comprising the following steps: S1: data pre-screening step: at the storage node, a pre-trained lightweight prediction model is used to infer the data batch to be transmitted, the lightweight prediction model is configured to evaluate the value of the data batch for the current artificial intelligence model training state, and a corresponding value score is generated; the value score is compared with a dynamically adjusted threshold value, and only the data batch with a value score higher than the threshold value is allowed to be selected into the subsequent transmission process; S2: data transmission and verification step: for the data batch that passes the pre-screening, a hybrid encoding scheme is used for packaging before transmission from the storage node to the computing node; after the data batch is transmitted to the computing node, decoding and data integrity check are performed first, and then a distributed consistency verification process is started to verify the integrity and consistency of the data after transmission through data fingerprint comparison with the source storage node and other data replica nodes; S3: data value confirmation and injection step: the verified data batch is further passed through a context association gateway before being input into the training process of the artificial intelligence model; the context association gateway analyzes the metadata information of the data batch and compares it with the current model memory state and historical training trajectory in real time; based on the comparison result, at least one of the following operations is performed on the data batch: if it is determined that the information novelty is insufficient, it is fused with another small batch of data; if it is determined that it meets the current training direction, it is directly injected into the training process; finally, only the successfully processed data is used to update the parameters of the artificial intelligence model. 2.The cloud data model information processing method based on artificial intelligence according to claim 1, characterized in that: In the data pre-screening step, the lightweight prediction model is constructed and run in the following way: the structural complexity of the lightweight prediction model is lower than that of the artificial intelligence model being trained; its training task is associated with the main training model, and it is configured to learn and predict the contribution potential of different data features to the loss function of the main training model; the dynamically adjusted threshold value is periodically adjusted based on one or more factors such as the current available computing resource utilization and network bandwidth status of the computing node side. 3.The AI-based cloud data model information processing method of claim 2, wherein: The updating mechanism of the lightweight prediction model is that the parameters of the lightweight prediction model are not fixed, but are updated asynchronously in an offline or online manner according to the actual learning effect feedback from the main training model during training, so that the value evaluation ability of the lightweight prediction model can adapt to the dynamic evolution of the training state of the main training model.

4. The artificial intelligence-based cloud data model information processing method of claim 1, characterized in that: In the data transmission and verification step, the hybrid encoding scheme combines forward error correction code and compression encoding based on specific data patterns; The distributed consistency verification process specifically includes that the computing node generates a data fingerprint of the received data batch, and sends the data fingerprint to the source storage node and one or more other nodes holding a copy of the data batch; the source storage node and the other nodes return the corresponding data fingerprints stored by themselves to the computing node for comparison, and only when all comparison results are consistent, the data batch is confirmed to pass the verification. 5.The AI-based cloud data model information processing method of claim 1, wherein: The metadata analysis performed by the context-related gateway includes: parsing the source identification, timestamp, and data distribution statistical characteristics of the data batch; the real-time comparison with the current model memory state and historical training trajectory includes: calculating the similarity of the data batch characteristics and the characteristics of the recently learned data batch, and evaluating the matching degree with the current gradient descent direction of the model. 6.The AI-based cloud data model information processing method of claim 5, wherein: In the data value confirmation and injection step, when it is determined that the information novelty of the data batch is insufficient, the data fusion operation performed is: weighting and combining the verified data batch with a high-value small batch of data dynamically selected according to the current training state of the model to generate a new, enhanced data batch, which is then input to the training process.

7. An artificial intelligence-based cloud data model information processing system for implementing an artificial intelligence-based cloud data model information processing method according to any one of claims 1 to 6, characterized in that, The system is deployed in a cloud environment, including: one or more storage nodes for storing training data; one or more computing nodes for running the training task of the artificial intelligence model; The system further includes a hierarchical data processing pipeline logically deployed between the storage nodes and the computing nodes, including: a data pre-screening module configured on the storage node side for carrying and running the lightweight prediction model to perform the data pre-screening step; a data verification module configured on the computing node side for decoding the hybrid encoding scheme, data integrity checking, and coordinating the distributed consistency verification process; a context-related gateway module configured on the computing node side and interacting with the training engine of the artificial intelligence model for performing the data value confirmation and injection step. 8.The cloud data model information processing system based on artificial intelligence of claim 7, wherein: The data pre-screening module further includes a threshold management unit configured to continuously monitor the resource indicators and network status of the computing node, and dynamically calculate and update the dynamic threshold for data screening according to a preset strategy. 9.The cloud data model information processing system based on artificial intelligence of claim 7, wherein: The data verification module communicates with the storage node and other copy nodes through a dedicated communication agent to complete the request and comparison operation of the data fingerprint. 10.The cloud data model information processing system based on artificial intelligence of claim 7, wherein: The context-related gateway module integrates a metadata parser and a state comparator; the metadata parser is used to extract the metadata information of the incoming data batch; the state comparator is used to access the training state cache of the artificial intelligence model and perform the real-time comparison logic.

Images