Data processing method and system based on cloud computing

Abstract

The invention provides a data processing method and system based on cloud computing, belongs to the technical field of data processing, and realizes high concurrent processing and automatic load balancing by dynamically scheduling computing power through elastic resource allocation and a distributed computing framework and combining a storage and computing separation framework. The system supports cross-node disaster recovery backup and multi-layer encryption, and data security is guaranteed; and an on-demand payment mode is adopted, so that the hardware input cost is reduced, the resource utilization rate is improved, and the large-scale data processing efficiency and the system expandability are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method and system based on cloud computing. Background Art

[0002] Existing cloud computing-based data processing methods and systems have the following problems:

[0003] 1. Real-time resource scheduling conflicts under extreme dynamic loads. In bursty traffic scenarios (such as flash sales and disaster warnings), existing elastic scaling algorithms rely on historical metrics (CPU / memory), making it difficult to achieve millisecond-level resource response.

[0004] 2. Heterogeneous hardware such as GUP / TPU quantum computing units lack cross-architecture instruction set mapping, resulting in task scheduling efficiency below 30%.

[0005] 3. IoT edge devices are often offline due to energy consumption limitations, resulting in a data version conflict rate exceeding 40% with the cloud.

[0006] 4. Video, text, and sensor data streams need to be integrated into unified semantics in real time (such as in autonomous driving scenarios). Existing ETL tools have a delay exceeding 500ms.

[0007] 5. Restoring cold data archived to tape libraries / Blu-ray storage takes several minutes, which cannot meet sudden retrieval needs (such as judicial evidence collection).

[0008] 6. In the blockchain + cloud computing model, there is a break in the trust chain between smart contract verification and computing task execution.

[0009] 7. Large model training needs to be completed in the cloud, and sub-networks cannot be dynamically unloaded based on the computing power of edge devices, resulting in a loss of model accuracy of more than 15%.

[0010] 8. The "fault-tolerant design" of the microservice architecture leads to cascading failures, and the existing circuit breaker mechanism has a misjudgment rate of over 30%.

[0011] 9. The analog computing characteristics of photonic chips are incompatible with the discretization of digital cloud storage, and the data conversion loss exceeds 50%.

[0012] Therefore, there is an urgent need in the art for a technical solution that can solve the problems existing in existing cloud computing data processing methods and systems.

[0013] The information disclosed in this background technology section is only intended to enhance understanding of the overall background of the invention and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the Invention

[0014] The purpose of the present invention is to provide a technical solution that can solve the problems existing in existing cloud computing data processing methods and systems.

[0015] To achieve the above object, the present invention provides the following solutions:

[0016] A data processing method based on cloud computing, comprising:

[0017] Step 1: Standardization of multi-source heterogeneous data;

[0018] Receive the original data stream through the API gateway and perform format verification;

[0019] Extract features from unstructured data;

[0020] Generate a unified spatiotemporal identifier;

[0021] Step 2: Dynamic encrypted storage;

[0022] Use the SMR algorithm to encrypt data shards, and the key is dynamically generated by the KMS hardware module;

[0023] Select a storage engine based on the data type;

[0024] Perform redundancy check before writing;

[0025] Step 3: Resource-aware task scheduling;

[0026] Real-time collection of resource indicators of each node;

[0027] Predict the optimal task allocation strategy through the DQN algorithm;

[0028] Package the computing tasks into Docker images and distribute them to the target nodes;

[0029] Step 4: Hybrid computing execution;

[0030] Start the Spark execution engine for batch tasks and divide the Stage dependency graph;

[0031] Streaming tasks are processed by the Flink engine, with Exactly-Once semantic guarantees.

[0032] FPGA accelerators perform hardware-level optimization of matrix operations;

[0033] Step 5: Security audit and self-healing;

[0034] A blockchain fingerprint that records all data access events;

[0035] Periodically perform storage integrity verification;

[0036] Automatic isolation and data redistribution are triggered when abnormal nodes are found.

[0037] Optionally, the steps of receiving the original data stream through the API gateway, performing format verification, extracting features from the unstructured data, and generating the unified spatiotemporal identifier are as follows:

[0038] The API gateway receives the original data stream:

[0039] Request access and protocol parsing: Receive HTTP / HTTPS and WebSocket protocol data streams through the API gateway, parse the request header, body (the request body sent by the client via HTTP / HTTPS or WebSocket protocol) and additional parameters; distribute the data stream to the corresponding backend service according to the routing rules;

[0040] Load balancing and flow control dynamically distribute requests to multiple backend instances to avoid single point overload; control data flow rate through current limiting strategies;

[0041] Perform format validation:

[0042] Structured data validation: checking whether request parameters contain necessary fields; verifying the value range, string regular expressions, and the legitimacy of JSON / XML structures;

[0043] Signature and security verification: Use the digest signature algorithm to verify the legitimacy of the request; check the access token permission scope and intercept unauthorized requests;

[0044] Exception handling, returning standard error codes and detailed error descriptions for illegal requests;

[0045] Unstructured data feature extraction:

[0046] Data routing and preprocessing: routing unstructured data to dedicated feature extraction services; performing data cleaning and format conversion;

[0047] Feature extraction algorithm call:

[0048] Text data is processed through NLP models (natural language processing models) to extract keywords, identify entities, or generate embedding vectors;

[0049] The image / video extracts visual features by calling the CV model (computer vision model);

[0050] Apply FFT or sliding window statistics to time series features;

[0051] Feature structured output, encapsulating the extracted results in standard JSON format, including feature vectors, confidence levels, and metadata;

[0052] Generate a unified spatiotemporal identifier:

[0053] Spatiotemporal information fusion: extract timestamp and geographic location from the request. If there is no explicit data, the gateway reception time and IP location are used.

[0054] Unique identification generation: Use the snowflake algorithm or UUIDv7 to generate a globally unique ID and embed timestamp information; combine spatiotemporal information with the unique ID to generate an identifier;

[0055] Identifier binding and storage: associate identifiers with raw data and feature extraction results, and write them to a distributed database; notify downstream systems of identifier availability through message queues.

[0056] Optionally, the SMR algorithm is used to encrypt data shards. The key is dynamically generated by the KMS hardware module, and the storage engine is selected according to the data type. The redundancy check is performed before writing as follows:

[0057] Dynamic key generation and shard encryption:

[0058] Step 1: Call KMS to generate a data key. Generate a data encryption key through the KMS hardware module interface and obtain its ciphertext and plaintext.

[0059] Step 2: Shard encryption and key binding, using the SMR sharding algorithm to divide the data into multiple logical shards; each shard is symmetrically encrypted using the DEK plaintext, and the shard number and metadata identifier are added during encryption;

[0060] Step 3: Securely store the key, bind the DEK ciphertext to the encrypted shard data and store it to ensure that the DEK plaintext can be restored through KMS during decryption;

[0061] Select the storage engine by data type:

[0062] Step 4: Dynamically adapt the storage engine:

[0063] Structured data: Choose a storage engine that supports column-level encryption and encrypt sensitive fields separately before storing them.

[0064] Unstructured data: Use the object storage engine, enable server-side encryption, and directly use the DEK managed by KMS for encryption;

[0065] Time series data: Use a time series database to compress sharded data before writing and then perform block-level encryption to reduce storage overhead;

[0066] Redundancy check before writing:

[0067] Step 5: Data integrity check:

[0068] Calculate the hash value for each encrypted shard and append the hash value to the shard metadata;

[0069] Before writing to the storage engine, verify the following conditions:

[0070] The shard data length complies with the storage engine's block size limit;

[0071] The hash value of each shard matches the hash value of the original data before encryption to prevent data corruption during transmission or encryption.

[0072] Step 6: Redundant writes are synchronized with the replicas:

[0073] For sharded data that has passed verification, multiple copies are written as required by the storage engine.

[0074] The shard encryption algorithm, DEK ciphertext identifier, and hash value are recorded in the replica metadata to ensure traceability during failure recovery.

[0075] Optionally, the real-time collection of resource indicators of each node; prediction of the optimal task allocation strategy using the DQN algorithm; and packaging the computing tasks into Docker images for distribution to target nodes are specifically:

[0076] Real-time resource indicator collection and monitoring:

[0077] Step 1: Deploy the data collection agent:

[0078] Deploy Prometheus Exporter on each GPU node in the Kubernetes cluster. Use the DaemonSet controller to ensure that all GPU nodes automatically run the Exporter component, which collects and exposes the node's GPU hardware performance metrics (such as memory utilization and computing load) in real time. This allows Prometheus to uniformly capture and monitor cluster resource status. After the Exporter is deployed as a DaemonSet, each GPU node automatically runs a Pod instance, eliminating the need to manually specify the node, ensuring that monitoring data for all GPU devices is collected. Use a custom metric collector to capture GPU core utilization, memory usage, power consumption and temperature, and CPU / memory usage.

[0079] Step 2: Build an indicator storage and alarm system:

[0080] Use Prometheus TSDB to store historical data, and configure the Grafana visualization panel to display global resource utilization heat maps and single-card operation status time series diagrams;

[0081] Set threshold alarm rules;

[0082] Step 3: Anomaly detection and feedback adjustment:

[0083] Identify GPU hardware failures through the XID error detection module;

[0084] Dynamically adjust the acquisition frequency;

[0085] DQN algorithm driven task allocation strategy:

[0086] Step 4: Define the reinforcement learning model input:

[0087] State space: the node's available GPU computing power, remaining memory capacity, and network bandwidth and latency;

[0088] Action space: task assignment target node number and GPU instance selection;

[0089] Reward function: resource utilization improvement weight, task completion delay penalty, and service quality violation cost;

[0090] Step 5: Model training and strategy generation:

[0091] Use offline historical data to pre-train the DQN network;

[0092] In the online stage, the Q-value table is updated in real time through the Actor-Critic architecture;

[0093] Step 6: Strategy execution and feedback loop:

[0094] Encapsulate the predicted optimal allocation strategy as a Kubernetes scheduler plug-in;

[0095] Synchronize scheduling decisions to cluster state storage through the API Server;

[0096] Containerized task distribution and deployment:

[0097] Step 7: Compute task image building:

[0098] Write Dockerfile to integrate the CUDA runtime environment;

[0099] Push the image to the private registry and add a version tag;

[0100] Step 8: Kubernetes Task Orchestration:

[0101] Create a Job or Deployment resource description file to declare GPU resource requirements;

[0102] Trigger deployments via kubectl or CI / CD pipelines;

[0103] Step 9: Dynamic distribution and elastic scaling:

[0104] Select the target node based on the DQN strategy: pull the image through kubelet and start the Pod;

[0105] Monitor the task execution status and trigger rescheduling when an exception occurs.

[0106] Optionally, starting a Spark execution engine for the batch tasks and dividing the Stage dependency graph;

[0107] Streaming tasks are processed by the Flink engine, with Exactly-Once semantic guarantees.

[0108] The FPGA accelerator performs hardware-level optimization on matrix operations as follows:

[0109] Step 1: Task submission and DAG (directed acyclic graph) construction:

[0110] Trigger the job submission and start the job by calling the Actions operator;

[0111] A directed acyclic graph (DAG) is constructed based on the dependency chain of RDD (Resilient Distributed Dataset). Each job corresponds to an independent DAG structure.

[0112] Step 2: Determine the stage division rules, including wide dependency segmentation and recursive backtracking;

[0113] Step 3: Task generation and scheduling, including stage submission and task allocation;

[0114] Step 4: Configure checkpoints, including:

[0115] Turn on checkpoints, enable checkpoints and specify the interval;

[0116] Persist state data to reliable storage;

[0117] Step 5: Barrier alignment and two-phase commit:

[0118] The source end periodically inserts barrier events (special marker events periodically inserted by the source end to demarcate checkpoint boundaries in the data stream and trigger each operator to save state snapshots), triggering a global state snapshot;

[0119] After receiving all input barriers, the operator asynchronously writes the memory state to external storage and records the offset;

[0120] Pre-submit data to the external system and commit the final transaction after the checkpoint is completed;

[0121] Step 6: Prepare the hardware environment:

[0122] Install the FPGA accelerator card and configure the PCIe interface to communicate with the host;

[0123] Install the OpenCL or Vitis development environment to support hardware description language compilation;

[0124] Step 7: Hardwareization of computing logic:

[0125] Convert computationally intensive operations such as matrix multiplication and convolution into parallel hardware circuit designs, optimizing pipelines and parallelism;

[0126] Use high-level synthesis tools to convert C / C++ code into FPGA executable IP cores;

[0127] Step 8: Framework Integration and Acceleration:

[0128] API encapsulation: Expose FPGA computing interfaces through OpenCL or custom drivers for upper-layer applications;

[0129] Data pipeline optimization: Design DMA transmission channels to achieve zero-copy data transmission between host memory and FPGA video memory.

[0130] Optionally, the security audit and self-healing; blockchain fingerprints recording all data access events;

[0131] Periodically perform storage integrity verification;

[0132] When an abnormal node is found, automatic isolation and data redistribution are triggered as follows:

[0133] Security audit and self-healing process:

[0134] Step 1: Full-link collection of audit logs:

[0135] Enable audit service: Enable the audit log function on the control plane and configure the log storage path and retention policy;

[0136] Define audit policy: Record scope: Cover all data access events;

[0137] Sensitive operation marking: Privileged operations are marked as high-risk events;

[0138] Log storage and encryption: Audit logs are written to an independent encrypted storage pool, using the AES-256 encryption algorithm to protect static data.

[0139] Step 2: Real-time analysis and self-healing triggering:

[0140] Preset rule base: matches known attack patterns;

[0141] Dynamic baseline learning: Analyze historical logs through machine learning to dynamically generate behavior baselines;

[0142] Automatic isolation: When an abnormal IP or user is detected, the firewall API is called to block the connection and freeze the account;

[0143] Data repair: If file tampering is detected, the backup system is triggered to perform incremental recovery;

[0144] Blockchain fingerprints record data access events:

[0145] Step 3: Data operation fingerprint is uploaded to the chain:

[0146] Hash generation: Generates a SHA-3 hash value for each data access event, which contains the operator identity, timestamp, and operation type triple;

[0147] Blockchain writing: The hash value is written into the private chain, and the PBFT consensus mechanism is used to ensure that it cannot be tampered with;

[0148] A block is generated every 8 minutes, and the block header contains the preceding hash and the Merkle tree root;

[0149] Step 4: Cross-node verification and tracing:

[0150] Distributed verification: All nodes synchronize blockchain data and verify the legitimacy of new blocks through smart contracts;

[0151] Audit traceability: The input operation hash can trace the complete operation chain and verify whether the data has been maliciously modified;

[0152] Periodic verification of storage integrity:

[0153] Step 5: Determine the integrity verification mechanism:

[0154] Checksum generation: Generate CRC64 checksums for all stored data daily and store them in an independent metadata repository.

[0155] Automated verification: Perform weekly verification and comparison. If any inconsistency is detected, an alarm is triggered and the abnormal location is recorded.

[0156] Step 6: Repair and Warning:

[0157] Data repair: For damaged blocks, healthy data is pulled from the mirror copy to overwrite;

[0158] Alarm classification: Level 1 alarms are notified to operations and maintenance via SMS / email, and level 2 alarms are recorded in the work order system;

[0159] Automatic isolation of abnormal nodes and data redistribution:

[0160] Step 7: Health status monitoring:

[0161] Metrics collection: real-time monitoring of node CPU / memory / disk IO and setting thresholds;

[0162] Heartbeat detection: The Consul service registration center sends heartbeat packets every 15 seconds. If the heartbeat packets time out three times, the node is considered faulty.

[0163] Step 8: Isolation and data migration, including isolation execution and data redistribution: According to the storage policy, migrate the data blocks to healthy nodes to ensure that the number of replicas meets the requirements; after the migration is complete, update the metadata and release the abnormal node resources.

[0164] A data processing system based on cloud computing, comprising:

[0165] Data ingestion layer: supports streaming / batch data access and has a built-in protocol conversion module;

[0166] Heterogeneous storage layer: A hybrid storage engine is built based on object storage (S3) + columnar database (ClickHouse) + graph database (Neo4j);

[0167] Elastic computing layer: Kubernetes-based containerized computing cluster with integrated FPGA acceleration nodes;

[0168] Intelligent scheduling layer: a multi-dimensional resource scheduler integrating reinforcement learning;

[0169] Security governance layer: end-to-end encryption channel and dynamic access control engine based on national secret algorithm.

[0170] A computer device includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor performs the steps of the above-mentioned cloud computing-based data processing method.

[0171] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor performs the steps of the above-mentioned cloud computing-based data processing method.

[0172] An information data processing terminal includes the above-mentioned cloud computing-based data processing system.

[0173] Compared with the prior art, the present invention has the following beneficial effects:

[0174] The cloud computing-based data processing method and system provided by this invention achieves high-concurrency processing and automatic load balancing through flexible resource allocation and dynamic computing power scheduling within a distributed computing framework, combined with a storage and computing separation architecture. The system supports cross-node disaster recovery and multi-layer encryption to ensure data security. Its pay-as-you-go model reduces hardware investment costs, improves resource utilization, and significantly enhances large-scale data processing efficiency and system scalability. It can achieve millimeter-level resource response, significantly improve task scheduling efficiency, meet sudden retrieval needs, significantly enhance model accuracy, and reduce data conversion losses, resolving the problems of existing cloud computing-based data processing methods and systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0175] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0176] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0177] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0178] The purpose of the present invention is to provide a method and system that can solve the problems existing in existing cloud computing data processing methods and systems.

[0179] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0180] Example 1:

[0181] This embodiment applies the above-mentioned cloud computing-based data processing method and system to predictive maintenance of industrial equipment. Figure 1 As shown, a predictive maintenance system for industrial equipment is provided:

[0182] 1. Hardware configuration:

[0183] Data collection end: deploy EdgeX Foundry edge gateway (ARM64 architecture);

[0184] Cloud infrastructure: 3 management nodes (64 cores / 256GB memory) + 20 compute nodes (equipped with NVIDIA T4 GPUs);

[0185] Storage cluster: Ceph object storage (5 nodes, 12TB NVMe SSD per node).

[0186] 2. Data processing flow

[0187] (1) Vibration sensor data access:

[0188] The edge gateway performs wavelet noise reduction on the 10kHz sampling signal;

[0189] Upload to the Kafka cluster (topic: / iot / vibration) through a TLS1.3 encrypted channel.

[0190] (2) Multimodal feature fusion:

[0191] Extract frequency domain features of vibration signals (FFT transform);

[0192] Related equipment maintenance logs (NLP entity recognition technology);

[0193] Generate a fused feature vector (dimension=512).

[0194] (3) Anomaly detection model training:

[0195] Use LSTM-Autoencoder to build a reconstruction error model;

[0196] Distributed training on a Spark cluster (data parallelism = 32);

[0197] Model update cycle: incremental training every 6 hours.

[0198] (4) Real-time warning trigger:

[0199] Flink jobs continuously calculate the device health index (Health Index);

[0200] When the index exceeds the threshold, a work order is automatically generated (RPA linked to ServiceNow);

[0201] The warning results are written into the HyperLedger Fabric blockchain for evidence storage.

[0202] 3. Performance indicators

[0203] Data processing throughput: 1.2TB / hour (compression ratio = 5:1);

[0204] Fault detection accuracy: 98.7% (F1-score);

[0205] End-to-end latency: <15 minutes for batch processing tasks and <200ms for stream processing.

[0206] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0207] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A data processing method based on cloud computing, characterized in that: include: Step 1: Standardization of multi-source heterogeneous data; Receive the original data stream through the API gateway and perform format verification; Extract features from unstructured data; Generate a unified spatiotemporal identifier; Step 2: Dynamic encrypted storage; Use the SMR algorithm to encrypt data shards, and the key is dynamically generated by the KMS hardware module; Select a storage engine based on the data type; Perform redundancy check before writing; Step 3: Resource-aware task scheduling; Real-time collection of resource indicators of each node; Predict the optimal task allocation strategy through the DQN algorithm; Package the computing tasks into Docker images and distribute them to the target nodes; Step 4: Hybrid computing execution; Start the Spark execution engine for batch tasks and divide the Stage dependency graph; Streaming tasks are processed by the Flink engine, with Exactly-Once semantic guarantees. FPGA accelerators perform hardware-level optimization of matrix operations; Step 5: Security audit and self-healing; A blockchain fingerprint that records all data access events; Periodically perform storage integrity verification; Automatic isolation and data redistribution are triggered when abnormal nodes are found.

2. The cloud computing-based data processing method according to claim 1, characterized in that: The steps of receiving the original data stream through the API gateway, performing format verification, extracting features from unstructured data, and generating a unified spatiotemporal identifier are as follows: The API gateway receives the original data stream: Request access and protocol parsing: receiving HTTP / HTTPS and WebSocket protocol data streams through the API gateway, parsing the request header, body, and additional parameters; distributing the data stream to the corresponding backend service according to routing rules; Load balancing and flow control dynamically distribute requests to multiple backend instances to avoid single point overload; control data flow rate through current limiting strategies; Perform format validation: Structured data validation: checking whether request parameters contain necessary fields; verifying the value range, string regular expressions, and the legitimacy of JSON / XML structures; Signature and security verification: Use the digest signature algorithm to verify the legitimacy of the request; check the access token permission scope and intercept unauthorized requests; Exception handling, returning standard error codes and detailed error descriptions for illegal requests; Unstructured data feature extraction: Data routing and preprocessing: routing unstructured data to dedicated feature extraction services; performing data cleaning and format conversion; Feature extraction algorithm call: Text data is used to extract keywords, identify entities, or generate embedding vectors through NLP models; The image / video extracts visual features by calling the CV model; Apply FFT or sliding window statistics to time series features; Feature structured output, encapsulating the extracted results in standard JSON format, including feature vectors, confidence levels, and metadata; Generate a unified spatiotemporal identifier: Spatiotemporal information fusion: extract timestamp and geographic location from the request. If there is no explicit data, the gateway reception time and IP location are used. Unique identifier generation: Use the snowflake algorithm or UUIDv7 to generate a globally unique ID and embed timestamp information; Combine spatiotemporal information with a unique ID to generate an identifier; Identifier binding and storage: associate identifiers with raw data and feature extraction results, and write them to a distributed database; notify downstream systems of identifier availability through message queues.

3. The cloud computing-based data processing method according to claim 1, characterized in that: The SMR algorithm is used to encrypt data shards. The key is dynamically generated by the KMS hardware module, and the storage engine is selected according to the data type. The redundancy check is performed before writing as follows: Dynamic key generation and shard encryption: Step 1: Call KMS to generate a data key. Generate a data encryption key through the KMS hardware module interface and obtain its ciphertext and plaintext. Step 2: Shard encryption and key binding, using the SMR sharding algorithm to divide the data into multiple logical shards; Each shard is symmetrically encrypted using the DEK plaintext, and the shard number and metadata identifier are added during encryption; Step 3: Securely store the key, bind the DEK ciphertext to the encrypted shard data and store it to ensure that the DEK plaintext can be restored through KMS during decryption; Select the storage engine by data type: Step 4: Dynamically adapt the storage engine: Structured data: Choose a storage engine that supports column-level encryption, encrypt sensitive fields separately, and then store unstructured data: Use an object storage engine, enable server-side encryption, and directly use the DEK managed by KMS for encryption; Time series data: Use a time series database to compress sharded data before writing and then perform block-level encryption to reduce storage overhead; Redundancy check before writing: Step 5: Data integrity check: Calculate the hash value for each encrypted shard and append the hash value to the shard metadata; Before writing to the storage engine, verify the following conditions: The shard data length complies with the storage engine's block size limit; The hash value of each shard matches the hash value of the original data before encryption to prevent data corruption during transmission or encryption. Step 6: Redundant writes are synchronized with the replicas: For sharded data that has passed verification, multiple copies are written as required by the storage engine. The shard encryption algorithm, DEK ciphertext identifier, and hash value are recorded in the replica metadata to ensure traceability during failure recovery.

4. The cloud computing-based data processing method according to claim 1, characterized in that: The real-time collection of resource indicators of each node; prediction of the optimal task allocation strategy through the DQN algorithm; and packaging the computing tasks into Docker images for distribution to the target nodes are specifically: Real-time resource indicator collection and monitoring: Step 1: Deploy the data collection agent: Deploy Prometheus Exporter on each GPU node in the Kubernetes cluster and use a custom metric collector to capture GPU core utilization, memory occupancy, power consumption and temperature, and CPU / memory usage; Step 2: Build an indicator storage and alarm system: Use Prometheus TSDB to store historical data, and configure the Grafana visualization panel to display global resource utilization heat maps and single-card operation status time series diagrams; Set threshold alarm rules; Step 3: Anomaly detection and feedback adjustment: Identify GPU hardware failures through the XID error detection module; Dynamically adjust the acquisition frequency; DQN algorithm driven task allocation strategy: Step 4: Define the reinforcement learning model input: State space: the node's available GPU computing power, remaining memory capacity, and network bandwidth and latency; Action space: task assignment target node number and GPU instance selection; Reward function: resource utilization improvement weight, task completion delay penalty, and service quality violation cost; Step 5: Model training and strategy generation: Use offline historical data to pre-train the DQN network; In the online stage, the Q-value table is updated in real time through the Actor-Critic architecture; Step 6: Strategy execution and feedback loop: Encapsulate the predicted optimal allocation strategy as a Kubernetes scheduler plug-in; Synchronize scheduling decisions to cluster state storage through the API Server; Containerized task distribution and deployment: Step 7: Compute task image building: Write Dockerfile to integrate the CUDA runtime environment; Push the image to the private registry and add a version tag; Step 8: Kubernetes Task Orchestration: Create a Job or Deployment resource description file to declare GPU resource requirements; Trigger deployments via kubectl or CI / CD pipelines; Step 9: Dynamic distribution and elastic scaling: Select the target node based on the DQN strategy: pull the image through kubelet and start the Pod; Monitor the task execution status and trigger rescheduling when an exception occurs.

5. The data processing method based on cloud computing according to claim 1, characterized in that: The Spark execution engine is started for the batch tasks and the Stage dependency graph is divided; Streaming tasks are processed by the Flink engine, with Exactly-Once semantic guarantees. The FPGA accelerator performs hardware-level optimization on matrix operations as follows: Step 1: Task submission and DAG construction: Trigger the job submission and start the job by calling the Actions operator; Build a directed acyclic graph based on the RDD dependency chain, and each job corresponds to an independent DAG structure; Step 2: Determine the stage division rules, including wide dependency segmentation and recursive backtracking; Step 3: Task generation and scheduling, including stage submission and task allocation; Step 4: Configure checkpoints, including: Turn on checkpoints, enable checkpoints and specify the interval; Persist state data to reliable storage; Step 5: Barrier alignment and two-phase commit: The Source end periodically inserts barrier events to trigger a global state snapshot; After receiving all input barriers, the operator asynchronously writes the memory state to external storage and records the offset; Pre-submit data to the external system and commit the final transaction after the checkpoint is completed; Step 6: Prepare the hardware environment: Install the FPGA accelerator card and configure the PCIe interface to communicate with the host; Install the OpenCL or Vitis development environment to support hardware description language compilation; Step 7: Hardwareization of computing logic: Convert computationally intensive operations such as matrix multiplication and convolution into parallel hardware circuit designs, optimizing pipelines and parallelism; Use high-level synthesis tools to convert C / C++ code into FPGA executable IP cores; Step 8: Framework Integration and Acceleration: API encapsulation: Expose FPGA computing interfaces through OpenCL or custom drivers for upper-layer applications; Data pipeline optimization: Design DMA transmission channels to achieve zero-copy data transmission between host memory and FPGA video memory.

6. The cloud computing-based data processing method according to claim 1, characterized in that: The aforementioned security audit and self-healing; blockchain fingerprints that record all data access events; Periodically perform storage integrity verification; When an abnormal node is found, automatic isolation and data redistribution are triggered as follows: Security audit and self-healing process: Step 1: Full-link collection of audit logs: Enable audit service: Enable the audit log function on the control plane and configure the log storage path and retention policy; Define audit policy: Record scope: Cover all data access events; Sensitive operation marking: Privileged operations are marked as high-risk events; Log storage and encryption: Audit logs are written to an independent encrypted storage pool, using the AES-256 encryption algorithm to protect static data. Step 2: Real-time analysis and self-healing triggering: Preset rule base: matches known attack patterns; Dynamic baseline learning: Analyze historical logs through machine learning to dynamically generate behavior baselines; Automatic isolation: When an abnormal IP or user is detected, the firewall API is called to block the connection and freeze the account; Data repair: If file tampering is detected, the backup system is triggered to perform incremental recovery; Blockchain fingerprints record data access events: Step 3: Data operation fingerprint is uploaded to the chain: Hash generation: Generates a SHA-3 hash value for each data access event, which contains the operator identity, timestamp, and operation type triple; Blockchain writing: The hash value is written into the private chain, and the PBFT consensus mechanism is used to ensure that it cannot be tampered with; A block is generated every 8 minutes, and the block header contains the preceding hash and the Merkle tree root; Step 4: Cross-node verification and tracing: Distributed verification: All nodes synchronize blockchain data and verify the legitimacy of new blocks through smart contracts; Audit traceability: The input operation hash can trace the complete operation chain and verify whether the data has been maliciously modified; Periodic verification of storage integrity: Step 5: Determine the integrity verification mechanism: Checksum generation: Generate CRC64 checksums for all stored data daily and store them in an independent metadata repository. Automated verification: Perform weekly verification and comparison. If any inconsistency is detected, an alarm is triggered and the abnormal location is recorded. Step 6: Repair and Warning: Data repair: For damaged blocks, healthy data is pulled from the mirror copy to overwrite; Alarm classification: Level 1 alarms are notified to operations and maintenance via SMS / email, and level 2 alarms are recorded in the work order system; Automatic isolation of abnormal nodes and data redistribution: Step 7: Health status monitoring: Metrics collection: real-time monitoring of node CPU / memory / disk IO and setting thresholds; Heartbeat detection: The Consul service registration center sends heartbeat packets every 15 seconds. If the heartbeat packets time out three times, the node is considered faulty. Step 8: Isolation and data migration, including isolation execution and data redistribution: Migrate data blocks to healthy nodes according to the storage policy to ensure that the number of replicas meets the requirements; After the migration is complete, the metadata is updated and abnormal node resources are released.

7. A data processing system based on cloud computing, characterized in that: include: Data ingestion layer: supports streaming / batch data access and has a built-in protocol conversion module; Heterogeneous storage layer: A hybrid storage engine is built based on object storage (S3) + columnar database (ClickHouse) + graph database (Neo4j); Elastic computing layer: Kubernetes-based containerized computing cluster with integrated FPGA acceleration nodes; Intelligent scheduling layer: a multi-dimensional resource scheduler integrating reinforcement learning; Security governance layer: end-to-end encryption channel and dynamic access control engine based on national secret algorithm.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the cloud computing-based data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the cloud computing-based data processing method according to any one of claims 1 to 6.

10. An information data processing terminal, comprising the cloud computing-based data processing system according to claim 7.

Images