Intelligent terminal working data processing method and system based on cloud computing

Abstract

The invention provides an intelligent terminal working data processing method and system based on cloud computing, and the method comprises the steps: extracting a generation interval variance, data block dispersion and calculation correlation degree of working data based on a time sequence sliding window; through an LSTM prediction model, according to the generation interval variance, the data block dispersion and the calculation association degree, outputting a data generation rate fluctuation interval correspondingly generated by each type of working data in a future preset time; according to the data generation rate fluctuation interval of each type of working data and a preset processing aging requirement, a pooling mapping algorithm is adopted to allocate a corresponding data processing task; distributing corresponding working data to computing nodes in a basic processing pool, an elastic expansion pool and an emergency buffer pool according to the data processing task, and executing parallel processing to generate a plurality of data processing subsets; and constructing a result association graph according to the association degree among the plurality of data processing subsets, and integrating the result association graph into a corresponding target processing data set. The treatment efficiency can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a method and system for processing working data of a smart terminal based on cloud computing. Background Technology

[0002] With the development of IoT and AI technologies, smart terminals such as sensors and smartwatches have been widely used in various fields. Their core function is to collect and process work data. Currently, smart terminal data processing generally adopts a "local terminal processing + local server assistance" model. This model was used in the early stages of terminal popularization because the data volume was small and the needs were simple.

[0003] With the surge in the number of terminals and the upgrading of processing demands, this model has exposed obvious bottlenecks: First, smart terminals are limited by size and power consumption, resulting in limited hardware performance. They can only complete basic operations such as data collection and simple statistics. When faced with complex tasks such as AI model training and multi-dimensional data mining, they often experience lag, delays, or even failure to run due to insufficient computing power. Second, local server resources are fixed. When encountering peak concurrent data uploads from terminals, they are prone to resource overload leading to processing interruptions, which cannot meet real-time requirements.

[0004] Furthermore, existing technologies have attempted to alleviate the problem by enhancing terminal hardware or upgrading servers, but these efforts have proven ineffective: enhancing terminal hardware would increase size, power consumption, and cost; upgrading servers would require significant upfront investment, and fixed configurations would not be able to adapt to fluctuations in data volume, leaving resource gaps or idle resources. Consequently, existing technologies are unable to fully process the work data generated by smart terminals, thus reducing data processing efficiency. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide a cloud computing-based method and system for processing smart terminal work data, in order to solve the problem that existing technologies cannot dynamically adapt to the fluctuations in the amount of smart terminal work data, and there will still be resource gaps or idle resources, resulting in the inability to fully process the work data generated by smart terminals.

[0006] The first aspect of the present invention proposes: A cloud computing-based method for processing working data of intelligent terminals specifically includes the following steps: Collect several types of work data generated by multiple smart terminals, extract the generation interval variance, data block dispersion and computational correlation of each type of work data based on a time-series sliding window, and output the data generation rate fluctuation range of each type of work data within a future preset time period based on the generation interval variance, data block dispersion and computational correlation through an LSTM prediction model. The computing nodes of the cloud computing cluster are divided into basic processing pool, elastic expansion pool and emergency buffer pool according to hardware configuration. Based on the fluctuation range of data generation rate of each type of work data and the preset processing time requirements, the corresponding data processing tasks are allocated using a pooling mapping algorithm. According to the data processing task, the corresponding working data is allocated to the computing nodes in the basic processing pool, the elastic expansion pool and the emergency buffer pool and parallel processing is performed to generate several data processing subsets accordingly. The integrity of several data processing subsets is verified by hash verification, and a result association graph is constructed based on the correlation between the several data processing subsets, and integrated into the corresponding target processing data set.

[0007] The beneficial effects of this invention are as follows: by parsing and processing the collected working data of smart terminals, the corresponding data generation rate fluctuation range can be obtained. Based on this, in order to facilitate subsequent data processing, the computing nodes within the cloud computing cluster are divided into a basic processing pool, an elastic expansion pool, and an emergency buffer pool. Based on this, corresponding working data is allocated to the computing nodes within each resource pool and parallel processing is performed to generate several preliminary data processing subsets. Based on this, a result correlation graph is constructed according to the correlation between the current several data processing subsets, and integrated into the final target processing data set. This allows for dynamic adaptation to the data volume fluctuations of the working data of each smart terminal, and comprehensive processing of the working data of the smart terminals.

[0008] Furthermore, the step of allocating corresponding data processing tasks using a pooling mapping algorithm based on the data generation rate fluctuation range and preset processing time requirements for each type of working data includes: Extract the data dimensions, number of field types, and encryption level of each type of work data to calculate the corresponding data complexity coefficient, and statistically analyze the task completion rate and processing accuracy of the basic processing pool, the elastic expansion pool, and the emergency buffer pool within the preset processing cycle to generate the corresponding priority matrix. Based on the fluctuation range of the generation rate fluctuation range of each type of working data, the corresponding target processing pool is matched in the priority matrix. Data processing tasks are assigned to the target processing pool based on its target attributes and the preset processing time requirements.

[0009] Furthermore, the step of allocating corresponding data processing tasks to the target processing pool based on its target attributes and the preset processing time requirements includes: The number of computing nodes, single-node computing power threshold, storage bandwidth limit and concurrent processing channel number of the target processing pool are extracted and quantized into corresponding resource supply vectors according to preset weights. Based on the data complexity coefficient of each type of work data, the preset processing time is converted into the corresponding maximum processing delay threshold, and the corresponding data processing task is divided into several independent task units according to the correlation of data fields. Each independent task unit is assigned a unique resource matching identifier according to the dimension of the resource supply vector. The processing parameters of each computing node in the target processing pool are collected at preset intervals, and the allocation weight of each independent task unit is dynamically adjusted according to the processing parameters to complete the data processing task according to the allocation weight.

[0010] Furthermore, the step of allocating corresponding working data to computing nodes in the basic processing pool, the elastic expansion pool, and the emergency buffer pool according to the data processing task and performing parallel processing to generate several data processing subsets includes: The processor utilization rate, memory utilization rate and network transmission latency of each computing node in the basic processing pool, the elastic expansion pool and the emergency buffer pool are collected. Combined with the data block size and data type of each type of work data, each type of work data is split into several data fragments. Each data fragment carries a corresponding category identifier and time sequence generation stamp. The corresponding data processing margin is calculated based on the processor utilization, memory usage and network transmission latency of each computing node, and a suitable data fragment is allocated to each computing node according to the data processing margin. After each computing node obtains the corresponding data fragment, data deduplication, format standardization, and key information extraction operations are performed sequentially according to the order of the time-series generated stamps. The fragment processing progress is synchronized through the preset TCP synchronous communication port between each node to generate several data processing subsets.

[0011] Furthermore, the step of synchronizing the fragment processing progress through preset TCP synchronous communication ports between various nodes to generate several data processing subsets includes: After each computing node completes its operation, a corresponding progress synchronization data packet is generated and uploaded to the coordination node of the cloud computing cluster through the preset TCP synchronization communication port. The coordination node constructs a corresponding processing progress matrix based on the progress synchronization data packet, and uses the processing progress matrix to determine in real time whether all the preset operations of each data segment have been completed. If the processing progress matrix indicates that all preset operations for data sharding have been completed, a corresponding subset encapsulation instruction is generated, and a corresponding index identifier is added to generate the corresponding data processing subset.

[0012] Furthermore, the step of constructing a result association graph based on the correlation between several data processing subsets and integrating it into a corresponding target processing data set includes: Extract the terminal scene label, processing task identifier, and key metadata of each data processing subset that passes the hash verification, and use the improved Jaccard similarity coefficient to calculate the cross-correlation degree between any two data processing subsets based on the terminal scene label, the processing task identifier, and the key metadata to generate the corresponding correlation matrix; Each of the data processing subsets is mapped to a graph node of the result association graph, and an edge between any two graph nodes is created according to the association degree matrix. Based on the edge type, the upstream and downstream dependency chains between each of the data processing subsets are sorted out, so as to integrate the target processing data set according to the upstream and downstream dependency chains.

[0013] Furthermore, the step of integrating the target processing data set according to the upstream and downstream dependency chains includes: The graph nodes are topologically sorted according to the order of the upstream and downstream dependency chains to generate several independent dependency data chains. Merge all field information in each dependent data chain and fill in missing field values ​​using metadata association rules; A semantic alignment algorithm based on knowledge graphs is used to remove semantically duplicate fields, and the data are aggregated sequentially according to the priority of each dependent data chain to generate the target processing data set.

[0014] The second aspect of the present invention proposes: A cloud computing-based intelligent terminal working data processing system, wherein the system includes: The acquisition module is used to collect several types of work data generated by multiple smart terminals. Based on the time-series sliding window, it extracts the generation interval variance, data block dispersion and computational correlation of each type of work data. Then, through the LSTM prediction model, it outputs the data generation rate fluctuation range of each type of work data within a future preset time according to the generation interval variance, data block dispersion and computational correlation. The partitioning module is used to divide the computing nodes of the cloud computing cluster into basic processing pools, elastic expansion pools and emergency buffer pools according to hardware configuration, and to allocate corresponding data processing tasks based on the data generation rate fluctuation range and preset processing time requirements of each type of work data using a pooling mapping algorithm. The allocation module is used to allocate corresponding working data to the computing nodes in the basic processing pool, the elastic expansion pool and the emergency buffer pool according to the data processing task and perform parallel processing to generate several data processing subsets accordingly. The integration module is used to complete the integrity verification of several data processing subsets through hash verification, construct the result association map according to the correlation between the several data processing subsets, and integrate them into the corresponding target processing data set.

[0015] Furthermore, the partitioning module is specifically used for: Extract the data dimensions, number of field types, and encryption level of each type of work data to calculate the corresponding data complexity coefficient, and statistically analyze the task completion rate and processing accuracy of the basic processing pool, the elastic expansion pool, and the emergency buffer pool within the preset processing cycle to generate the corresponding priority matrix. Based on the fluctuation range of the generation rate fluctuation range of each type of working data, the corresponding target processing pool is matched in the priority matrix. Data processing tasks are assigned to the target processing pool based on its target attributes and the preset processing time requirements.

[0016] Furthermore, the partitioning module is specifically used for: The number of computing nodes, single-node computing power threshold, storage bandwidth limit and concurrent processing channel number of the target processing pool are extracted and quantized into corresponding resource supply vectors according to preset weights. Based on the data complexity coefficient of each type of work data, the preset processing time is converted into the corresponding maximum processing delay threshold, and the corresponding data processing task is divided into several independent task units according to the correlation of data fields. Each independent task unit is assigned a unique resource matching identifier according to the dimension of the resource supply vector. The processing parameters of each computing node in the target processing pool are collected at preset intervals, and the allocation weight of each independent task unit is dynamically adjusted according to the processing parameters to complete the data processing task according to the allocation weight.

[0017] Furthermore, the allocation module is specifically used for: The processor utilization rate, memory utilization rate and network transmission latency of each computing node in the basic processing pool, the elastic expansion pool and the emergency buffer pool are collected. Combined with the data block size and data type of each type of work data, each type of work data is split into several data fragments. Each data fragment carries a corresponding category identifier and time sequence generation stamp. The corresponding data processing margin is calculated based on the processor utilization, memory usage and network transmission latency of each computing node, and a suitable data fragment is allocated to each computing node according to the data processing margin. After each computing node obtains the corresponding data fragment, data deduplication, format standardization, and key information extraction operations are performed sequentially according to the order of the time-series generated stamps. The fragment processing progress is synchronized through the preset TCP synchronous communication port between each node to generate several data processing subsets.

[0018] Furthermore, the allocation module is specifically used for: After each computing node completes its operation, a corresponding progress synchronization data packet is generated and uploaded to the coordination node of the cloud computing cluster through the preset TCP synchronization communication port. The coordination node constructs a corresponding processing progress matrix based on the progress synchronization data packet, and uses the processing progress matrix to determine in real time whether all the preset operations of each data segment have been completed. If the processing progress matrix indicates that all preset operations for data sharding have been completed, a corresponding subset encapsulation instruction is generated, and a corresponding index identifier is added to generate the corresponding data processing subset.

[0019] Furthermore, the integration module is specifically used for: Extract the terminal scene label, processing task identifier, and key metadata of each data processing subset that passes the hash verification, and use the improved Jaccard similarity coefficient to calculate the cross-correlation degree between any two data processing subsets based on the terminal scene label, the processing task identifier, and the key metadata to generate the corresponding correlation matrix; Each of the data processing subsets is mapped to a graph node of the result association graph, and an edge between any two graph nodes is created according to the association degree matrix. Based on the edge type, the upstream and downstream dependency chains between each of the data processing subsets are sorted out, so as to integrate the target processing data set according to the upstream and downstream dependency chains.

[0020] Furthermore, the integration module is specifically used for: The graph nodes are topologically sorted according to the order of the upstream and downstream dependency chains to generate several independent dependency data chains. Merge all field information in each dependent data chain and fill in missing field values ​​using metadata association rules; A semantic alignment algorithm based on knowledge graphs is used to remove semantically duplicate fields, and the data are aggregated sequentially according to the priority of each dependent data chain to generate the target processing data set.

[0021] The third aspect of the present invention proposes: A computer includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the cloud computing-based smart terminal working data processing method described above.

[0022] The fourth aspect of the present invention proposes: A readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the cloud computing-based smart terminal working data processing method described above.

[0023] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] Figure 1 A flowchart illustrating the cloud computing-based smart terminal working data processing method provided in the first embodiment of the present invention; Figure 2 The structural block diagram of the cloud computing-based smart terminal working data processing system provided in the third embodiment of the present invention is shown.

[0025] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0026] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0027] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0029] Please see Figure 1 The figure shows a cloud computing-based smart terminal working data processing method provided in the first embodiment of the present invention. The cloud computing-based smart terminal working data processing method provided in this embodiment can dynamically and comprehensively process the working data generated by each smart terminal, thereby improving the data processing efficiency.

[0030] Specifically, this embodiment provides: A cloud computing-based method for processing working data of intelligent terminals specifically includes the following steps: Step S10: Collect several types of work data generated by multiple smart terminals, extract the generation interval variance, data block dispersion and computational correlation of each type of work data based on the time-series sliding window, and output the data generation rate fluctuation range of each type of work data within a future preset time period based on the generation interval variance, data block dispersion and computational correlation through the LSTM prediction model. It's important to note that, firstly, several types of operational data (including business operation data, device status data, and interaction log data) generated by multiple smart terminals (such as mobile phones, IoT devices, and office terminals) need to be collected. This data is characterized by its temporal and diverse nature, and direct processing can easily lead to inefficiency due to load fluctuations. Therefore, a time-series sliding window is used to extract three key features: generation interval variance (reflecting the stability of data generation; a larger variance indicates more uneven data generation), data block dispersion (reflecting the degree of data storage dispersion; high dispersion requires more integration resources), and computational correlation (characterizing the dependencies between different types of data; high correlation requires collaborative processing). An LSTM prediction model (excelling at capturing long-term dependencies in time-series data) is used to learn these three features, outputting the data generation rate fluctuation range for each type of data within a preset future timeframe. This provides a quantitative basis for subsequent task allocation, specifically avoiding overload of processing nodes due to sudden data growth or idle resources due to insufficient data volume, thus facilitating subsequent processing.

[0031] Step S20: The computing nodes of the cloud computing cluster are divided into basic processing pool, elastic expansion pool and emergency buffer pool according to hardware configuration. Based on the fluctuation range of data generation rate of each type of work data and the preset processing time requirements, the corresponding data processing tasks are allocated using a pooling mapping algorithm. It's worth noting that the computing nodes of the cloud computing cluster are then divided into three functional pools based on hardware configuration: a basic processing pool (composed of nodes with stable configurations and moderate computing power, suitable for processing regular, low-fluctuation data), an elastic expansion pool (composed of dynamically expandable nodes to handle scenarios with large fluctuations in data generation rates), and an emergency buffer pool (composed of high-performance nodes for processing data with extremely high timeliness requirements or sudden peak data). This division method achieves "resource matching on demand." Combining the fluctuation range of each type of data generation rate (to determine the load pressure of data processing) and preset processing timeliness requirements (e.g., real-time business data requires second-level processing, log data can be processed in minutes), a pooling mapping algorithm allocates data processing tasks to the corresponding processing pool, ensuring precise matching between tasks and resource supply. This facilitates subsequent processing.

[0032] Step S30: According to the data processing task, allocate corresponding working data to the computing nodes in the basic processing pool, the elastic expansion pool and the emergency buffer pool and perform parallel processing to generate several data processing subsets accordingly. It should be noted that, subsequently, based on the assigned processing tasks, corresponding working data is distributed to the computing nodes in the three types of processing pools and parallel processing is performed. Specifically, parallel processing can significantly improve data processing efficiency, especially suitable for large-scale processing scenarios involving multiple terminals and multiple types of data, ultimately generating several data processing subsets (i.e., processing results of a single type of data or a single task unit) to facilitate subsequent processing.

[0033] Step S40: Complete the integrity verification of several data processing subsets through hash verification, construct the result association map according to the correlation between several data processing subsets, and integrate them into the corresponding target processing data set.

[0034] It's worth noting that, finally, hash verification (an efficient integrity verification method that determines whether data has been tampered with or lost by calculating the data's hash value) ensures the integrity of all data processing subsets. Simultaneously, considering the potential business relationships between different subsets (such as operation data and status data from the same terminal), a result correlation graph is constructed based on the correlation between subsets. This graph visually presents the logical relationships between the data, ultimately integrating them into a structurally complete and clearly correlated target processing data set. This provides reliable data support for subsequent business analysis or application calls, facilitating subsequent processing.

[0035] Second Embodiment Furthermore, the step of allocating corresponding data processing tasks using a pooling mapping algorithm based on the data generation rate fluctuation range and preset processing time requirements for each type of working data includes: Extract the data dimensions, number of field types, and encryption level of each type of work data to calculate the corresponding data complexity coefficient, and statistically analyze the task completion rate and processing accuracy of the basic processing pool, the elastic expansion pool, and the emergency buffer pool within the preset processing cycle to generate the corresponding priority matrix. Based on the fluctuation range of the generation rate fluctuation range of each type of working data, the corresponding target processing pool is matched in the priority matrix. Data processing tasks are assigned to the target processing pool based on its target attributes and the preset processing time requirements.

[0036] It is important to note that, firstly, two key indicators need to be quantified: The first is the processing difficulty of the data itself, namely, extracting the data dimensions of each type of working data (e.g., simple data with 10 fields vs. complex data with 100 fields), the number of field types (e.g., only text fields vs. multiple types of fields including text, numbers, and images), and the encryption level (encrypted data requires additional decryption steps, resulting in higher processing costs). A "data complexity coefficient" is calculated through weighted averages. Specifically, a higher coefficient indicates greater data processing difficulty and requires more computing power. The second indicator is the processing capacity and reliability of the processing pool, namely, the task completion rate (reflecting processing efficiency) and processing accuracy rate (reflecting processing quality) of the basic processing pool, elastic expansion pool, and emergency buffer pool within a preset processing period (e.g., the past 24 hours). These two indicators are integrated according to preset weights to generate a "priority matrix," where each element corresponds to the adaptation priority of a certain type of processing pool for a certain type of data.

[0037] Based on the aforementioned quantitative indicators, and according to the fluctuation range of the generation rate of each type of working data (the greater the fluctuation range, the higher the requirement for the elasticity or emergency response capability of the processing pool), the target processing pool is matched in the priority matrix. For example, data with small fluctuation range and low complexity is matched with the entry with the highest score in the basic processing pool in the priority matrix; data with large fluctuation range and high timeliness requirements is matched with the entry with the highest score in the emergency buffer pool or elastic expansion pool. Finally, combining the core attributes of the target processing pool (such as the number of nodes and the upper limit of computing power) and the preset processing timeliness requirements, the corresponding data processing tasks are allocated. Specifically, the core of this step is to ensure that the task allocation not only conforms to the capacity boundary of the processing pool but also meets the time constraints of data processing, avoiding situations where "large tasks are assigned to small resource pools" leading to timeouts or "small tasks are assigned to large resource pools" causing waste.

[0038] Furthermore, the step of allocating corresponding data processing tasks to the target processing pool based on its target attributes and the preset processing time requirements includes: The number of computing nodes, single-node computing power threshold, storage bandwidth limit and concurrent processing channel number of the target processing pool are extracted and quantized into corresponding resource supply vectors according to preset weights. Based on the data complexity coefficient of each type of work data, the preset processing time is converted into the corresponding maximum processing delay threshold, and the corresponding data processing task is divided into several independent task units according to the correlation of data fields. Each independent task unit is assigned a unique resource matching identifier according to the dimension of the resource supply vector. The processing parameters of each computing node in the target processing pool are collected at preset intervals, and the allocation weight of each independent task unit is dynamically adjusted according to the processing parameters to complete the data processing task according to the allocation weight.

[0039] It is important to note that, firstly, the resource supply capacity of the target processing pool needs to be clearly defined: extract the number of computing nodes in the processing pool (reflecting the scale of parallel processing), the single-node computing power threshold (reflecting the upper limit of single-node processing), the upper limit of storage bandwidth (reflecting data transmission and storage efficiency), and the number of concurrent processing channels (reflecting the number of tasks processed simultaneously). By pre-setting weights (such as a computing power threshold weight of 0.4, a bandwidth upper limit weight of 0.3, a node number weight of 0.2, and a concurrent channel number weight of 0.1), these heterogeneous parameters are quantified into a unified-dimensional "resource supply vector," so that the resource capacity of the processing pool can be quantitatively calculated and compared.

[0040] Meanwhile, based on the data complexity coefficient of each type of work data, the abstract "preset processing timeliness requirement" is converted into a specific "maximum processing delay threshold" (for example, for high-difficulty data with a complexity coefficient of 0.8, if the timeliness requirement is "real-time", the maximum processing delay threshold is set to 1 second; for simple data with a complexity coefficient of 0.3, if the timeliness requirement is "near real-time", the threshold is set to 5 seconds), ensuring that the timeliness requirement can be implemented. To improve parallel processing efficiency, data processing tasks are divided into several independent task units according to the correlation of data fields (such as grouping fields in the same business process together). Specifically, independent task units can be executed in parallel on different computing nodes, and the units do not interfere with each other, significantly shortening the overall processing time.

[0041] Next, a unique "resource matching identifier" is assigned to each independent task unit (used to track the binding relationship between tasks and computing nodes), and a dynamic adjustment mechanism is established: every preset time interval (e.g., 100 milliseconds), processing parameters (such as real-time processor utilization, memory usage, and task completion progress) of each computing node in the target processing pool are collected. Based on these parameters, the allocation weight of each independent task unit is adjusted (for example, if a node's processor utilization drops from 70% to 30%, the task weight allocated to that node is increased; if a node's utilization rises to 90%, the weight is decreased). This dynamic adjustment ensures balanced load across computing nodes, preventing some nodes from being overloaded and others from being idle. Ultimately, the data processing task is completed according to the adjusted allocation weights, achieving a dual optimization of resource utilization and processing efficiency, thus facilitating subsequent processing.

[0042] Furthermore, the step of allocating corresponding working data to computing nodes in the basic processing pool, the elastic expansion pool, and the emergency buffer pool according to the data processing task and performing parallel processing to generate several data processing subsets includes: The processor utilization rate, memory utilization rate and network transmission latency of each computing node in the basic processing pool, the elastic expansion pool and the emergency buffer pool are collected. Combined with the data block size and data type of each type of work data, each type of work data is split into several data fragments. Each data fragment carries a corresponding category identifier and time sequence generation stamp. The corresponding data processing margin is calculated based on the processor utilization, memory usage and network transmission latency of each computing node, and a suitable data fragment is allocated to each computing node according to the data processing margin. After each computing node obtains the corresponding data fragment, data deduplication, format standardization, and key information extraction operations are performed sequentially according to the order of the time-series generated stamps. The fragment processing progress is synchronized through the preset TCP synchronous communication port between each node to generate several data processing subsets.

[0043] It's important to note that, firstly, it's crucial to monitor the operational status of computing nodes in real time: This involves collecting data on the processor utilization (reflecting computational load), memory usage (reflecting storage load), and network latency (reflecting data transmission efficiency) of each computing node in the basic processing pool, elastic expansion pool, and emergency buffer pool. These parameters directly determine the node's remaining processing capacity. Secondly, considering the block size (e.g., 100MB vs. 1GB) and data type (e.g., structured data vs. unstructured data) of each type of working data, each type of data is split into several data fragments. Specifically, the purpose of this splitting is to ensure the data size is adapted to the processing capacity of a single computing node, preventing node processing timeouts due to excessively large data blocks. Each data fragment carries a category identifier (distinguishing its data type) and a time-series generation stamp (recording the data generation time), providing a basis for subsequent data integration and traceability.

[0044] Based on the processor utilization, memory usage, and network transmission latency of each computing node, the "data processing margin" (i.e., the additional data processing capacity that a node can currently handle, margin = node's maximum processing capacity - current capacity) is calculated. Appropriate data shards are then allocated to each node according to the processing margin: for example, nodes with a processing margin of 80% are allocated larger or more complex data shards, while nodes with a processing margin of 30% are allocated smaller or simpler shards, ensuring that the load is balanced across all nodes.

[0045] After each computing node receives the data shards, it executes a standardized processing flow according to the order of the time-series generated stamps: first, data deduplication (removing duplicate records to avoid redundant calculations); second, format standardization (converting heterogeneous data from different terminals into a unified format, such as unifying date formats to "YYYY-MM-DD" and numerical units); and finally, extraction of key information (such as extracting core fields like operator and action information from log data). These three steps ensure the standardization and effectiveness of the data sharding. To avoid shard omissions or inconsistencies in progress during parallel processing, the sharding processing progress is synchronized in real time through a pre-defined TCP synchronization communication port between nodes (TCP protocol has reliable transmission characteristics, suitable for progress data synchronization). For example, after node A completes shard 1 processing, it immediately synchronizes its progress with other nodes and the coordinating node, ensuring that all shards proceed as planned, ultimately generating several structurally unified and information-complete data processing subsets for subsequent processing.

[0046] Furthermore, the step of synchronizing the fragment processing progress through preset TCP synchronous communication ports between various nodes to generate several data processing subsets includes: After each computing node completes its operation, a corresponding progress synchronization data packet is generated and uploaded to the coordination node of the cloud computing cluster through the preset TCP synchronization communication port. The coordination node constructs a corresponding processing progress matrix based on the progress synchronization data packet, and uses the processing progress matrix to determine in real time whether all the preset operations of each data segment have been completed. If the processing progress matrix indicates that all preset operations for data sharding have been completed, a corresponding subset encapsulation instruction is generated, and a corresponding index identifier is added to generate the corresponding data processing subset.

[0047] It should be noted that, firstly, after each computing node completes the deduplication, standardization, and key information extraction operations for data shards, it generates a corresponding "progress synchronization data packet". Specifically, the data packet contains information such as shard identifier, processing node ID, processing completion time, and processing result summary, and is uploaded to the coordinating node of the cloud computing cluster through a preset TCP synchronization communication port (the coordinating node is responsible for global task scheduling and status monitoring).

[0048] After receiving progress synchronization data packets from all nodes, the coordinating node constructs a "processing progress matrix": rows represent data shards, and columns represent processing status (e.g., "Not Started," "Processing," "Completed"). The matrix visually presents the processing progress of each data shard. The coordinating node uses the progress matrix to determine in real-time whether all preset operations for each data shard have been completed. Specifically, for example, if a data shard requires three steps—deduplication, standardization, and extraction—and the progress matrix shows all three steps marked as "Completed," then the shard is considered processed. If any step is not completed, a node retry or task migration mechanism is triggered.

[0049] If all preset operations for data sharding are completed, the coordinating node generates a "subset encapsulation instruction." This instruction includes metadata such as the shard's category identifier, time-series generation stamp, and associated shard ID. It also adds a unique index identifier to this data processing subset (for subsequent association graph construction and data traceability). Finally, the processed sharded data and metadata are integrated according to the encapsulation instruction to generate a structurally complete and traceable data processing subset. Specifically, this step ensures that each data subset can be accurately identified and associated, laying the foundation for subsequent integrity verification and result integration, thus facilitating subsequent processing.

[0050] Furthermore, the step of constructing a result association graph based on the correlation between several data processing subsets and integrating it into a corresponding target processing data set includes: Extract the terminal scene label, processing task identifier, and key metadata of each data processing subset that passes the hash verification, and use the improved Jaccard similarity coefficient to calculate the cross-correlation degree between any two data processing subsets based on the terminal scene label, the processing task identifier, and the key metadata to generate the corresponding correlation matrix; Each of the data processing subsets is mapped to a graph node of the result association graph, and an edge between any two graph nodes is created according to the association degree matrix. Based on the edge type, the upstream and downstream dependency chains between each of the data processing subsets are sorted out, so as to integrate the target processing data set according to the upstream and downstream dependency chains.

[0051] It should be noted that, firstly, for the valid data processing subsets that pass hash verification, three types of core information are extracted: terminal scenario tags (such as "office scenario" and "industrial monitoring scenario," identifying the application scenario to which the data belongs), processing task identifiers (identifying the processing task corresponding to the data, ensuring association with the task subset), and key metadata (such as data generation time, terminal ID, and core field summary). An improved Jaccard similarity coefficient is used to calculate the "cross-association degree" of any two data processing subsets. Specifically, the Jaccard similarity coefficient is commonly used to calculate set similarity; the improved version can adapt to the comprehensive comparison of multi-dimensional information (tags, identifiers, metadata). The higher the correlation degree value, the stronger the logical association between the two subsets. Based on this, an "association degree matrix" is generated (the matrix elements are the correlation degree values ​​of any two subsets).

[0052] Each data processing subset is mapped to a "graph node" in the result association graph. Node attributes include the extracted labels, identifiers, and metadata. "Edges" are created between nodes based on the association degree matrix: solid edges (representing strong association) are established between nodes with an association degree higher than a preset threshold (e.g., 0.7); dashed edges (representing weak association) are established between nodes with an association degree lower than the threshold but higher than the minimum association threshold (e.g., 0.3); and no edges are established between nodes with no association. Through the construction of graph nodes and edges, the association relationships of all data subsets are visually presented. Finally, based on the edge type (strong association / weak association), the "upstream and downstream dependency chains" between subsets are sorted out (e.g., the "login operation data" subset of terminal A is upstream of the "business operation data" subset, and "business operation data" is upstream of "device status data"). All data processing subsets are integrated according to the logical order of the dependency chains to avoid logical confusion in the integrated data, ultimately forming a clearly structured and coherent target processing data set for subsequent processing.

[0053] Furthermore, the step of integrating the target processing data set according to the upstream and downstream dependency chains includes: The graph nodes are topologically sorted according to the order of the upstream and downstream dependency chains to generate several independent dependency data chains. Merge all field information in each dependent data chain and fill in missing field values ​​using metadata association rules; A semantic alignment algorithm based on knowledge graphs is used to remove semantically duplicate fields, and the data are aggregated sequentially according to the priority of each dependent data chain to generate the target processing data set.

[0054] It should be noted that, firstly, all nodes in the association graph are "topologically sorted" according to the order of upstream and downstream dependency chains. Specifically, topological sorting ensures that all nodes are arranged in order of dependency relationships, avoiding the logical error of "downstream data being integrated before upstream data". Ultimately, several independent "dependency data chains" are generated (such as the "login operation - business operation - data upload" dependency chain and the "device startup - status monitoring - fault alarm" dependency chain).

[0055] For each dependent data chain, merge the field information of all data processing subsets within it. For example, the "Terminal ID" and "Login Time" fields of the "Login Operation" subset are merged with the "Operation Content" and "Operation Duration" fields of the "Business Operation" subset to form a complete field set. Simultaneously, missing field values ​​are filled in using metadata association rules. Specifically, for example, if a subset is missing the "Terminal Model" field, it can be filled in by querying other subsets or the terminal basic information database using the associated "Terminal ID," ensuring field completeness.

[0056] Finally, a knowledge graph-based semantic alignment algorithm is used to remove semantically duplicated fields. Specifically, for example, "Terminal ID" and "Device Number" essentially refer to the same information; semantic alignment identifies and retains only one of these fields, avoiding data redundancy. All fields are then aggregated sequentially according to the priority of each dependent data chain (e.g., the core business chain has higher priority than auxiliary chains). Core fields from high-priority chains are retained first, followed by supplementary fields from lower-priority chains. This ultimately generates a complete, non-redundant, and logically coherent target processing data set, meeting the needs of subsequent business analysis while ensuring efficient data storage and retrieval, thus facilitating subsequent processing.

[0057] Please see Figure 2 The third embodiment of the present invention provides: A cloud computing-based intelligent terminal working data processing system, wherein the system includes: The acquisition module is used to collect several types of work data generated by multiple smart terminals. Based on the time-series sliding window, it extracts the generation interval variance, data block dispersion and computational correlation of each type of work data. Then, through the LSTM prediction model, it outputs the data generation rate fluctuation range of each type of work data within a future preset time according to the generation interval variance, data block dispersion and computational correlation. The partitioning module is used to divide the computing nodes of the cloud computing cluster into basic processing pools, elastic expansion pools and emergency buffer pools according to hardware configuration, and to allocate corresponding data processing tasks based on the data generation rate fluctuation range and preset processing time requirements of each type of work data using a pooling mapping algorithm. The allocation module is used to allocate corresponding working data to the computing nodes in the basic processing pool, the elastic expansion pool and the emergency buffer pool according to the data processing task and perform parallel processing to generate several data processing subsets accordingly. The integration module is used to complete the integrity verification of several data processing subsets through hash verification, construct the result association map according to the correlation between the several data processing subsets, and integrate them into the corresponding target processing data set.

[0058] Furthermore, the partitioning module is specifically used for: Extract the data dimensions, number of field types, and encryption level of each type of work data to calculate the corresponding data complexity coefficient, and statistically analyze the task completion rate and processing accuracy of the basic processing pool, the elastic expansion pool, and the emergency buffer pool within the preset processing cycle to generate the corresponding priority matrix. Based on the fluctuation range of the generation rate fluctuation range of each type of working data, the corresponding target processing pool is matched in the priority matrix. Data processing tasks are assigned to the target processing pool based on its target attributes and the preset processing time requirements.

[0059] Furthermore, the partitioning module is specifically used for: The number of computing nodes, single-node computing power threshold, storage bandwidth limit and concurrent processing channel number of the target processing pool are extracted and quantized into corresponding resource supply vectors according to preset weights. Based on the data complexity coefficient of each type of work data, the preset processing time is converted into the corresponding maximum processing delay threshold, and the corresponding data processing task is divided into several independent task units according to the correlation of data fields. Each independent task unit is assigned a unique resource matching identifier according to the dimension of the resource supply vector. The processing parameters of each computing node in the target processing pool are collected at preset intervals, and the allocation weight of each independent task unit is dynamically adjusted according to the processing parameters to complete the data processing task according to the allocation weight.

[0060] Furthermore, the allocation module is specifically used for: The processor utilization rate, memory utilization rate and network transmission latency of each computing node in the basic processing pool, the elastic expansion pool and the emergency buffer pool are collected. Combined with the data block size and data type of each type of work data, each type of work data is split into several data fragments. Each data fragment carries a corresponding category identifier and time sequence generation stamp. The corresponding data processing margin is calculated based on the processor utilization, memory usage and network transmission latency of each computing node, and a suitable data fragment is allocated to each computing node according to the data processing margin. After each computing node obtains the corresponding data fragment, data deduplication, format standardization, and key information extraction operations are performed sequentially according to the order of the time-series generated stamps. The fragment processing progress is synchronized through the preset TCP synchronous communication port between each node to generate several data processing subsets.

[0061] Furthermore, the allocation module is specifically used for: After each computing node completes its operation, a corresponding progress synchronization data packet is generated and uploaded to the coordination node of the cloud computing cluster through the preset TCP synchronization communication port. The coordination node constructs a corresponding processing progress matrix based on the progress synchronization data packet, and uses the processing progress matrix to determine in real time whether all the preset operations of each data segment have been completed. If the processing progress matrix indicates that all preset operations for data sharding have been completed, a corresponding subset encapsulation instruction is generated, and a corresponding index identifier is added to generate the corresponding data processing subset.

[0062] Furthermore, the integration module is specifically used for: Extract the terminal scene label, processing task identifier, and key metadata of each data processing subset that passes the hash verification, and use the improved Jaccard similarity coefficient to calculate the cross-correlation degree between any two data processing subsets based on the terminal scene label, the processing task identifier, and the key metadata to generate the corresponding correlation matrix; Each of the data processing subsets is mapped to a graph node of the result association graph, and an edge between any two graph nodes is created according to the association degree matrix. Based on the edge type, the upstream and downstream dependency chains between each of the data processing subsets are sorted out, so as to integrate the target processing data set according to the upstream and downstream dependency chains.

[0063] Furthermore, the integration module is specifically used for: The graph nodes are topologically sorted according to the order of the upstream and downstream dependency chains to generate several independent dependency data chains. Merge all field information in each dependent data chain and fill in missing field values ​​using metadata association rules; A semantic alignment algorithm based on knowledge graphs is used to remove semantically duplicate fields, and the data are aggregated sequentially according to the priority of each dependent data chain to generate the target processing data set.

[0064] The fourth embodiment of the present invention provides a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the cloud computing-based smart terminal working data processing method described above.

[0065] The fifth embodiment of the present invention provides a readable storage medium on which a computer program is stored, wherein when the program is executed by a processor, it implements the cloud computing-based smart terminal working data processing method described above.

[0066] In summary, the cloud computing-based smart terminal working data processing method and system provided in the above embodiments of the present invention can dynamically and comprehensively complete the processing of working data of various smart terminals, thereby improving data processing efficiency.

[0067] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0068] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0069] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0070] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0071] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0072] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for processing working data of a smart terminal based on cloud computing, characterized in that, The method includes: Collect several types of work data generated by multiple smart terminals, extract the generation interval variance, data block dispersion and computational correlation of each type of work data based on a time-series sliding window, and output the data generation rate fluctuation range of each type of work data within a future preset time period based on the generation interval variance, data block dispersion and computational correlation through an LSTM prediction model. The computing nodes of the cloud computing cluster are divided into basic processing pool, elastic expansion pool and emergency buffer pool according to hardware configuration. Based on the fluctuation range of data generation rate of each type of work data and the preset processing time requirements, the corresponding data processing tasks are allocated using a pooling mapping algorithm. According to the data processing task, the corresponding working data is allocated to the computing nodes in the basic processing pool, the elastic expansion pool and the emergency buffer pool and parallel processing is performed to generate several data processing subsets accordingly. The integrity of several data processing subsets is verified by hash verification, and a result association graph is constructed based on the correlation between the several data processing subsets, and integrated into the corresponding target processing data set.

2. The cloud computing-based intelligent terminal working data processing method according to claim 1, characterized in that, The step of allocating corresponding data processing tasks using a pooling mapping algorithm based on the data generation rate fluctuation range and preset processing time requirements for each type of working data includes: Extract the data dimensions, number of field types, and encryption level of each type of work data to calculate the corresponding data complexity coefficient, and statistically analyze the task completion rate and processing accuracy of the basic processing pool, the elastic expansion pool, and the emergency buffer pool within the preset processing cycle to generate the corresponding priority matrix. Based on the fluctuation range of the generation rate fluctuation range of each type of working data, the corresponding target processing pool is matched in the priority matrix. Data processing tasks are assigned to the target processing pool based on its target attributes and the preset processing time requirements.

3. The cloud computing-based intelligent terminal working data processing method according to claim 2, characterized in that, The step of allocating corresponding data processing tasks to the target processing pool based on its target attributes and the preset processing time requirements includes: The number of computing nodes, single-node computing power threshold, storage bandwidth limit and concurrent processing channel number of the target processing pool are extracted and quantized into corresponding resource supply vectors according to preset weights. Based on the data complexity coefficient of each type of work data, the preset processing time is converted into the corresponding maximum processing delay threshold, and the corresponding data processing task is divided into several independent task units according to the correlation of data fields. Each independent task unit is assigned a unique resource matching identifier according to the dimension of the resource supply vector. The processing parameters of each computing node in the target processing pool are collected at preset intervals, and the allocation weight of each independent task unit is dynamically adjusted according to the processing parameters to complete the data processing task according to the allocation weight.

4. The cloud computing-based intelligent terminal working data processing method according to claim 1, characterized in that, The step of allocating corresponding working data to computing nodes in the basic processing pool, the elastic expansion pool, and the emergency buffer pool according to the data processing task and performing parallel processing to generate several data processing subsets includes: Collect the processor utilization rate, memory utilization rate and network transmission latency of each computing node in the basic processing pool, the elastic expansion pool and the emergency buffer pool. Combine the data block size and data type of each type of work data, split each type of work data into several data fragments. Each data fragment carries a corresponding category identifier and time sequence generation stamp. The corresponding data processing margin is calculated based on the processor utilization, memory usage and network transmission latency of each computing node, and a data fragment suitable for it is allocated to each computing node according to the data processing margin. After each computing node obtains the corresponding data fragment, data deduplication, format standardization, and key information extraction operations are performed sequentially according to the order of the time-series generated stamps. The fragment processing progress is synchronized through the preset TCP synchronous communication port between each node to generate several data processing subsets.

5. The cloud computing-based intelligent terminal working data processing method according to claim 4, characterized in that, The step of synchronizing the fragment processing progress through preset TCP synchronous communication ports between various nodes to generate several data processing subsets includes: After each computing node completes its operation, a corresponding progress synchronization data packet is generated and uploaded to the coordination node of the cloud computing cluster through the preset TCP synchronization communication port. The coordination node constructs a corresponding processing progress matrix based on the progress synchronization data packet, and uses the processing progress matrix to determine in real time whether all the preset operations of each data segment have been completed. If the processing progress matrix indicates that all preset operations for data sharding have been completed, a corresponding subset encapsulation instruction is generated, and a corresponding index identifier is added to generate the corresponding data processing subset.

6. The cloud computing-based intelligent terminal working data processing method according to claim 1, characterized in that, The step of constructing a result association graph based on the correlation between several data processing subsets and integrating it into a corresponding target processing data set includes: Extract the terminal scene label, processing task identifier, and key metadata of each data processing subset that passes the hash verification, and use the improved Jaccard similarity coefficient to calculate the cross-correlation degree between any two data processing subsets based on the terminal scene label, the processing task identifier, and the key metadata to generate the corresponding correlation matrix; Each of the data processing subsets is mapped to a graph node of the result association graph, and an edge between any two graph nodes is created according to the association degree matrix. Based on the edge type, the upstream and downstream dependency chains between each of the data processing subsets are sorted out, so as to integrate the target processing data set according to the upstream and downstream dependency chains.

7. The cloud computing-based intelligent terminal working data processing method according to claim 6, characterized in that, The step of integrating the target processing data set according to the upstream and downstream dependency chains includes: The graph nodes are topologically sorted according to the order of the upstream and downstream dependency chains to generate several independent dependency data chains. Merge all field information in each dependent data chain and fill in missing field values ​​using metadata association rules; A semantic alignment algorithm based on knowledge graphs is used to remove semantically duplicate fields, and the data are aggregated sequentially according to the priority of each dependent data chain to generate the target processing data set.

8. A cloud computing-based intelligent terminal working data processing system, characterized in that, The system includes: The acquisition module is used to collect several types of work data generated by multiple smart terminals. Based on the time-series sliding window, it extracts the generation interval variance, data block dispersion and computational correlation of each type of work data. Then, through the LSTM prediction model, it outputs the data generation rate fluctuation range of each type of work data within a future preset time according to the generation interval variance, data block dispersion and computational correlation. The partitioning module is used to divide the computing nodes of the cloud computing cluster into basic processing pools, elastic expansion pools and emergency buffer pools according to hardware configuration, and to allocate corresponding data processing tasks based on the data generation rate fluctuation range and preset processing time requirements of each type of work data using a pooling mapping algorithm. The allocation module is used to allocate corresponding working data to the computing nodes in the basic processing pool, the elastic expansion pool and the emergency buffer pool according to the data processing task and perform parallel processing to generate several data processing subsets accordingly. The integration module is used to complete the integrity verification of several data processing subsets through hash verification, construct the result association map according to the correlation between the several data processing subsets, and integrate them into the corresponding target processing data set.

9. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the cloud computing-based smart terminal working data processing method as described in any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the cloud computing-based smart terminal working data processing method as described in any one of claims 1 to 7.

Images