Enterprise data intelligent processing method and system based on cloud edge collaboration

Abstract

The application discloses an enterprise data intelligent processing method and system based on cloud edge cooperation, relates to the technical field of cloud computing, and comprises the following steps: collecting a plurality of data collection sources of a target enterprise; analyzing local data calling transmission scenes of the plurality of data collection sources to establish a plurality of local calling relationship networks; then analyzing local calling demands and planning edge end data processing paths according to the local calling relationship networks, and generating the edge end data processing paths; finally analyzing cloud end joint calling demands, taking the edge end data processing paths as the basis, executing edge end to cloud end data processing path planning, and obtaining edge-cloud end data processing path planning. The application solves the technical problem that enterprise data cannot adapt to differentiated processing demands of multi-source data in local and cloud end cooperative processing, leading to disordered data processing paths and resource waste, and achieves the technical effects of adapting to differentiated processing demands of multi-source data of enterprises, realizing efficient processing of enterprise data and data value conversion.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a method and system for intelligent processing of enterprise data based on cloud-edge collaboration. Background Technology

[0002] In enterprise data processing scenarios, data intelligence is crucial for business decision-making, operational efficiency, and resource optimization, and efficient collaborative processing of multi-source data is one of the key links in realizing data value. Currently, enterprise data processing largely relies on either centralized cloud processing or independent edge processing. The former requires uploading all data to the cloud, which can easily lead to processing efficiency issues due to transmission latency and excessive bandwidth consumption; the latter is limited by edge computing power and storage, making it unable to handle complex collaborative data processing needs. Because enterprise data sources are dispersed and access scenarios are diverse, pure cloud or pure edge models struggle to adapt to the differentiated processing requirements of data, resulting in chaotic data processing paths, resource waste, and an inability to efficiently transform data value, thus failing to meet enterprises' demands for accuracy and efficiency in data intelligence processing. Summary of the Invention

[0003] This application provides a cloud-edge collaborative enterprise data intelligent processing method and system to solve the technical problem that enterprise data cannot adapt to the differentiated processing needs of multi-source data in local and cloud collaborative processing, resulting in chaotic data processing paths and wasted resources.

[0004] The first aspect of this application provides a cloud-edge collaborative enterprise data intelligent processing method, the method comprising: collecting multiple data collection sources of a target enterprise; performing local data call and transmission scenario analysis on the multiple data collection sources to establish multiple local call relationship networks; performing local call demand analysis based on the multiple local call relationship networks, executing data processing path planning for the data collection sources at the edge, and generating an edge data processing path; performing cloud-to-edge joint call demand analysis based on the multiple local call relationship networks, and executing edge-to-cloud data processing path planning based on the edge data processing path to obtain an edge-to-cloud data processing path plan.

[0005] A second aspect of this application provides an enterprise data intelligent processing system based on cloud-edge collaboration. The system includes: a data collection source acquisition module for collecting data from multiple data collection sources of a target enterprise; a local call relationship network construction module for analyzing local data call and transmission scenarios from the multiple data collection sources and establishing multiple local call relationship networks; an edge data processing path acquisition module for analyzing local call requirements based on the multiple local call relationship networks, executing data processing path planning for the edge data collection sources, and generating an edge data processing path; and an edge-cloud data processing path planning acquisition module for analyzing cloud joint call requirements based on the multiple local call relationship networks, and, based on the edge data processing path, executing edge-to-cloud data processing path planning to obtain an edge-cloud data processing path plan.

[0006] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0007] This application, based on a cloud-edge collaborative architecture, first analyzes the local call transmission scenarios of multi-source data collection and establishes a local call relationship network. Then, it plans the edge data processing path accordingly. Finally, it optimizes and generates the edge-cloud data processing path plan by combining the cloud joint call requirements. This enables efficient and accurate collaborative processing of enterprise multi-source data in a cloud-edge environment, making enterprise data intelligent processing more efficient and flexible. It achieves the technical effect of adapting to the differentiated processing needs of enterprise multi-source data and realizing efficient data processing and data value transformation. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart illustrating the enterprise data intelligent processing method based on cloud-edge collaboration provided in the embodiments of this application.

[0010] Figure 2 This is a schematic diagram of the structure of the cloud-edge collaborative enterprise data intelligent processing system provided in the embodiments of this application.

[0011] Figure labeling: Data collection source acquisition module 1, local call relationship network construction module 2, edge data processing path acquisition module 3, edge-cloud data processing path planning and acquisition module 4. Detailed Implementation

[0012] This application provides a cloud-edge collaborative enterprise data intelligent processing method and system to solve the technical problem that enterprise data cannot adapt to the differentiated processing needs of multi-source data in local and cloud collaborative processing, resulting in chaotic data processing paths and wasted resources.

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0014] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0015] Example 1, as Figure 1 As shown, an enterprise data intelligent processing method based on cloud-edge collaboration is described, wherein the method includes:

[0016] Step A100: Collect data from multiple data sources for the target company.

[0017] Specifically, when collecting data from multiple sources of a target company, the specific types and distribution of data sources are first determined by combining the target company's business scenarios and operational architecture. For example, the data sources for manufacturing companies may include temperature sensors on production lines, equipment operation status monitoring terminals, and production order management systems, while the data sources for service companies may cover user interaction log systems, service order processing platforms, and customer feedback record modules. Based on the analysis of the data flow of various business modules within different types of companies, those skilled in the art can initially identify the scope of all data sources that need to be collected.

[0018] Next, for different types of data collection sources, appropriate acquisition methods are adopted to ensure the effectiveness of data acquisition: For hardware data collection sources, including sensors, monitoring terminals, etc., the Modbus real-time data acquisition protocol is deployed to acquire real-time data at a preset acquisition frequency of once per minute, where the specific acquisition frequency can be determined and modified according to the data generation cycle; For software data collection sources, including business management systems, log systems, etc., structured data, including order data, user information, etc., and unstructured data, including text logs, image files, etc., are periodically synchronized by calling the API interfaces open by the corresponding systems or establishing direct database connection channels, such as JDBC connection (Java database connection) or ODBC connection (Open Database Connection). At the same time, basic information of each data collection source is recorded, including the business scenario in which the data is generated, the data format, whether it is JSON format, CSV format or binary format, etc., and the data volume, including the size of the status data generated in a single instance and the size of the log file generated in a single instance.

[0019] During the data collection process, each data collection source needs to be uniquely identified and assigned a value. For example, the temperature sensor on production line 1 is assigned the identifier SC001-T001, and the user log system is assigned the identifier SY002-LOG01. Here, SC and SY are the abbreviations for the major category attributes of the sensor and system, respectively. Simultaneously, each data collection source is associated with its corresponding internal business number, such as a specific production workshop or business department, and the physical location or deployment node of the data collection source is recorded. For example, the sensor is installed on the north side of production line 1, and the log system is deployed on the company's intranet server. This ensures that each collected data source can be clearly distinguished and located, avoiding data source confusion in subsequent processing.

[0020] In addition, to ensure the integrity and accuracy of the collected data source information, a data verification mechanism needs to be added during the collection process: for hardware acquisition sources, the stability of the acquisition signal is verified, and the signal fluctuation is considered stable if it does not exceed 5% in three consecutive acquisitions; for software acquisition sources, data field verification can be used to check whether the order data contains required fields such as order number and amount, to ensure that the collected source information is complete. If an acquisition anomaly is found in a data collection source, such as no sensor signal or API call returning an error code, the backup acquisition scheme is activated, that is, switching to the backup sensor and calling the backup API interface until the information of the data collection source is successfully acquired.

[0021] By clarifying the types and distribution of data collection sources, adopting appropriate collection methods, performing unique identification and attribute recording, and adding data verification and anomaly handling, multiple data collection sources of the target enterprise were successfully collected, providing accurate and comprehensive basic data support for subsequent analysis of local data call and transmission scenarios and the establishment of a local call relationship network.

[0022] Step A200: Analyze the local data call and transmission scenarios of the multiple data collection sources and establish multiple local call relationship networks.

[0023] Optionally, to establish multiple local call relationship networks for multiple data collection sources, it is necessary to first collect multiple historical data flow records of these data collection sources and the cloud-edge collaborative architecture covering them, and then analyze the call and transmission relationship of local data generated by the data collection sources on the architecture nodes based on the historical data flow records, thereby establishing multiple local call relationship networks. The specific steps are explained in detail in A210-A230.

[0024] Step A300: Analyze local call requirements based on the multiple local call relationship networks, execute data processing path planning for the data collection source at the edge, and generate the edge data processing path.

[0025] In this embodiment of the application, the edge end is the specific edge computing node in the cloud-edge collaborative architecture responsible for processing local related data and requirements.

[0026] In one embodiment of this application, any local call network of any edge end is segmented according to multiple local call relationship networks. Within the local call network, each data collection source is clustered based on a preset call reference index, and a partitioned and hierarchical storage plan is constructed based on the results. Then, the edge end data processing path is constructed based on the plan. The specific steps are described in detail in A310-A330.

[0027] Step A400: Based on the multiple local call relationship networks, perform cloud joint call requirement analysis, and based on the edge data processing path, perform edge-to-cloud data processing path planning to obtain the edge-cloud data processing path planning.

[0028] In this embodiment, the cloud is the cloud center node in the cloud-edge collaborative architecture, which embeds the corresponding data analysis model. It can form data call relationships with the data collection source and the edge terminal through the cloud calling the network.

[0029] Specifically, the cloud call network is segmented based on multiple local call relationship networks. Its data transmission nodes are analyzed to segment the first and second cloud local call networks. In the first cloud local call network, the data collection source is subjected to secondary similarity clustering and the edge storage is optimized. In the second cloud local call network, the intermediate data at the edge is stored according to the receiving nodes in the cloud. The two are combined to generate the edge-cloud data processing path plan. The specific steps are explained in detail in A410-A440.

[0030] Furthermore, step A200 in the method provided in this application embodiment includes:

[0031] A210: Collect multiple historical data flow records from the multiple data collection sources.

[0032] A220: Cloud-edge collaborative architecture that collects data from multiple data collection sources.

[0033] A230: Based on the multiple historical data flow records, analyze the call and transmission relationships of local data generated by the multiple data collection sources in the cloud-edge collaborative architecture, and establish the multiple local call relationship network.

[0034] Specifically, firstly, for the multiple data collection sources identified by the target enterprise, corresponding data acquisition methods are used to obtain multiple historical data flow records. For hardware-based data collection sources, such as temperature sensors on the production line and equipment operation status monitoring terminals, by deploying acquisition modules using Modbus or OP CUA protocols, historical data flow information over the past few months can be extracted. Each record must include key information such as the data collection source identifier, data generation time, architecture node identifier to which the data is transmitted, single data transmission volume, and data call frequency. For example, the historical record of temperature sensor No. 1 on a certain production line shows that its data is transmitted to a designated edge node every 5 minutes from 9:00 to 18:00 every day, with a single data volume of 1KB and an average of 216 calls per day.

[0035] For software-based data collection sources, such as user order management systems and customer feedback record modules, data flow logs from the past few months can be synchronized by calling the system's open API interfaces or establishing a direct JDBC database connection channel. This records the complete path of data from system generation to transmission to the corresponding node. For example, historical records of a user log system show that the 10MB of log data generated daily is transmitted to a specific edge node within 30 minutes of generation and is called by the cloud 4 times per month for user behavior analysis.

[0036] After completing the collection of historical data flow records, further information on the cloud-edge collaborative architecture covering these data collection sources is collected. Based on existing cloud-edge collaborative architecture design specifications, the node composition of the architecture is first determined, clarifying the number of edge computing nodes, the number of cloud center nodes, and their respective physical deployment locations. For example, the cloud-edge collaborative architecture of a manufacturing enterprise includes 10 edge computing nodes deployed in 3 production workshops and 2 warehouse areas, and 1 cloud center node deployed in the enterprise data center.

[0037] Next, confirm the type of data analysis model embedded in each node. Edge computing nodes need to embed models that match local data processing needs. For example, edge nodes in production workshops embed equipment fault diagnosis models, edge nodes in warehouse areas embed inventory early warning models, and cloud center nodes embed global production efficiency optimization models. All of the above models are common data analysis models in the fields of industrial data processing and enterprise operation management.

[0038] Then, the connection relationship between each data collection source and the architecture node is sorted out to form a data collection source-edge computing node correspondence table. For example, the five temperature sensors and three pressure sensors in workshop 1 are all connected to the same edge computing node, while the user order system and financial settlement system are connected to another edge computing node. All edge computing nodes establish data interaction channels with the cloud center node to ensure that the collected architecture information can completely cover all the identified data collection sources.

[0039] Finally, based on multiple historical data flow records, the first historical data flow records of the first data collection source are extracted, and the flow nodes of the first local data generated by the data collection source in the cloud-edge collaborative architecture are analyzed to build the first local call relationship network. Then, it is added to multiple local call relationship networks to establish multiple local call relationship networks. The specific steps are explained in detail in A231-A233.

[0040] By employing adapted technical means to collect historical data flow records from multiple data collection sources, and by collecting data from a cloud-edge collaborative architecture covering all data collection sources, complete and accurate basic data support is provided for subsequent analysis of the call and transmission relationships of local data on architecture nodes based on historical data flow records, and for establishing multiple local call relationship networks.

[0041] Furthermore, step A220 in the method provided in this application embodiment includes:

[0042] A221: The cloud-edge collaborative architecture includes several edge computing nodes and cloud center nodes, and the edge computing nodes and cloud center nodes embed corresponding data analysis models. The corresponding data analysis models are input by calling local data generated from multiple data collection sources.

[0043] Optionally, when building a cloud-edge collaborative architecture, the deployment scheme for edge computing nodes and cloud center nodes should first be determined based on the distribution of data collection sources and business processing needs of the target enterprise. Taking a manufacturing enterprise as an example, based on the concentrated areas of data collection sources such as production lines and warehouses, 10 edge computing nodes are deployed, corresponding to 5 production lines and 2 warehouses respectively. Each edge computing node covers 5-8 surrounding data collection sources, such as temperature sensors and equipment operation monitoring terminals. At the same time, one cloud center node is deployed in the enterprise data center as the core of global data processing and collaboration, ensuring that the architecture can fully cover all data collection sources and provide the hardware foundation for subsequent data flow and processing.

[0044] After node deployment is complete, a corresponding data analysis model is embedded in each edge computing node and cloud center node. Edge computing nodes need to embed lightweight models adapted to local real-time processing needs. For example, edge nodes corresponding to production lines embed equipment temperature anomaly detection models, and edge nodes corresponding to warehouse areas embed inventory turnover rate analysis models. These models can quickly process small volumes of local data generated frequently, with a single model input data volume controlled at 1-5KB and a processing response time of no more than 1 second. Cloud center nodes, on the other hand, embed comprehensive models adapted to global collaborative analysis needs, such as global production efficiency optimization models and cross-departmental data correlation analysis models. These models can process large volumes of data aggregated from multiple edge nodes, with a single model input data volume reaching 50-100MB, supporting in-depth mining of the overall data value of the enterprise. The above example models are common data analysis models in the fields of industrial data processing and enterprise operation management.

[0045] Next, it is necessary to establish a call association between nodes and data collection sources to ensure that nodes can accurately call local data to input the corresponding data analysis model. Edge computing nodes directly call local data generated by data collection sources within their coverage area through industrial communication protocols such as Modbus and OPCUA. For example, the edge computing node corresponding to production line 1 calls real-time data from three surrounding temperature sensors and two pressure sensors every 2 seconds, formats the data, and inputs it into the equipment temperature anomaly detection model. Cloud center nodes call two types of data through the SD-WAN network: one is local data directly transmitted from data collection sources that needs to be processed globally, such as order data across production lines; the other is intermediate data output by edge computing nodes after processing, such as equipment anomaly statistics for each production line. Then, these data are integrated and input into the global production efficiency optimization model of the cloud center node to achieve precise matching between data calls and model processing.

[0046] By deploying edge computing nodes and cloud center nodes adapted to enterprise needs, embedding corresponding data analysis models, and establishing precise call associations between nodes and data collection sources, a complete cloud-edge collaborative architecture is built. This provides stable architectural support for subsequent analysis of local data call and transmission relationships and the establishment of a local call relationship network, ensuring that data can be processed through the appropriate model on the right node.

[0047] Furthermore, step A230 in the method provided in this application embodiment includes:

[0048] A231: Extract the first historical data flow record of the first data collection source.

[0049] A232: Based on the analysis of the first historical data flow records, analyze the flow nodes of the first local data generated by the first data collection source in the cloud-edge collaborative architecture, and build a first local call relationship network.

[0050] A233: Add the first local call relationship network into the plurality of local call relationship networks.

[0051] Specifically, firstly, for the primary data collection source of the target enterprise, for example, temperature sensor No. 1 on production line, identified as SC001-T001, the first historical data flow record is extracted using data log parsing tools such as Logstash or Fluentd. During the extraction process, complete flow information for the past three months of this data collection source is filtered. Each record must include the data generation time, the identifier of the cloud-edge collaborative architecture node involved, the amount of data transmitted in a single transaction, the data call direction, and the call frequency. Edge computing nodes can be identified as EXX, and cloud center nodes can be identified as CXX, where XX is the corresponding node number, ensuring that the extracted first historical data flow record can fully reflect the local data flow pattern of this data collection source.

[0052] Next, based on the extracted historical data flow records, professional network topology modeling tools, such as Neo4j graph database or Visio, are used to analyze the flow nodes of the first local data generated by the first data collection source in the cloud-edge collaborative architecture. By sorting out the node identifiers and calling relationships in the records, the flow path of the first local data is clarified: for example, after the first local data is generated from the first data collection source SC001-T001, it is first transmitted to the edge computing node E03. This node embeds a temperature anomaly detection model. After processing by the model, high-frequency real-time data, including temperature values ​​updated every 5 minutes, is stored locally on the edge computing node E03. Then, the summary data that needs to be analyzed globally, such as the hourly average temperature, is transmitted to the cloud center node C02. With SC001-T001, E03, and C02 as core nodes, and the transmission direction, frequency, and data volume between nodes as correlation attributes, a first local calling relationship network is built, intuitively presenting the calling and transmission logic of the first local data.

[0053] Then, repeat the above process for other data collection sources of the enterprise: extract their historical data flow records for the past few months, including the data generation time, information on transmission to edge computing nodes and cloud center nodes, as well as the size and frequency of a single transmission, the frequency of daily transmission, analyze the path of the flow nodes, build a second local call relationship network, and then add the first local call relationship network to multiple local call relationship networks to gradually expand the network set.

[0054] By extracting the historical flow records of the first data collection source and analyzing the flow nodes of its local data in the cloud-edge architecture, a first local call relationship network is built. This network is then incorporated into multiple local call relationship networks. This provides a precise basis for subsequent local call demand analysis and cloud-joint call demand analysis, as well as for planning edge and edge-cloud data processing paths. This ensures that subsequent path planning aligns with the actual flow characteristics of each data collection source.

[0055] Furthermore, step A300 in the method provided in this application embodiment includes:

[0056] A310: Based on the multiple local call relationship networks, any local call network at any edge end is segmented.

[0057] A320: In any local call network, perform a similarity clustering of each data collection source according to a preset call reference index, and construct a partitioned and hierarchical storage plan based on the clustering results.

[0058] A330: Construct the edge data processing path based on the partitioned and hierarchical storage plan.

[0059] Specifically, firstly, for the multiple established local call relationship networks, and combining the business coverage and data processing boundaries of the target edge, network topology segmentation tools such as GraphCut and Metis are used to segment any local call network of any edge. Taking edge E01 covering production line 1 as an example, the range of data collection sources that this edge needs to process is first defined, including 5 temperature sensors of production line 1, labeled SC001-T001 to SC001-T005; 3 pressure sensors, labeled SC001-P001 to SC001-P003; and 1 equipment operation status terminal, labeled SC001-S001.

[0060] Then, the call transmission relationships between these data collection sources and their associated nodes with the edge terminal E01 are filtered out from the multiple local call relationship networks constructed in the previous steps. Finally, a local call network of the edge terminal E01 is formed. This local call network contains 9 data collection source nodes, 1 edge computing node and multiple call transmission records between nodes, ensuring that the segmented local network can fully reflect the local data call scenario of the edge terminal E01.

[0061] Next, within this local call network, the K-Means clustering algorithm is used to perform a similarity clustering of each data collection source according to preset call reference indicators. The preset call reference indicators include data type, data generation cycle, single data size, data lifespan, and call frequency: In the example above, the data generated by the 5 temperature sensors and 3 pressure sensors is real-time monitoring data, the data type is numerical real-time data, the data generation cycle is 5 minutes / time, the single data size is 1KB, the data lifespan is 1 month, and the call frequency is 288 times / day, which is assigned to the first cluster group; the data generated by the 1 device operation status terminal is batch log data, the data type is text batch data, the data generation cycle is 1 hour / time, the single data size is 10MB, the data lifespan is 6 months, and the call frequency is 24 times / day, which is assigned to the second cluster group. Simultaneously, the storage media information of the edge terminal E01 is collected. This edge terminal is equipped with two types of storage media: SSD and HDD. The former has a significantly faster read speed than the latter. According to storage performance, SSD is divided into primary storage and HDD is divided into secondary storage. Combined with the results of a clustering, a partitioned and hierarchical storage plan is constructed. That is, the real-time monitoring data of the first cluster group is stored in the primary SSD, and the batch log data of the second cluster group is stored in the secondary HDD.

[0062] Subsequently, based on the constructed partitioned and hierarchical storage plan, and combined with the real-time requirements and data flow logic of edge data processing, an edge data processing path was constructed. For real-time monitoring data of the first cluster group, after being generated from the data collection source, it is directly transmitted to the primary SSD storage area of ​​edge E01 via industrial Ethernet. The device anomaly detection model embedded in the edge prioritizes reading data from the SSD for real-time processing, and the processed anomaly warning results are stored on the SSD for subsequent high-frequency calls. For batch log data of the second cluster group, after being generated from the data collection source, it is first cached in the edge temporary storage area. After the data is summarized every hour, it is transmitted to the secondary HDD storage area. The device operation trend analysis model at the edge periodically reads data from the HDD for batch processing, and the processed trend report is synchronized to the primary SSD or stored in the HDD as needed.

[0063] By constructing the edge data processing path through the above steps, the local call requirements of the edge are met, the efficiency of edge data processing and the reasonable allocation of storage resources are achieved, and the foundation is laid for the subsequent planning of the data processing path from the edge to the cloud.

[0064] Furthermore, step A320 in the method provided in this application embodiment includes:

[0065] A321: The preset call reference indicators include at least the data type, data generation cycle, single data size, data lifecycle, and call frequency.

[0066] Specifically, firstly, for the multiple data collection sources of the target enterprise, corresponding data format parsing tools such as JSON parsers, CSV parsers, and binary data decoders are used to determine the data type of each data collection source. Taking a manufacturing enterprise as an example, three temperature sensors on production line 1 generate real-time numerical data, each data entry containing a timestamp and temperature value; two user log systems in the customer service department generate text data, each data entry recording the user ID, operation behavior, and operation time; and one monthly report system in the finance department generates tabular structured data, including fields such as month, revenue amount, and cost data. The aforementioned data format parsing tools can accurately distinguish these three data types, providing a basic classification basis for subsequent clustering.

[0067] Next, using data acquisition timers and data logging tools, the data generation cycle of each data collection source was statistically analyzed. Specifically, the temperature sensor needs to monitor device status in real time, with a data generation cycle set at 5 minutes per instance, accumulating 288 data entries per day; the user log system needs to record real-time user operations, with a data generation cycle of 10 seconds per instance, accumulating 8640 data entries per day; and the monthly report system only needs to summarize the financial data for the current month, with a data generation cycle of 30 days per instance, accumulating 12 data entries per day.

[0068] Subsequently, a data byte statistics tool was used to measure the size of each data collection source's single-time data output. Temperature sensor data output contained only three core fields: timestamp, sensor identifier, and temperature value, with a single data output size of approximately 1KB. User log system data output contained five fields: user ID, operation behavior, operation page, timestamp, and device model, with a single data output size of approximately 500B. Monthly report system data output contained 10 dimensions of financial indicators and detailed data, with a single data output size of approximately 20MB. Through measurements of 100 random data samples, the average size of each data type's single-time output was calculated, with an error controlled within 5%, providing a quantitative reference for subsequent storage resource allocation.

[0069] Then, using a data lifecycle management tool, combined with the company's business needs and compliance requirements, the data lifecycle for each data collection source is determined. Temperature data is mainly used for real-time equipment fault early warning; its warning value decreases after one month, so its lifecycle is set to one month. User log data needs to be used for quarterly user behavior analysis, so its lifecycle is set to six months. Monthly report data needs to meet financial audit requirements, so its lifecycle can be set to five years. After configuring lifecycle rules in the data lifecycle management tool, the entire process of data from generation, activation, archiving to destruction is automatically recorded, ensuring compliance in data lifecycle management.

[0070] Finally, using data access log analysis tools, the frequency of data access from each data collection source was statistically analyzed. Temperature data is accessed 12 times per hour by the edge device fault diagnosis model, totaling 288 times per day; user log data is accessed once every 6 hours by the cloud-based user profiling model, totaling 4 times per day; monthly report data is only accessed by management when viewing reports, totaling 24 times per month and 24 times per year. By analyzing historical access logs, the average access frequency of each data type was determined, clearly defining the boundary between high-frequency and low-frequency data access.

[0071] By employing technologies such as data format parsing, timed statistics, byte measurement, lifecycle management, and access log analysis, specific data of preset call reference indicators for each data collection source are obtained. This provides a quantitative basis for subsequent similarity clustering of data collection sources in the local call network and the construction of partitioned and hierarchical storage planning, ensuring that the clustering results and storage planning can accurately match the actual data processing needs.

[0072] Furthermore, step A400 in the method provided in this application embodiment includes:

[0073] A410: Divide the cloud call network into the multiple local call relationship networks.

[0074] A420: Analyze the data transmission nodes in the cloud-based call network to segment a first cloud-based local call network with the data collection source as the call node and a second cloud-based local call network with the edge as the call node.

[0075] A430: The first cloud-based local network calls the data collection source for secondary similarity clustering according to preset call reference indicators. Based on the secondary clustering results, the edge-end partitioned and hierarchical storage plan is adapted and the path is optimized to obtain the edge-end storage optimization results.

[0076] A440: After partitioning and storing the intermediate data at the edge end of the second cloud local call network according to the receiving nodes in the cloud, the edge-cloud data processing path plan is generated together with the edge end storage optimization result.

[0077] In one embodiment, firstly, for the multiple established local call relationship networks, network topology filtering tools are used, such as the Cypher query statement of the Neo4j graph database and the subgraph extraction function of the Python NetworkX library, to segment the cloud call network. During operation, all nodes with direct or indirect call transmission relationships to the cloud center node are first filtered from the multiple local call relationship networks, including data collection source nodes, edge computing nodes, and associated call transmission records, ultimately forming the cloud call network. This network clearly reflects all call transmission logic involving the cloud, providing a focused scope for subsequent node analysis.

[0078] Next, within the aforementioned cloud-based invocation network, data transmission node role identification tools such as Wireshark's node type tagging function are used to classify and analyze the roles of all data transmission nodes within the cloud-based invocation network, thereby separating the first cloud-based local invocation network and the second cloud-based local invocation network. By analyzing the invocation initiation role of the nodes, nodes with data collection sources as invocation nodes (i.e., data directly transmitted from the data collection source to the cloud) and their associated records are separated to form the first cloud-based local invocation network; simultaneously, nodes with edge terminals as invocation nodes (i.e., data transmitted from the edge terminal to the cloud) and their associated records are separated to form the second cloud-based local invocation network.

[0079] Subsequently, within the first cloud-based local access network, the K-Means clustering algorithm is employed to perform secondary similarity clustering on multiple data collection sources within the network, based on the aforementioned preset access reference indicators. These preset access reference indicators include data type, data generation cycle, single data size, data lifecycle, and access frequency. The process is similar to step A320, ultimately yielding clear secondary clustering results and clarifying the classification characteristics of the data collection sources within the first cloud-based local access network.

[0080] Next, a secondary partitioned and hierarchical storage plan is performed based on the secondary clustering results. The degree of difference between the secondary planning result and the edge partitioned and hierarchical storage plan is judged. If the difference is less than a preset threshold, no optimization is performed. If the difference is greater than or equal to the preset threshold, the results of the first and second clustering are combined to adjust and optimize the edge storage, generating the edge storage optimization result. The specific steps are explained in detail in A431-A432.

[0081] Furthermore, we conducted cloud receiving node identification and attribute analysis. Using node mapping tools, such as the node attribute query function of Neo4j graph database, we traversed the cloud nodes in the second cloud local call network to identify all cloud receiving nodes that need to receive intermediate data from the edge. At the same time, we extracted the core attributes of each receiving node, including the type of data processed by the node, the upper limit of the node's data receiving bandwidth, and the node's data storage capacity limit, thereby determining the basis for matching intermediate data with receiving nodes.

[0082] Next, the intermediate data at the edge is classified according to the receiving nodes in the cloud. The K-Means clustering algorithm is used to use the matching degree between data type, data processing requirements and receiving node attributes as clustering features. The various types of intermediate data generated at the edge, such as the analysis results of production line equipment operation status and warehouse inventory dynamic evaluation data, are divided into the corresponding receiving node categories in the cloud. This ensures that each type of intermediate data can be matched with a receiving node in the cloud that meets its processing requirements, avoiding the problem of low processing efficiency caused by data transmission to mismatched nodes.

[0083] Then, construct a partitioned storage plan for intermediate data, and use storage configuration tools to allocate dedicated storage partitions for each intermediate data category corresponding to each cloud receiving node: set the storage path of each partition, data access priority, set high-priority access permissions for frequently accessed intermediate data, and data retention rules, for example, intermediate data used for temporary analysis is retained for 7 days, and intermediate data used for long-term archiving is retained for 1 year, forming a structured intermediate data partitioned storage scheme.

[0084] Next, the edge storage optimization results are integrated with the intermediate data partition storage plan. A path integration algorithm is used to minimize data transmission latency and optimize bandwidth usage. The storage levels determined in the edge storage optimization results (such as SSD primary storage and SAS hard disk secondary storage) are associated with the partition storage paths of intermediate data. For example, high-frequency intermediate data stored in edge SSDs is directly mapped to the high-priority storage partition of the corresponding receiving node in the cloud, and mid-frequency intermediate data stored in edge SAS hard disks is mapped to the regular storage partition of the receiving node in the cloud. This ensures that the data transmission path from edge storage to cloud receiving node is coherent and efficient.

[0085] Finally, an edge-to-cloud data processing path plan is generated. Using a path generation tool, the integrated storage relationships, data transmission order (e.g., transmitting high-priority intermediate data first, then regular-priority data), and node interaction rules (e.g., the authentication process before the edge transmits data to the receiving node in the cloud) are transformed into an executable path planning document. This document clarifies the complete process of each type of intermediate data starting from the storage location at the edge, passing through the transmission channel, and reaching the corresponding storage partition of the receiving node in the cloud. At the same time, key nodes and data processing triggering conditions in the path are marked, thus completing the generation of the edge-to-cloud data processing path plan.

[0086] Furthermore, step A420 in the method provided in this application embodiment includes:

[0087] A421: The first cloud-based local call network is a data call network composed of a data collection source and the cloud, and the second cloud-based local call network is a data call network composed of an edge terminal and the cloud. The call data in the second cloud-based local call network refers to the intermediate data obtained by the edge terminal after processing the local data of the data collection source through an embedded data analysis model.

[0088] Optionally, the core components of the first cloud-based partial call network are the data collection source and the cloud, which form a direct data call relationship through a specific data transmission channel. The data collection source here encompasses various carriers of data generated during enterprise operations, such as user order management systems, product production data statistics terminals, and customer feedback record modules. The cloud specifically refers to the cloud center node in the cloud-edge collaborative architecture. In actual data processing scenarios, when the data generated by these data collection sources needs to be analyzed globally or stored long-term, a direct call connection is established with the cloud center node. For example, the tens of thousands of order data generated daily by the user order management system will be directly transmitted to the order data analysis model in the cloud at a preset frequency, forming a network structure centered on the data collection source actively initiating the call and the cloud receiving and processing the data—this is the first cloud-based partial call network.

[0089] The second cloud-based local access network is jointly constructed by the edge and the cloud. Its data access logic differs significantly from the first cloud-based local access network. The key difference lies in the fact that the accessed data in this network is not the raw local data directly generated by the data collection source, but rather intermediate data processed by the edge. As edge computing nodes in the cloud-edge collaborative architecture, the edge embeds data analysis models that match local data processing needs. Examples include equipment operation status analysis models embedded in production line edge nodes and inventory dynamic monitoring models embedded in warehouse edge nodes. When the data collection source generates local data, it is first transmitted to the corresponding edge. The edge then processes this local data in real time using its embedded data analysis models, generating intermediate data with specific analytical value. For example, equipment failure risk assessment data obtained after processing temperature data by the equipment operation status analysis model. This intermediate data is then transmitted to the cloud as access data, forming the second cloud-based local access network where the edge initiates the access and the cloud receives the intermediate data.

[0090] Furthermore, step A430 in the method provided in this application embodiment includes:

[0091] A431: Based on the results of the secondary clustering, perform secondary partitioning and hierarchical storage planning, and determine the degree of difference between the secondary planning results and the partitioning and hierarchical storage planning at the edge. If the degree of difference is less than the preset threshold, no optimization is performed.

[0092] A432: If the degree of difference is greater than or equal to the preset threshold, the path is adjusted and optimized by combining the results of the first and second clustering to generate the edge storage optimization result.

[0093] In one embodiment, firstly, based on the secondary clustering results of data collection sources in the first cloud-based local access network, secondary partitioning and hierarchical storage planning is performed using a partitioning and hierarchical storage planning tool such as the storage planning component of IBM Storage Scale. The classification characteristics of the data collection sources after secondary clustering are first clarified. For example, secondary clustering might divide data collection sources into high-frequency real-time interaction types, such as user operation log systems; medium-frequency summary and analysis types, such as hourly data statistics terminals for production lines; and low-frequency archiving types, such as monthly financial statement systems.

[0094] Next, based on the performance differences of edge storage media, such as the different read and write speeds of SSDs, SAS hard drives, and SATA hard drives, corresponding storage tiers are allocated to data collection sources of different cluster categories. High-frequency real-time interactive data corresponds to the fastest read and write speed SSD Tier 1 storage, mid-frequency summary and analysis data corresponds to the second fastest read and write speed SAS hard drive Tier 2 storage, and low-frequency archiving data corresponds to SATA hard drive Tier 3 storage. At the same time, the partition location and access priority of each type of data in the corresponding storage tier are planned to form a complete secondary partitioned hierarchical storage plan.

[0095] Next, the storage configuration comparison function of the Python DiffLib library in the storage planning difference analysis tool was used to determine the degree of difference between the secondary partitioned hierarchical storage plan and the existing partitioned hierarchical storage plan at the edge. The comparison was conducted from three core dimensions: storage tier allocation, data partition range, and access path settings. First, it was checked whether the storage tier corresponding to the same data collection source was consistent in both plans. Then, it was confirmed whether there were deviations in the starting address and capacity allocation of the data partitions. Finally, it was verified whether the access path from the collection source to the storage node was the same. The tool automatically counted the number of difference dimensions and the deviation values ​​of key parameters. The statistical results were compared with a preset threshold, i.e., no more than two difference dimensions and a key parameter deviation value of less than 15%. If the statistical difference was less than the preset threshold, it indicated that the existing edge partitioned hierarchical storage plan could already adapt to the secondary clustering requirements and no optimization was needed.

[0096] If the difference is greater than or equal to a preset threshold, path adjustment and optimization are carried out by combining the results of the first and second clustering. First, the core logic of storage planning corresponding to the local call requirements of the edge end in the first clustering result is traced back. For example, in the first clustering, to meet the local real-time processing requirements, high-frequency real-time data is allocated to SSD Tier 1 storage. Then, the differences in storage requirements for adapting to cloud joint calls in the second clustering result are compared. For example, in the second clustering, it is found that some high-frequency real-time data only needs to be processed locally for a short time and does not require frequent synchronization with the cloud. It can be adjusted to SAS hard disk Tier 2 storage to free up SSD resources. Based on the core requirements of the two types of clustering results, the data transmission path and storage configuration are optimized through storage path adjustment tools. That is, the storage level of non-essential high-frequency data is adjusted, the capacity allocation of data partitions is corrected to match the data volume requirements of cloud calls, and the data access path is optimized to reduce the latency of data interaction between the edge end and the cloud. Finally, edge end storage optimization results that take into account both local call and cloud joint call requirements are generated.

[0097] By constructing a secondary partitioned and hierarchical storage plan based on the results of secondary clustering, using difference analysis tools to determine the relationship between the planning differences and preset thresholds, and combining the results of primary and secondary clustering to adjust and optimize the path, the edge partitioned and hierarchical storage plan is adapted to both local and cloud joint call requirements, ensuring a more reasonable allocation of edge storage resources and providing an optimized storage foundation for the subsequent generation of edge-cloud data processing path planning.

[0098] In summary, the cloud-edge collaborative enterprise data intelligent processing method provided in this application has the following technical effects:

[0099] This application collects data from multiple data collection sources of a target enterprise, analyzes the local data call and transmission scenarios of these multiple data collection sources to establish multiple local call relationship networks, and then analyzes local call demand based on these multiple local call relationship networks to generate edge data processing paths. Finally, it combines these edge data processing paths with cloud-based joint call demand analysis to generate edge-cloud data processing path planning. This adapts to the differentiated processing needs of enterprises with multiple data sources, solving the problems of chaotic data processing paths and resource waste in pure cloud or pure edge processing modes. It makes enterprise data intelligent processing more efficient and data value transformation more accurate and reliable, achieving the technical effect of adapting to the differentiated processing needs of enterprises with multiple data sources and realizing efficient data processing and data value transformation.

[0100] Example 2, as Figure 2 As shown, based on the same inventive concept as in Embodiment 1 above, this application provides an enterprise data intelligence processing system based on cloud-edge collaboration, the system comprising:

[0101] Data collection source acquisition module 1 is used to collect multiple data collection sources from the target enterprise.

[0102] Local call relationship network construction module 2 is used to analyze the local data call and transmission scenarios of the multiple data collection sources and establish multiple local call relationship networks.

[0103] Edge data processing path acquisition module 3 is used to perform local call demand analysis based on the multiple local call relationship networks, execute data processing path planning for edge data collection sources, and generate edge data processing paths.

[0104] Edge-to-cloud data processing path planning and acquisition module 4 is used to perform cloud joint call requirement analysis based on the multiple local call relationship networks, and to perform edge-to-cloud data processing path planning based on the edge data processing path to obtain the edge-to-cloud data processing path plan.

[0105] Furthermore, the local call relationship network construction module 2 is used to perform the following steps:

[0106] Collect multiple historical data flow records from the multiple data collection sources; collect cloud-edge collaborative architecture covering the multiple data collection sources; based on the multiple historical data flow records, analyze the call and transmission relationships of local data generated by the multiple data collection sources in the cloud-edge collaborative architecture, and establish the multiple local call relationship network.

[0107] Furthermore, the local call relationship network construction module 2 is used to perform the following steps:

[0108] The cloud-edge collaborative architecture includes several edge computing nodes and cloud center nodes, with each edge computing node and cloud center node embedding a corresponding data analysis model. The model inputs local data generated by multiple data collection sources.

[0109] Furthermore, the local call relationship network construction module 2 is used to perform the following steps:

[0110] Extract the first historical data flow record of the first data collection source; analyze the flow nodes of the first local data generated by the first data collection source in the cloud-edge collaborative architecture based on the first historical data flow record, and build a first local call relationship network; add the first local call relationship network into the multiple local call relationship networks.

[0111] Furthermore, the edge data processing path acquisition module 3 is used to perform the following steps:

[0112] Based on the multiple local call relationship networks, any local call network is segmented for any edge end; in any local call network, each data collection source is clustered once according to a preset call reference index, and a partitioned and hierarchical storage plan is constructed based on the clustering result; the edge end data processing path is constructed based on the partitioned and hierarchical storage plan.

[0113] Furthermore, the edge data processing path acquisition module 3 is used to perform the following steps:

[0114] The preset reference indicators include at least the data type, data generation cycle, single data size, data lifecycle, and call frequency.

[0115] Furthermore, the edge-cloud data processing path planning and acquisition module 4 is used to perform the following steps:

[0116] A cloud-based call network is segmented from the multiple local call relationship networks. Data transmission nodes are analyzed within the cloud-based call network to segment a first cloud-based local call network with data collection sources as call nodes and a second cloud-based local call network with edge nodes as call nodes. In the first cloud-based local call network, data collection sources are subjected to secondary similarity clustering based on preset call reference indicators. Based on the secondary clustering results, the partitioned and hierarchical storage plan for the edge nodes is adapted and path optimized to obtain edge storage optimization results. After partitioning and storing the intermediate data at the edge nodes in the second cloud-based local call network according to the receiving nodes in the cloud, the edge-cloud data processing path plan is generated together with the edge storage optimization results.

[0117] Furthermore, the edge-cloud data processing path planning and acquisition module 4 is used to perform the following steps:

[0118] The first cloud-based local access network is a data access network consisting of a data collection source and the cloud. The second cloud-based local access network is a data access network consisting of an edge device and the cloud. The access data in the second cloud-based local access network refers to the intermediate data obtained by the edge device after processing the local data of the data collection source through an embedded data analysis model.

[0119] Furthermore, the edge-cloud data processing path planning and acquisition module 4 is used to perform the following steps:

[0120] Based on the secondary clustering results, a secondary partitioned and hierarchical storage plan is performed. The degree of difference between the secondary planning result and the partitioned and hierarchical storage plan at the edge is determined. If the degree of difference is less than a preset threshold, no optimization is performed. If the degree of difference is greater than or equal to the preset threshold, the path is adjusted and optimized by combining the primary clustering result and the secondary clustering result to generate the edge storage optimization result.

[0121] The cloud-edge collaborative enterprise data intelligent processing system provided in this embodiment of the invention can execute the cloud-edge collaborative enterprise data intelligent processing method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0122] Although this application makes various references to certain modules in the system according to the embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of this invention.

[0123] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application. In some cases, the actions or steps described in this application can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A cloud-edge collaboration-based enterprise data intelligent processing method, characterized in that, The method comprises the following steps: Collecting a plurality of data collection sources of a target enterprise; Performing local data call transmission scenario analysis on the plurality of data collection sources, and establishing a plurality of local call relationship networks; According to the plurality of local call relationship networks, performing local call demand analysis, executing data processing path planning of the data collection source at the edge, and generating an edge data processing path; According to the plurality of local call relationship networks, performing cloud joint call demand analysis, and executing data processing path planning from the edge to the cloud based on the edge data processing path to obtain edge-cloud data processing path planning; According to the plurality of local call relationship networks, performing local call demand analysis, executing data processing path planning of the data collection source at the edge, and generating an edge data processing path, comprising: According to the plurality of local call relationship networks, any local call network of any edge is segmented; In the any local call network, each data collection source is subjected to one similarity clustering according to a preset call reference index, and a partition hierarchical storage planning is constructed according to the one clustering result; The edge data processing path is constructed based on the partition hierarchical storage planning; The preset call reference index at least includes data type, data generation period, single data size, data life cycle and call frequency; According to the plurality of local call relationship networks, performing cloud joint call demand analysis, and executing data processing path planning from the edge to the cloud based on the edge data processing path to obtain edge-cloud data processing path planning, comprising: The cloud call network is segmented in the plurality of local call relationship networks; The data transmission node is analyzed in the cloud call network, and the first cloud local call network with the data collection source as the call node and the second cloud local call network with the edge as the call node are segmented; In the first cloud local call network, the data collection source is subjected to two similarity clusterings according to the preset call reference index, and the partition hierarchical storage planning of the edge is subjected to adaptive analysis and path optimization according to the two clustering results to obtain an edge storage optimization result; The intermediate data of the edge in the second cloud local call network is subjected to partition storage planning according to the receiving node in the cloud, and the edge-cloud data processing path planning is generated together with the edge storage optimization result; The first cloud local call network is a data call network composed of the data collection source and the cloud, and the second cloud local call network is a data call network composed of the edge and the cloud, wherein the call data in the second cloud local call network is the intermediate data obtained by the edge through the embedded data analysis model to process the local data of the data collection source; According to the two clustering results, the two partition hierarchical storage planning is performed, the difference degree between the two planning results and the partition hierarchical storage planning of the edge is judged, and if the difference degree is less than a preset threshold, no optimization is performed. ​ If the difference is greater than or equal to a preset threshold, the path adjustment optimization is performed in combination with the first clustering result and the second clustering result, and the edge-end storage optimization result is generated. 2.The cloud edge collaboration based enterprise data intelligence processing method of claim 1, wherein, The local data calling transmission scenario analysis is performed on the plurality of data collection sources, and a plurality of local calling relationship networks are established, including: Collecting a plurality of historical data flow records of the plurality of data collection sources; Collecting a cloud-edge collaborative architecture covering the plurality of data collection sources; Based on the plurality of historical data flow records, the calling transmission relationship of the local data generated by the plurality of data collection sources in the nodes of the cloud-edge collaborative architecture is analyzed, and the plurality of local calling relationship networks are established. 3.The cloud edge collaboration based enterprise data intelligence processing method of claim 2, wherein, The cloud-edge collaborative architecture includes a plurality of edge computing nodes and a cloud center node, and the plurality of edge computing nodes and the cloud center node embed corresponding data analysis models, and input the local data generated by the plurality of data collection sources into the corresponding data analysis models by calling. 4.The cloud edge collaboration based enterprise data intelligence processing method of claim 2, wherein, Based on the plurality of historical data flow records, the calling transmission relationship of the local data generated by the plurality of data collection sources in the nodes of the cloud-edge collaborative architecture is analyzed, and the plurality of local calling relationship networks are established, including: Extracting a first historical data flow record of a first data collection source; Based on the first historical data flow record, the flow nodes of the first local data generated by the first data collection source in the cloud-edge collaborative architecture are analyzed, and a first local calling relationship network is built; The first local calling relationship network is added to the plurality of local calling relationship networks.

5. The cloud-edge collaboration based enterprise data intelligent processing system, characterized in that, The system for implementing the cloud-edge collaborative enterprise data intelligent processing method of any one of claims 1-4, the system comprising: A data collection source acquisition module for collecting a plurality of data collection sources of a target enterprise; A local calling relationship network construction module for performing local data calling transmission scenario analysis on the plurality of data collection sources, and establishing a plurality of local calling relationship networks; An edge-end data processing path acquisition module for performing local calling demand analysis according to the plurality of local calling relationship networks, executing data processing path planning of the data collection sources at the edge end, and generating an edge-end data processing path; An edge-cloud data processing path planning acquisition module for performing cloud joint calling demand analysis according to the plurality of local calling relationship networks, executing edge-end to cloud-end data processing path planning based on the edge-end data processing path, and obtaining edge-cloud data processing path planning.

Images