Multi-source data collector cooperative processing method based on industrial big data

Abstract

The invention provides a multi-source data collector cooperative processing method based on industrial big data, and relates to the technical field of industrial big data processing, and the method comprises the following steps: S1, constructing an edge-cloud cooperative architecture, deploying a multi-source data collector and an edge cooperative node at an edge layer, the cloud layer deploys a global scheduling center, an industrial knowledge graph database and a model training unit; s2, the edge cooperation node collects the dynamic attributes of each collector to generate a capability matrix, and the cloud layer disassembles the industrial business demand into an atomic task queue with labeled attributes; s3, based on a multi-objective optimization model and an improved genetic algorithm, dynamic allocation of collection tasks and seamless migration under abnormal conditions are achieved; according to the method, the key path task can be preferentially allocated to the high-performance collector, and the special bandwidth is reserved, so that the key task collection delay is reduced, the system resource utilization efficiency and the task response real-time performance are improved, and the contradiction between the key data collection delay and the occupation of redundant resources by non-key data is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial big data processing, and in particular to a multi-source data collector cooperative processing method based on industrial big data. BACKGROUND

[0002] Industrial big data is a collection of massive data generated in the whole life cycle of products and services in the industrial field, covering key links such as research and development, design, production and manufacturing, operation and management, and operation and maintenance, and extending to various types of information collected in the industrial internet platform. It takes product whole life cycle data as the core, expands the data boundary on the basis of traditional industrial data, and the data sources mainly include business data related to enterprise production and operation, equipment and internet of things data, and external environment data. In terms of technical system, industrial big data covers the whole process processing chain from data planning, collection, storage to analysis and mining. In this scenario, the multi-source data collector needs to synchronously collect heterogeneous data such as temperature, pressure, equipment running parameters, and production scheduling information, which puts forward higher requirements for real-time performance, consistency and system adaptability.

[0003] However, the traditional industrial big data collection and processing method has several key defects: first, the cooperative scheduling mechanism is rigid, and a static task allocation strategy is mostly used, which does not dynamically adjust according to the real-time load of the collector, the data priority and the network state, causing delay in collecting key data, while non-key data occupies redundant resources; second, the data conflict processing efficiency is low, and for the write-write conflict and time sequence conflict problems commonly seen in industrial scenarios, the existing technology mostly relies on a single timestamp or version number mechanism, lacks the fusion of industrial business logic, and leads to low conflict correction rate, which seriously affects data availability; third, the edge and cloud cooperative ability is insufficient, and most solutions still use centralized processing architecture, the edge-side collector only has a simple data uploading function, and cannot realize the closed-loop feedback of edge preprocessing and cloud deep optimization, which aggravates the network transmission pressure and processing delay; fourth, the system adaptability is poor, and it is difficult to dynamically adapt to new collectors, and needs to rely on manual configuration of analysis rules, with high system expansion cost and long cycle. SUMMARY

[0004] The present application provides a multi-source data collector cooperative processing method based on industrial big data, which realizes dynamic optimization allocation of collection tasks, intelligent resolution of multi-type data conflicts, and efficient edge-cloud cooperation, and improves the overall efficiency of industrial data collection and processing.

[0005] The present application provides a multi-source data collector cooperative processing method based on industrial big data, which specifically includes the following steps: Step S1. Build an edge-cloud cooperative architecture, deploy multi-source data collectors and edge cooperative nodes in the edge layer, and deploy a global scheduling center, an industrial knowledge graph library and a model training unit in the cloud layer; Step S2. The edge coordination node collects the dynamic attributes of each collector to generate a capability matrix, and the cloud layer decomposes the industrial business demand into an atomic task queue with labeled attributes; Step S3. Based on the multi-objective optimization model and the improved genetic algorithm, the dynamic allocation of collection tasks and seamless migration in abnormal situations are realized. Step S4. The edge layer 1 processes multi-source data through an adaptive parsing engine, detects conflicts based on a triple mechanism, and resolves data conflicts based on a conflict decision tree. Step S5. The edge layer and the cloud layer realize data hierarchical transmission and model incremental iteration, and the global scheduling center completes the optimization configuration of cross-regional collector resources. Step S6. The edge node monitors the state of the collector, starts the cache and retransmission mechanism to ensure data continuity.

[0006] Further, the collector dynamic attributes in step S2 include computing power, communication bandwidth, energy consumption state, and supported protocol type, and the task attributes include data type, real-time requirement, and precision threshold.

[0007] Further, the multi-objective optimization model in step S3 aims to minimize task completion delay, minimize total collector energy consumption, and minimize task rejection rate. Key path tasks are preferentially allocated to high-performance collectors and dedicated bandwidth is reserved.

[0008] Further, the triple mechanism conflict detection in step S4 includes industrial knowledge graph relevance verification, timing consistency detection, and version number comparison. The priority of the conflict decision tree is business rule > precision > timing.

[0009] Further, in step S5, the edge layer classifies data into hot data and cold data. Hot data is uploaded in real time, and cold data is uploaded incrementally. The cloud layer optimizes the model through incremental learning and delivers it to the edge node.

[0010] Further, in step S1, the edge layer and the cloud layer establish an encrypted communication link through 5G or industrial Ethernet. The multi-source data collector includes a temperature sensor, a pressure collector, a device log collection unit, and an AI vision collector.

[0011] Further, in step S3, the improved genetic algorithm reserves a dedicated bandwidth channel for key path tasks. When the collector bandwidth drops by more than 50%, the edge coordination node triggers task migration within 100ms, and based on the pre-stored backup collector list, the task switching is realized.

[0012] Further, in step S4, the adaptive parsing engine uses numerical verification for sensor data and regular expression parsing for device logs. For new types of collectors, adaptive parsing rules are generated through transfer learning.

[0013] The application provides a multi-source data collector cooperative processing method based on industrial big data, and has the following beneficial effects: The application aims at the problems of low resource utilization rate and key data acquisition delay caused by the traditional static task allocation mechanism, dynamically schedules through building a multi-objective optimization model and combining an improved genetic algorithm, and realizes comprehensive optimization of the real-time load of the collector, the network state and the data priority, so that the key path task is preferentially allocated to a high-performance collector and a dedicated bandwidth is reserved, the key task acquisition delay is reduced, the system resource utilization rate is improved, the system resource utilization efficiency and the task response real-time performance are improved, and the contradiction between the key data acquisition delay and the non-key data occupation of redundant resources is effectively solved.

[0014] In addition, the application introduces a triple conflict detection mechanism based on an industrial knowledge graph, and intelligently resolves the conflict by combining a conflict decision tree with clear priorities "business rule>precision>timing", thereby improving the conflict correction rate, greatly enhancing the data consistency and usability, meeting the high requirements of industrial scenes on data accuracy, and significantly improving the accuracy and reliability of data conflict processing.

[0015] In addition, by constructing an edge-cloud collaborative architecture, the application performs data preprocessing and hierarchical transmission in the edge layer, uploads hot data in real time and uploads cold data incrementally, significantly reduces the network transmission pressure, continuously optimizes the scheduling model and conflict rules based on incremental learning in the cloud layer, and dynamically issues them to the edge node to form a closed-loop optimization mechanism, the system supports plug-and-play of new types of collectors, quickly generates analysis rules through transfer learning, shortens the adaptation time of the new collector from 24 hours to 1 hour, and significantly enhances the system scalability and adaptability.

[0016] In addition, through real-time monitoring of the state of the collector by the edge node, the local cache mechanism and the task seamless migration strategy, the system can quickly complete task switching when the collector is abnormal, and realize data retransmission after connection recovery, ensuring continuous data acquisition, which effectively deals with equipment failure and network fluctuation in the complex environment of the industrial field, ensures that data is not lost and acquisition is not interrupted. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings of the embodiments will be briefly introduced below.

[0018] The drawings in the following description only relate to some embodiments of the application, and are not a limitation of the application.

[0019] In the drawings: Figure 1 The overall method flowchart of the present application is shown; Figure 2The edge layer composition diagram of the application is shown; Figure 3 The cloud layer composition diagram of the application is shown; Figure 4 The system architecture deployment diagram of the application is shown; Figure 5 The dynamic task scheduling flowchart of the application is shown; Figure 6 The full life cycle management flowchart of the application is shown.

[0020] Reference signs: 1, edge layer; 2, multi-source data collector; 3, edge collaborative node; 4, cloud layer; 5, global scheduling center; 6, industrial knowledge graph library; 7, model training unit; 9, adaptive analysis engine; 10, triple mechanism conflict detection; 11, conflict decision tree. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme of the embodiments of the application will be described clearly and completely below with reference to the drawings of the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments. Based on the described embodiments of the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0022] Embodiment one: please refer to Figures 1 to 6 : The application proposes a multi-source data collector collaborative processing method based on industrial big data, including the following steps: step S1. Build an edge-cloud collaborative architecture, deploy multi-source data collectors 2 and edge collaborative nodes 3 in the edge layer 1, deploy global scheduling centers 5, industrial knowledge graph libraries 6 and model training units 7 in the cloud layer 4, through the division and cooperation of the edge layer 1 and the cloud layer 4, sink real-time, high-frequency data processing tasks to the edge collaborative nodes 3 close to the data source, and the cloud layer 4 focuses on the macro resource coordination of the global scheduling center 5, the maintenance of the industrial knowledge graph library 6 and the iteration of the complex algorithm of the model training unit 7; Step S2. The edge collaborative node 3 collects the dynamic attributes of each collector to generate a capability matrix, and the cloud layer 4 disassembles industrial business requirements into an atomic task queue with labeled attributes. By constructing a refined collector capability matrix, the system can quantitatively evaluate the real-time state of each multi-source data collector 2. At the same time, the macro business requirements are decomposed into an atomic task queue with clear attributes, providing reliable data input for precise and dynamic task matching and resource allocation in step S3, ensuring the feasibility and efficiency of task execution; Step S3. Based on the multi-objective optimization model and the improved genetic algorithm, the dynamic allocation of the collection task and the seamless migration under abnormal conditions are realized, which overcomes the rigid problem of static task allocation. Through the multi-objective optimization model considering task delay, system energy consumption and rejection rate, and combined with the improved genetic algorithm for solving, the task allocation strategy can be dynamically adjusted according to the real-time capability changes of the multi-source data collector 2 such as load and bandwidth. Step S4. The edge layer 1 processes multi-source data through the adaptive parsing engine 9, combines the triple conflict detection mechanism 10 and eliminates data conflicts based on the conflict decision tree 11. The adaptive parsing engine 9 adopts differentiated parsing strategies for heterogeneous data of different sources and protocols, improving the accuracy and efficiency of data processing. The triple conflict detection mechanism 10 integrates the business logic of the industrial knowledge graph, the timing consistency and the version control, which can comprehensively identify data anomalies in complex industrial scenarios. The conflict decision tree 11 intelligently eliminates conflicts according to the clear priority of "business rules > precision > timing", significantly improving the accuracy and rationality of conflict processing, and ensuring the consistency and reliability of data. Step S5. The edge layer 1 and the cloud layer 4 realize data hierarchical transmission and model incremental iteration, and the global scheduling center 3 completes the cross-regional collector resource optimization configuration. This step forms a closed-loop optimization of edge and cloud. The hierarchical data transmission optimizes the bandwidth utilization by transmitting hot data in real time and cold data incrementally. The model incremental iteration enables the cloud model training unit 7 to continuously optimize algorithm parameters based on edge feedback, and updates the parameters through the global scheduling center 5, realizing the self-evolution of system performance. The cross-regional resource configuration capability of the global scheduling center 5 further improves the global optimality of resource scheduling in large-scale industrial scenarios. Step S6. The edge node monitors the collector state and starts the cache and retransmission mechanism to ensure data continuity. The edge coordination node 3 continuously monitors the state of the multi-source data collector 2, and combines the local cache and the breakpoint resume mechanism to ensure that critical data is not lost in extreme situations such as network interruption or device failure, and can be completely retransmitted after connection recovery.

[0023] In the embodiment of the present disclosure, the collector dynamic attributes in step S2 include computing power, communication bandwidth, energy consumption state and supported protocol type, and the task attributes include data type, real-time requirement and precision threshold. High real-time tasks can be scheduled to high bandwidth collectors, and high precision requirement tasks can be allocated to high precision sensors, so as to realize the best matching of resources and demands at the system level and optimize the overall execution efficiency.

[0024] In this embodiment, the multi-objective optimization model in step S3 aims to minimize task completion delay, minimize total energy consumption of the collector, and minimize task rejection rate. Critical path tasks are preferentially allocated to high-performance collectors and dedicated bandwidth is reserved. This model takes into account system performance, economy, and reliability. By prioritizing critical path tasks and reserving resources, it ensures timely response to core business needs and effectively avoids the problem of critical data collection delay caused by resource competition.

[0025] In this embodiment, the triple conflict detection mechanism 10 in step S4 includes industrial knowledge graph correlation verification, temporal consistency detection, and version number comparison. The conflict decision tree priority is business rules > accuracy > temporal sequence. The combination of the triple mechanism 10 provides a multi-dimensional perspective on conflict verification. Industrial knowledge graph verification ensures that the data conforms to physical laws and business logic. Temporal sequence detection detects abnormal fluctuations. Version comparison handles concurrent writing. The clear priority decision tree 11 makes the conflict resolution process systematic. Especially in security-related scenarios, the priority performance of business rules ensures the security of system behavior.

[0026] In this embodiment of the disclosure, in step S5, edge layer 1 divides the data into hot data and cold data. Hot data is uploaded in real time, while cold data is uploaded incrementally. The cloud layer optimizes the model through incremental learning and distributes it to the edge nodes. The data classification strategy significantly reduces the ineffective use of network bandwidth, allowing valuable communication resources to be concentrated on key hot data. The cloud continuously optimizes the model based on new data through incremental learning and feeds it back to the edge, enabling the processing capability of edge layer 1 to adaptively improve with changes in the environment, forming a continuous optimization closed loop from data to knowledge.

[0027] In this embodiment, in step S1, the edge layer 1 and the cloud layer 4 establish an encrypted communication link via 5G or industrial Ethernet; the multi-source data acquisition device 2 includes a temperature sensor, a pressure acquisition device, an equipment log acquisition unit, and an AI vision acquisition device; in step S3, the improved genetic algorithm reserves a dedicated bandwidth channel for critical path tasks when allocating them; when the acquisition device bandwidth drops by more than 50%, the edge collaborative node triggers task migration within 100ms, and task switching is achieved based on a pre-stored list of backup acquisition devices; in step S4, the adaptive parsing engine 9 uses numerical verification to parse sensor data and regular expression parsing to parse equipment logs; for newly added acquisition device types, adaptive parsing rules are generated through transfer learning, enhancing the system's scalability and adaptability; the customized parsing method 9 for different data characteristics ensures high parsing accuracy; and for newly added acquisition device types, the application of transfer learning avoids cumbersome manual rule configuration, achieving "plug-and-play" rapid adaptation, significantly reducing the cost and time of system maintenance and expansion.

[0028] The working principle of this embodiment is as follows: It begins with the construction of the edge-cloud collaborative architecture S1. The edge layer 1 collects data through the multi-source data collector 2, and the edge collaborative node 3 performs preliminary processing. The global scheduling center 5 of the cloud layer 4 is responsible for macro-coordination, the industrial knowledge graph library 6 provides business rule support, and the model training unit 7 performs algorithm optimization. In the S2 stage, the edge collaborative node 3 establishes the collector capability matrix, and the cloud layer 4 parses business requirements to generate an atomic task queue. After entering S3, the system realizes dynamic task scheduling based on a multi-objective optimization model and an improved genetic algorithm. When the collector is abnormal, the task migration can be completed within 100ms. In the S4 stage, the adaptive parsing engine 9 processes heterogeneous data, uses a triple conflict detection mechanism 10 to identify conflicts, and resolves them according to the conflict decision tree 11. In S5, data hierarchical transmission and model iteration are realized, and the global scheduling center 5 optimizes resource allocation. Finally, in S6, the system ensures data continuity through continuous status monitoring and caching mechanisms, forming a complete closed loop from data collection, processing, optimization to feedback, ultimately achieving the effect of reducing collection task latency and improving conflict correction rate.

[0029] The following points should be noted in this article: 1. The accompanying drawings of the embodiments disclosed herein only relate to the structures involved in the embodiments disclosed herein; other structures can be referred to in general design.

[0030] 2. Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0031] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A collaborative processing method for multi-source data acquisition devices based on industrial big data, characterized in that, Includes the following steps: Step S1. Construct an edge-cloud collaborative architecture. The edge layer (1) deploys a multi-source data collector (2) and an edge collaborative node (3). The cloud layer (4) deploys a global scheduling center (5), an industrial knowledge graph library (6), and a model training unit (7). Step S2. Edge collaborative node (3) collects the dynamic attributes of each collector to generate a capability matrix, and cloud layer (4) decomposes the industrial business requirements into an atomic task queue of labeled attributes; Step S3. Based on a multi-objective optimization model and an improved genetic algorithm, achieve dynamic allocation of data collection tasks and seamless migration under abnormal conditions; Step S4. Edge layer 1 processes multi-source data through an adaptive parsing engine (9), combines a triple-mechanism conflict detection (10), and resolves data conflicts based on a conflict decision tree (11); Step S5. The edge layer (1) and the cloud layer (4) realize hierarchical data transmission and incremental model iteration, and the global scheduling center (3) completes the cross-regional collector resource optimization configuration; Step S6. Monitor the status of the edge node collector and start the caching and retransmission mechanism to ensure data continuity.

2. The multi-source data acquisition device collaborative processing method based on industrial big data according to claim 1, characterized in that, In step S2, the dynamic attributes of the data collector include computing power, communication bandwidth, energy consumption status, and supported protocol types. The task attributes include data type, real-time requirements, and accuracy threshold.

3. The multi-source data acquisition device collaborative processing method based on industrial big data according to claim 1, characterized in that, The multi-objective optimization model in step S3 aims to minimize task completion delay, minimize total energy consumption of the collector, and minimize task rejection rate. Critical path tasks are preferentially allocated to high-performance collectors and dedicated bandwidth is reserved.

4. The multi-source data acquisition device collaborative processing method based on industrial big data according to claim 1, characterized in that, The triple mechanism conflict detection (10) in step S4 includes industrial knowledge graph correlation verification, temporal consistency detection and version number comparison. The priority of the conflict decision tree is business rule > accuracy > temporal sequence.

5. The multi-source data acquisition device collaborative processing method based on industrial big data according to claim 1, characterized in that, In step S5, the edge layer (1) divides the data into hot data and cold data. Hot data is uploaded in real time, and cold data is uploaded incrementally. The cloud layer optimizes the model through incremental learning and distributes it to the edge nodes.

6. The multi-source data acquisition device collaborative processing method based on industrial big data according to claim 1, characterized in that, In step S1, the edge layer (1) and the cloud layer (4) establish an encrypted communication link through 5G or industrial Ethernet; the multi-source data collector (2) includes a temperature sensor, a pressure collector, an equipment log collection unit and an AI vision collector.

7. The multi-source data acquisition device collaborative processing method based on industrial big data according to claim 1, characterized in that, In step S3, the improved genetic algorithm reserves a dedicated bandwidth channel for critical path tasks. When the collector bandwidth drops by more than 50%, the edge collaborative node triggers task migration within 100ms, and task switching is achieved based on the pre-stored list of backup collectors.

8. The multi-source data acquisition device collaborative processing method based on industrial big data according to claim 1, characterized in that, In step S4, the adaptive parsing engine (9) uses numerical verification to parse the sensor data and regular expression to parse the device logs; for newly added data collectors, it generates adaptation parsing rules through transfer learning.

Images