Data acquisition method and system
By introducing a unified data processing process and format standards, the data upload and polling mechanism is optimized, and the flexibility and performance problems of data acquisition methods in the existing technology are solved, and efficient and accurate data acquisition and upload are achieved.
Patent Information
- Application Number
- CN202411029397.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-29
AI Technical Summary
Existing data acquisition methods lack flexibility and adaptability when processing multiple OPCUA server nodes, resulting in a large number of customized development and testing when changing data formats. The increase in URL string length leads to performance problems, and it is impossible to effectively deal with dynamic changes in data quantity. Causes data loss or upload failure.
Introduce unified data processing flow and format standards, optimize the encoding method and transmission protocol of data upload, optimize the polling mechanism through comprehensive scoring and dynamic adjustment strategies, establish an intelligent exception processing process, and use JSON format for data encapsulation and uploading to ensure that Unified encapsulation and upload of data.
It realizes unified packaging and uploading of different types and structure data, improves the efficiency and accuracy of data collection, reduces system load, and significantly improves the efficiency and data quality of problem solving.
Smart Images

Figure CN119046350A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data acquisition, and in particular to a data acquisition method and system. Background Art
[0002] Different equipment in the entire production workshop uses different production processes, and the process-related data generated by each is not the same. For example, the key data of the water injection equipment is the remaining water volume of the product, which is a floating point number, while the key data of the degassing equipment is the process change value of the vacuum value during the degassing process. Due to the different degassing times, the process change value is a set of floating point arrays with an uncertain number. Different formats of data from different equipment have the following problems during uploading:
[0003] First, the formats of different device interfaces vary greatly. Even for the same device, the interface suffix may change when uploading data of different types or structures. This variety of interface formats requires the system to be highly adaptable and flexible to cope with various complex situations. However, existing upload mechanisms often lack this flexibility, resulting in a lot of custom development and testing work when the data format changes, increasing the complexity and maintenance cost of the system.
[0004] Secondly, the existing data upload method usually encapsulates the data in the packet header and assembles the URL for transmission through key-value pairs. This method can barely cope with the situation of small data volume, but once the data volume increases, the length of the URL string will increase sharply, which is not only difficult to control, but may also cause performance problems. At the same time, this URL splicing method lacks readability and is not conducive to problem troubleshooting and debugging. More seriously, when the amount of data changes dynamically, the existing upload mechanism often cannot effectively cope with it, resulting in an increased risk of data loss or upload failure.
[0005] Existing data collection methods often use a simple polling mechanism when processing multiple OPC UA server nodes. This mechanism does not fully consider the performance differences of server nodes, network conditions, and the importance and real-time requirements of data. This results in low data collection efficiency and is easily affected by factors such as network fluctuations and server failures, resulting in data loss or delays. Summary of the invention
[0006] Based on the above problems, the present invention provides a data collection method and system, which realizes the unified packaging and uploading of data of different types and structures by introducing a unified data processing process and format standard; solves the problems of uncontrollable URL string length, lack of readability and inability to cope with dynamic changes in data quantity by optimizing the encoding method and transmission protocol of data upload; improves the efficiency and accuracy of data upload and collection by optimizing the polling mechanism, introducing comprehensive scoring and dynamic adjustment strategies, and establishing an intelligent exception handling process.
[0007] In a first aspect, the present invention provides a data collection method, the method comprising:
[0008] Acquire first data;
[0009] Classifying the first data based on data types to obtain second data, wherein the second data includes first process data, first result data, and first transit data;
[0010] defining a minimum data unit, and processing the first process data through the minimum data unit to generate second process data;
[0011] Establishing a common format, wherein the common format is used to process the first result data; generating second result data;
[0012] The first transit data, the second process data and the second result data are encapsulated by using a preset transit data format to obtain third data;
[0013] The third data is converted into JSON format and uploaded to the database.
[0014] Preferably, the minimum data unit includes a timestamp, a name, a value and a unit; and establishing a universal format includes:
[0015] Expand the minimum data unit, add data qualification formula and qualification flag field based on the minimum data unit, and establish a universal format.
[0016] Preferably, converting the third data into JSON format and uploading it to the database comprises:
[0017] Determine the URL address of an http server;
[0018] Pack the third data in JSON format and send it to the URL address in POST format;
[0019] After receiving the uploaded data, the http server stores the data in a database.
[0020] Preferably, the first data is device data, and the method comprises:
[0021] Acquire first data of a corresponding device through a Codesys controller, and classify the first data based on data type to obtain second data;
[0022] The second data is acquired by interacting with the OPC UA server through the acquisition unit; the acquisition unit is connected to the core server; the OPC UA server is connected and deployed on the Codesys controller.
[0023] Preferably, the acquiring unit interacts with the OPC UA server to acquire the second data; includes:
[0024] Get device information from the database;
[0025] Poll multiple OPC UA server nodes in the queue through the acquisition unit;
[0026] Determine whether the device is connected. If the device is connected, obtain the data node information from the database; if the device is not connected, alarm according to the device information and obtain the next device information;
[0027] Determine whether the data node is valid. If the data nodes corresponding to the server node are all valid, read the data of each data node corresponding to the server node; otherwise, return to obtain the next device information;
[0028] Read the data of the data node and determine whether the data of the data node is abnormal. If not, save the data of the data node to the database and poll the next server node. If any data node data corresponding to the server node is abnormal, transfer the server node to the abnormal thread for processing, remove it from the inquiry team, and poll the next server node;
[0029] If the device is not connected, any data node of the device is invalid, or the data of the data node is abnormal, the first data collection of the device is stopped, and the next data collection is performed after the interaction is completed.
[0030] Preferably, the method of polling multiple OPC UA server nodes in a queue by a collection unit comprises:
[0031] Determine the comprehensive score of the server node through the server node priority score, the success rate and response time of the interaction between the server node and the acquisition unit;
[0032] Monitor the current network status, and adjust the polling order and single query duration of the server node according to the current network status, the timestamp of the most recent successful interaction between the server node and the collection unit, and the server node score.
[0033] Preferably, if the data of any data node corresponding to the server node is abnormal, the server node is transferred to the abnormal thread for processing, including:
[0034] If multiple reads are performed within the first time period, the server node data is abnormal; or,
[0035] If the preset number of reads is exceeded and the server node data is abnormal, an alarm will be issued and the abnormality will be recorded;
[0036] Obtain the cause of the exception and the processing result through terminal feedback.
[0037] Preferably, the third data is generated according to the second data through a unified data upload interface, and the third data is converted into a JSON format and uploaded to a database. The method further includes:
[0038] Through historical data, a variety of network conditions, data types and data volumes are obtained;
[0039] Create test data of different data types, encapsulate the test data, and perform sending and receiving tests through a unified data upload interface;
[0040] When the sent data is inconsistent with the received data, machine learning is used to locate the cause and specific link of the problem.
[0041] The present invention provides a data acquisition system for implementing any of the methods described in the present invention, comprising:
[0042] A Codesys controller, the Codesys controller is used to obtain first data; classify the first data based on data type to obtain second data, wherein the second data includes first process data, first result data and first transit data;
[0043] An OPCUA server, wherein the OPCUA server is deployed in the Codesys controller and is used to read second data in the Codesys controller;
[0044] A communication device, wherein the communication device is used to read the second data of the OPC UA server and send it to the core server;
[0045] A collection unit, the collection unit is used to read the second data in the core server;
[0046] A unified data upload interface, the unified data upload interface is connected to the acquisition unit, the unified data upload interface is used to define a minimum data unit, the minimum data unit is used to process the first process data and generate the second process data; establish a common format, the common format is used to process the first result data; generate the second result data; use a preset transit data format to encapsulate the first transit data, the second process data and the second result data to obtain third data; convert the third data into JOSN format and upload it to the database.
[0047] Preferably, the communication equipment includes: a 5G gateway, a room antenna, an RRU, a BBU, a UPF, and a core server. The 5G gateway, the room antenna, the RRU, the BBU, the UPF, and the core server are connected in sequence. The 5G gateway is connected to the OPC UA server; the core server is connected to the acquisition unit.
[0048] Compared with the prior art, the beneficial effects of the present invention include at least: by introducing a unified data processing process and format standard, unified encapsulation and uploading of data of different types and structures are realized. At the same time, the method also solves the problems of uncontrollable URL string length, lack of readability and inability to cope with dynamic changes in the amount of data by optimizing the encoding method and transmission protocol of data upload; by introducing a comprehensive scoring mechanism of server nodes, and dynamically adjusting the polling order and single inquiry duration according to factors such as network conditions, interaction success rate and response time, resource allocation is effectively optimized, system load is reduced, and the efficiency and stability of data collection are improved; when the server node data is abnormal, the abnormal processing process can be automatically triggered, including alarm, abnormal recording and obtaining abnormal causes and processing results through terminal feedback. This intelligent abnormal processing method significantly improves the efficiency and timeliness of problem solving and reduces the cost of manual intervention. The data collection method adopts a modular design, and each module is relatively independent, which is convenient for expansion and customization according to actual needs. At the same time, through a unified data upload interface and test mechanism, the good adaptability of the system to different network conditions, data types and data volumes is ensured. Through strict data classification, processing and packaging processes, as well as timely processing and feedback of abnormal data, the quality and reliability of data are effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 is a schematic diagram of a data collection method according to an embodiment of the present invention;
[0050] Figure 2 It is a schematic diagram of a data acquisition system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0051] The exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; on the contrary, these embodiments are provided to make the present invention more comprehensive and complete and to fully convey the concept of the exemplary embodiments to those skilled in the art.
[0052] See attached Figure 1 :Some embodiments of the present invention provide a data collection method, the method comprising:
[0053] Step 1: Obtain first data.
[0054] Among them, the first data is the data of factory automation equipment, including but not limited to temperature data, weight data, time data, pressure data, current data, voltage data, flow data, position data, production batch information and serial number, etc.
[0055] Step 2: classify the first data based on data type to obtain second data, wherein the second data includes first process data, first result data and first transit data.
[0056] The first process data refers to data generated in real time during the production process and changing with time, such as the remaining water volume of the water injection equipment, the vacuum value change process of the degassing equipment, etc.
[0057] The first and second result data refer to the data generated after the production process is completed and used to evaluate the production results, such as product quality inspection results, qualified rate, etc. The first station data refers to a comprehensive data set that combines station information, process data and result data, and is used to describe the complete situation of a station or stage in the entire production process.
[0058] Step 3, define a minimum data unit, the minimum data unit is used to process the first process data, generate the second process data, establish a common format, the common format is used to process the first result data; generate the second result data, use a preset transit data format to encapsulate the first transit data, the second process data and the second result data, and obtain the third data.
[0059] Specifically, the minimum data unit includes a timestamp, a name, a value, and a unit. As shown in Table 1:
[0060] Table 1
[0061] Number Field Type Note 1 Second double seconds 2 Name String name 3 Data Double data 4 Uom String Unit Of Measure
[0062] Table 1 defines the basic composition of the minimum data unit, which is the most basic data structure used when processing process data. It contains the following fields:
[0063] Second (double): Seconds usually refer to timestamps. The time recorded by timestamps is the relative time of process data. For example, if a test process lasts for 60 seconds, the value here is 0-60, which is the relative time in seconds.
[0064] Name (String): The name of the data item, such as "temperature", "pressure", etc.
[0065] Data (Double): The actual measured or recorded value.
[0066] Uom (String): abbreviation of Unit of Measure, that is, the unit of data, such as ℃, Pa, etc.
[0067] The establishment of the general format includes: expanding the minimum data unit, adding a data qualification formula and a qualified flag field on the basis of the minimum data unit, and establishing the general format. As shown in Table 2:
[0068] Table 2
[0069] Number Field Type Note 1 Second double seconds 2 Name String name 3 Data Double data 4 Uom String Unit Of Measure 5 Standard String Qualified formula for data 6 Pass bool Conformity mark
[0070] Table 2 shows the data structure with additional fields added to the minimum data unit to form a common format. In addition to the original four fields, it also includes:
[0071] Standard (String): The qualifying formula for data, which is used to define whether the data meets the standard calculation rules.
[0072] Pass (bool): Qualified flag, Boolean value, indicating whether the data meets the qualified conditions defined by the Standard.
[0073] The station information, process data and result data are summarized and packaged using the station data format as shown in Table 3:
[0074] Table 3
[0075] Number Field Type Note 1 Step Int Current station operation type 2 SerialNumber String unique serial number 3 Model String Material Model 4 StationAddress String System registration station unique ID 5 Pass Bool Test results 6 StartTime DataTime Entry time 7 EndTime DataTime Departure time 8 SecondData List <secondvalue> < / secondvalue> Seconds data 9 ResultData List <resultvalue> < / resultvalue> Results Data
[0076] Table 3 describes the data format of the packaging process data, result data, and station information. This data structure is used to describe the complete situation of a station in the production process, including:
[0077] Step (Int): The operation type of the current workstation.
[0078] SerialNumber (String): The unique serial number of the product.
[0079] Model (String): Model of the material or product.
[0080] StationAddress(String): The unique ID of the station registered in the system.
[0081] Pass (Bool): Test result, indicating whether the production result of this workstation is qualified.
[0082] StartTime(DateTime): The time when the product enters the workstation.
[0083] EndTime(DateTime): The time when the product leaves the workstation.
[0084] SecondData(List <secondvalue>): List of second-level process data recorded at this workstation.
[0085] ResultData(List <resultvalue>): A list of production result data related to this workstation.
[0086] These three tables together constitute the data structure required in the entire process of data collection, processing and uploading to the database, ensuring the consistency and readability of the data.
[0087] Step 4: convert the third data into JOSN format and upload it to the database.
[0088] Considering that there are many http interfaces and they are difficult to maintain, converting the third data into JOSN format and uploading it to the database includes:
[0089] Determine the URL address of an http server;
[0090] Pack the third data in JSON format and send it to the URL address in POST format;
[0091] After receiving the uploaded data, the http server stores the data in a database.
[0092] Uploading data through a unified http interface reduces the number of interfaces, reduces maintenance costs, and avoids compatibility issues caused by different interface formats.
[0093] It should be noted that JSON (JavaScript Object Notation) is a lightweight data exchange format that is easy for people to read and write, and easy for machines to parse and generate. It is based on a subset of JavaScript. Although JSON is a subset of JavaScript, it is language independent, and many programming languages support data exchange in JSON format.
[0094] The syntax rules of JSON include: (1) Data is stored in key-value pairs, with the key and value separated by a colon. (2) Data is surrounded by curly braces {}, representing an object. (3) Data items are surrounded by arrays [], and arrays can contain multiple values. (4) Strings must use double quotes ". (5) Supported data types include strings, numbers, arrays, Boolean values (true / false), and null. (6) Arrays and objects can be nested.
[0095] For reference, first, you need to know the exact network location of the http server, which is usually a URL address, such as https: / / example.com / data / upload.
[0096] Convert the third data (i.e., the classified, processed, and encapsulated data) into JSON format. JSON (JavaScript Object Notation) is a lightweight data exchange format that is easy for humans to read and write, and is also easy for machines to parse and generate.
[0097] Use the http POST format to send the JSON data to the URL address determined earlier. This usually involves building an http request, setting the correct header (such as Content-Type should be set to application / json), and sending the JSON data as the request body. When the server receives the uploaded data, it parses the JSON data and stores it in the database.
[0098] The working principle of this embodiment is as follows: the automation equipment continuously generates various types of data, such as temperature, weight, time, etc. These raw data are called first data. By data type classification, these data are divided into first process data, first result data and first station data. In order to process process data, a minimum data unit is defined, which contains basic information such as timestamp, name, value and unit. This helps to unify the data format and facilitate subsequent processing. On the basis of the minimum data unit, the data qualification formula and qualified flag field are added to create a common format. This not only standardizes the data, but also introduces the concept of data quality control to ensure the validity and accuracy of the data. The station information, process data and result data are encapsulated according to the preset station data format. This encapsulation ensures the integrity of the data and facilitates tracking the status of each production link. The encapsulated data is converted into JSON format, which is a lightweight, easy-to-read and write data exchange format. Use the POST format to upload JSON data to the URL address of the specified server, and the server receives the data and stores it in the database. Compared with the traditional URL parameter transmission method.
[0099] In this embodiment, a unified data interface is used to upload the equipment data of the entire factory, which is convenient for development and maintenance, and provides unlimited expansion space for subsequent data stringing, and provides data premise for realizing smart engineering. The JSON format can better cope with the upload scenarios of large data volume and dynamic data changes, and improves the flexibility and scalability of the system.
[0100] In some embodiments, the first data is device data, and the method includes:
[0101] Acquire first data of a corresponding device through a Codesys controller, and classify the first data based on data type to obtain second data;
[0102] The second data is acquired by interacting with the OPC UA server through the acquisition unit; the acquisition unit is connected to the core server; the OPC UA server is connected and deployed on the Codesys controller.
[0103] In some embodiments, the acquiring unit interacts with the OPC UA server to acquire the second data; includes:
[0104] Get device information from the database; the device information in the database is the information of the registered device, the IP and port number of the registered device, use the OPCUA server node to read the accessible data nodes in the device, select the data nodes to be read, and save the device, server node and data node information in the database; the device refers to the hardware component, which can be a sensor, actuator, controller, computer or any other physical entity that can connect to the network and send or receive data;
[0105] Poll multiple OPC UA server nodes in the queue through the acquisition unit;
[0106] Determine whether the device is connected. If the device is connected, obtain the data node information from the database; if the device is not connected, alarm according to the device information and obtain the next device information;
[0107] Determine whether the data node is valid. If the data nodes corresponding to the server node are all valid, read the data of each data node corresponding to the server node; otherwise, return to obtain the next device information;
[0108] Read the data of the data node and determine whether the data of the data node is abnormal. If not, save the data of the data node to the database and poll the next server node. If any data node data corresponding to the server node is abnormal, transfer the server node to the abnormal thread for processing, remove it from the inquiry team, and poll the next server node;
[0109] If the device is not connected, any data node of the device is invalid, or the data of the data node is abnormal, the first data collection of the device is stopped, and the next data collection is performed after the interaction is completed.
[0110] The working principle of the above technical solution is: the acquisition unit first obtains all device information that needs to be monitored and collected data from the database; this device information usually includes the device's unique identifier, IP address, port number and relevant configuration information of the OPC UA server.
[0111] The acquisition unit creates a device queue based on the device information list; polls multiple OPC UA server nodes in these queues; polling is a mechanism that sequentially accesses each node to check its status or obtain data.
[0112] The acquisition unit queries each OPC UA server node. If the device is connected within the query time, it proceeds to the next step. If the device is not connected, it records an alarm based on the device information (for example, sends an alarm email or message to the monitoring platform), skips the current device, and continues to obtain the next device information.
[0113] Once the device connection is confirmed, the acquisition unit retrieves the data node information corresponding to the device from the database. A data node is an entity in the OPC UA server that is used to store and transmit specific data (such as sensor readings, device status, etc.). The acquisition unit checks whether these data nodes are all valid, that is, whether they exist and are accessible. If all data nodes are valid, continue to read the data; if any data node is invalid, return and try to obtain the information of the next device.
[0114] For valid data nodes, the acquisition unit will read the data of these data nodes. During the reading process, it will also check whether the data is abnormal, such as whether it exceeds the reasonable range, whether it is empty or contains illegal characters. If the data is normal, the data will be saved in the database for subsequent analysis and use; if the data is abnormal, the server node will be transferred to the exception handling thread for further analysis (for example, try to re-read, record error logs or trigger alarms), and the server node corresponding to the data node will be removed from the current polling queue, and the next server node will continue to be polled.
[0115] If the device is not connected, the data node is invalid, or the data is abnormal during the interaction, the collection unit will stop the data collection of the server node device. This means that it will pause the data collection from the server node, but will not completely stop the entire data collection process. Once these problems are resolved (for example, the device is reconnected, the data node returns to normal, or the abnormality is handled), the collection unit will resume data collection for the server node device.
[0116] The whole process is ongoing, and the acquisition unit will continuously poll the OPC UA server nodes in the device queue, check the device connection status, data node validity, and read and save data. In this way, the system can obtain the latest device data in real time or near real time, providing support for subsequent data analysis, monitoring and decision-making.
[0117] The effect of the above technical solution is that by creating a device queue and polling the OPC UA server node, data from multiple devices can be efficiently collected. This mechanism ensures the continuity and timeliness of data collection and avoids long-term data loss due to a single request failure. Pre-registering device information, server nodes, and data nodes in the database makes the data collection process more flexible. Administrators can easily add, delete, or modify device information without modifying the code or reconfiguring the system. In addition, by saving data node information, the system can automatically select the data to be read, reducing the possibility of human error. When the device is not connected, the data node is invalid, or the data is abnormal, the system will immediately take corresponding measures, such as sending an alarm, recording an error log, or triggering an exception handling thread. These measures ensure the timely discovery and resolution of problems and improve the stability and reliability of the system. By acquiring device data in real time or near real time, the technical solution provides strong support for data analysis, monitoring, and decision-making. Administrators can understand the operating status and performance indicators of the equipment in real time, discover potential problems in time, and take corresponding measures to avoid production interruptions or safety accidents.
[0118] Through modular design and clear interface definition, the system can easily integrate new devices or functions, while reducing maintenance costs and time. In addition, each part of the system is well encapsulated and abstracted, making it easier for developers to understand and modify the code. Through strict data node validity and data anomaly checks, the technical solution ensures the consistency and accuracy of the collected data. This helps reduce decision-making errors and losses caused by data errors.
[0119] In summary, this technical solution has demonstrated significant effects and advantages in data collection, data management, exception handling, data monitoring, scalability, and data accuracy.
[0120] In some embodiments, polling OPC UA server nodes in multiple device queues by a collection unit includes:
[0121] Determine the comprehensive score of the server node through the server node priority score, the success rate of interaction between the server node and the acquisition unit, and the response time;
[0122] Monitor the current network status, and adjust the polling order and single query duration of the server node according to the server node score, the current network status, and the timestamp of the most recent successful interaction between the server node and the collection unit;
[0123] In some embodiments, the comprehensive score of the server node is;
[0124]
[0125] Among them, Pj is the score of the j-th server node; Aj is the data score of the j-th server;
[0126] ACj is the success rate of interaction between the jth server node and the collection unit within the preset time period, ACmin is the minimum value of the success rate of interaction between the jth server node and the collection unit within the preset time period; ACmin is the maximum value of the success rate of interaction between the jth server node and the collection unit within the preset time period; Zij is the importance score of the i-th data node of the jth server node; Sij is the real-time score of the i-th data node of the jth server node; m is the total number of data nodes contained in the jth server node;
[0127] REj is the response time of the jth server within the preset time period; REmin is the minimum response time of the jth servers within the preset time period; REmax is the maximum response time of the jth servers within the preset time period; w1 and w2 are both the first weight coefficients; x1 and x2 are both the second weight coefficients;
[0128] Sort all server nodes according to their comprehensive scores Pj, and poll server nodes with high scores first; when two or more server nodes have the same comprehensive score Pj, use the timestamp Tjlast of the most recent successful interaction for secondary sorting;
[0129] The single query duration of the node is determined based on the server node score, the current network status, the timestamp of the most recent successful interaction between the server node and the collection unit, and the server node score. The single query duration is:
[0130]
[0131] Wherein, Tj is the single query duration of the jth server node, Ty is the preset single query duration, Pa is the average comprehensive score of the jth server nodes; Nj is the network status score of the jth server node; Na is the average of the network status scores of the jth server nodes in the preset time period; Tjnow is the current query start time point of the jth server node, Tjlast is the time point of the last successful interaction between the jth server node and the collection unit; TCa is the average of the time intervals between two adjacent interactions of multiple server nodes that normally interact with the collection unit in the preset time period; e is the first adjustment coefficient, 1 <e<4;
[0132] The collection unit queries the server node. If the single query time is exceeded and there is no feedback from the server node, the next server node will be polled.
[0133] The working principle of the above technical solution is as follows: each server node calculates a priority score based on the importance score (Zij) and real-time score (Sij) of its data node. These two scores are weighted and averaged by the second weight coefficient (x1, x2). The importance score reflects the importance of the data node to the system, while the real-time score measures the frequency and timeliness of data updates. The success rate of each server node interacting with the acquisition unit within a preset time period is recorded, and normalized by comparing with the minimum and maximum values of all nodes to reflect the stability of the node.
[0134] The priority score, interaction success rate and response time are weighted by the first weight coefficient (w1, w2, w3) and then summed to obtain the comprehensive score of each server node. This score takes into account the importance, stability and response speed of the node.
[0135] All server nodes are sorted according to the comprehensive score, and nodes with higher scores are polled first. If two or more nodes have the same score, they are sorted according to the timestamp of their most recent successful interaction to ensure that the most recently active nodes are polled more promptly.
[0136] The single query time is dynamically adjusted based on multiple factors such as the node's comprehensive score, network status score, current query start time, the last successful interaction time, and the average interaction interval. Nodes with high scores, good network conditions, and a long time since the last interaction will receive a longer query time to obtain as much data as possible.
[0137] The collection unit queries the server nodes according to the optimized polling order and single query time; if a server node has no feedback within a single query time, the collection unit will automatically switch to the next server node for query to ensure the continuity and efficiency of data collection; continuously monitor the network status and the performance of the server nodes, and dynamically adjust the polling order and single query time based on these real-time data to adapt to changing environmental conditions.
[0138] The effect of the above technical solution is: by comprehensively considering the priority score, interaction success rate and response time of the server node to determine the comprehensive score, and adjusting the polling order accordingly, it can ensure that high-priority, stable and responsive server nodes are processed more preferentially, thereby optimizing data collection efficiency and resource allocation.
[0139] Dynamically adjusting the single query time according to the comprehensive score of the server node can reduce unnecessary waiting time while ensuring the integrity of data collection, thereby improving the overall efficiency of data collection. For nodes with high scores, good network conditions, and frequent recent interactions, the single query time can be appropriately increased to obtain more data; otherwise, the query time can be reduced to quickly switch to other nodes.
[0140] By monitoring the network status and the interaction history of server nodes, the system can promptly detect and resolve potential problems such as network delays, node failures, etc., thereby enhancing the stability and reliability of the system. In addition, for nodes that have not successfully interacted for a long time, the system can automatically reduce their priority or increase the query frequency to ensure the timeliness and integrity of the data.
[0141] The polling mechanism takes into account a variety of factors, including server node scores, network conditions, interaction history, etc., allowing the system to flexibly respond to complex operating environments. Whether in a high-load, low-latency or high-latency, low-load environment, the system can dynamically adjust the polling order and query time to ensure the efficiency and stability of data collection.
[0142] By optimizing the data collection process, the system can obtain the required data more quickly and accurately, providing users with more timely and accurate information support. This helps to improve the user experience and enhance users' trust and satisfaction with the system.
[0143] In some embodiments, if the data of any data node corresponding to the server node is abnormal, the server node is transferred to the abnormal thread for processing, including:
[0144] If multiple reads are performed within the first time period, the server node data is abnormal; or,
[0145] If the preset number of reads is exceeded and the server node data is abnormal, an alarm will be issued and the abnormality will be recorded;
[0146] Obtain abnormal causes and processing results through terminal feedback;
[0147] In some embodiments, the first duration is:
[0148] Sk=(1+α×Dk)*Sy
[0149]
[0150] The preset reading times are:
[0151] Ck=Ceiling(1+α×Gk)*Cy
[0152]
[0153] Among them, Fk is the average data update frequency of the kth abnormal server node, Fa is the mean of the average data update frequency of all server node data; Ak is the data score of the kth abnormal server node; REk is the current response time of the kth abnormal server node; REa is the average response time of all server nodes; y1, y2, y3 are all the third weight coefficients; Sy is the reference reading number; Ceiling() is rounded up; Cy is the reference reading number; α is the second adjustment coefficient, ranging from (0,1);
[0154] Among them, if any of the conditions of "first duration" and "number of reads" is met, the exception handling process will be triggered.
[0155] The working principle of the above technical solution is: continuously monitor the data node status corresponding to each server node. When an abnormality is detected in the data of a server node, a specific abnormality detection process is started.
[0156] There are two main conditions to determine whether the server node needs to be transferred to the exception thread processing:
[0157] First duration condition: multiple data readings are performed within the first duration Sk. If data anomalies are confirmed in each reading during this period, this condition is met.
[0158] This condition is met when the number of data reads exceeds the preset number of reads Ck, and each read confirms that the data is abnormal.
[0159] It is calculated according to the formula Sk=(1+Dk)*Sy, where Dk is a dynamic adjustment factor, which is calculated by weighting the data update frequency Fk, data score Ak and the difference between the current response time REk and the global average value (Fa, Aa, REa) of the abnormal server node through the third weight coefficient (y1, y2, y3). Sy is a reference reading number, which is used for basic time length setting.
[0160] Preset reading times Ck: calculated by the formula Ck = Ceiling (1 + Gk) * Cy, where Gk is a ratio calculated based on the difference between the data update frequency Fk and the global average value Fa, and Cy is also a reference reading number for basic number settings. The Ceiling() function ensures that the result is rounded up.
[0161] During the anomaly detection process, when any of the conditions in "first duration" and "number of readings" is met, the anomaly handling process is triggered. This means that if the system reads data abnormally multiple times within Sk time, or if the system has confirmed that the data has been abnormal before reaching Ck readings, the anomaly handling process will be initiated.
[0162] Once the exception handling process is triggered, the system transfers the server node to the exception thread for processing.
[0163] Send alarm notifications and record exception details, including the time when the exception occurred, the exception type, the scope of impact, etc.
[0164] Wait for the abnormal cause and processing result to be obtained through terminal feedback. This usually involves the administrator or system maintenance personnel to check the abnormal details according to the alarm information and take corresponding measures to solve the problem.
[0165] After the problem is solved, continue to monitor the status of the server node and its data nodes to ensure that the problem does not recur. At the same time, based on the experience of this exception handling, adjust the monitoring strategy and exception handling process to improve the stability and reliability of the system.
[0166] Through the above method, it is possible to flexibly respond to data node abnormalities, trigger the exception handling process in a timely manner, and record detailed exception information for subsequent analysis and improvement.
[0167] The effect of the above technical solution is: by timely detecting and processing server node data anomalies, the problem node can be quickly isolated to prevent abnormal data from further affecting the overall performance of the system, thereby improving the stability and reliability of the system. Setting the first duration and the preset number of reads as trigger conditions can ensure a rapid response when an anomaly occurs, reduce the time interval from the occurrence of the anomaly to the resolution of the problem, and improve the efficiency of fault recovery. At the same time, unnecessary abnormal triggering is reduced. By dynamically adjusting the calculation of factors Dk and Gk, the mechanism can flexibly adjust detection parameters according to the characteristics of different server nodes (such as data update frequency, data scoring, response time, etc.), making exception handling more accurate and efficient. Ensure that the exception handling process is triggered only when it is really needed, thereby avoiding waste of resources due to frequent false alarms.
[0168] In some embodiments, the method further includes: generating third data according to the second data through a unified data upload interface, converting the third data into a JSON format and uploading the third data to a database;
[0169] Through historical data, a variety of network conditions, data types and data volumes are obtained;
[0170] Create test data of different data types, encapsulate the test data, and perform sending and receiving tests through a unified data upload interface;
[0171] When the sent data is inconsistent with the received data, machine learning is used to locate the cause and specific link of the problem.
[0172] The working principle of the above technical solution is as follows: First, a unified data upload interface is defined, which is the only channel for uploading various data types and data volumes to the server; the purpose is to ensure data consistency and manageability. In order to simulate various situations that may be encountered in the real environment, the system will extract information on various network conditions (such as network delay, packet loss rate, etc.), data types (such as text, pictures, videos, etc.) and data volumes (small data, big data, etc.) from historical data.
[0173] Based on historical data, the system creates multiple different types of test data to cover various possible input situations; appropriately encapsulates the test data (such as adding necessary metadata, checksums, etc.), and then sends it through a unified data upload interface. At the same time, the system receives the data returned from the interface and verifies it to ensure that the data is not lost, damaged, or tampered with during transmission. If the sent data is found to be inconsistent with the received data during the test process, the system will trigger the problem diagnosis process.
[0174] Using machine learning technology, the system can automatically analyze inconsistent data, combine historical data with the current test environment, and try to locate the cause and specific links of the problem. This involves network problems (such as network delays and packet loss), data transmission problems (such as protocol errors and encoding errors), server processing problems (such as processing logic errors and insufficient resources), and many other aspects.
[0175] Once the problem is located, a detailed problem report will be generated, including the description of the problem, possible causes, scope of impact, and recommended solutions. Operations and maintenance personnel can conduct further investigation and repair based on this information.
[0176] In summary, this embodiment ensures the reliability, accuracy and efficiency of data upload by unifying the data upload interface, generating and converting data formats, uploading to the database, testing and verification, and machine learning problem diagnosis. At the same time, through the automated testing and problem diagnosis process, the cost of manual intervention is reduced, and the maintainability and scalability of the system are improved.
[0177] The embodiment of the present application provides a data acquisition system for implementing any method described in the embodiment of the present application, including: field equipment, 5G communication and a hyper-converged server (core server);
[0178] The field device includes an automation device, a Codesys controller and an OPC UA server; the Codesys controller is used to obtain first data of the automation device; the first data is classified based on the data type to obtain second data, wherein the second data includes first process data, first result data and first transit data; the OPC UA server is deployed in the Codesys controller and is used to read the second data in the Codesys controller;
[0179] 5G communication is used to read the second data of the OPC UA server and send it to the core server; the 5G communication includes: 5G gateway, room antenna, RRU, BBU, UPF, core server, the 5G gateway, room antenna, RRU, BBU, UPF, core server are connected in sequence, and the 5G gateway is connected to the OPC UA server. The workshop uses room antennas in conjunction with RRU to complete signal coverage. The 5G gateway communication can solve the problem that there are many network cables on the production site, which is not conducive to the movement of production equipment. RRU, namely Remote Radio Unit, is used to transmit and receive signals. RRU and antenna are connected by feeders, and they together form a cell. BBU, namely Building Baseband Unit, the bottom of the base station is the machine room, which is the cabinet in the machine room, equivalent to the "brain" of the cell. UPF, namely User Plane Function, is an important part of the 3GPP 5G core network system architecture, and is mainly responsible for the routing and forwarding related functions of the 5G core network user plane data packets.
[0180] The hyper-converged server includes a collection unit, a unified data upload interface and a database, and the collection unit, the unified data upload interface and the database are all deployed on the hyper-converged server.
[0181] The hyper-converged server (core server) is connected to the collection unit, and the collection unit is used to read the second data in the core server; a unified data upload interface is connected to the collection unit, and the unified data upload interface is used to define a minimum data unit, and the minimum data unit is used to process the first process data and generate the second process data; establish a common format, and the common format is used to process the first result data; generate the second result data; use a preset transit data format to encapsulate the first transit data, the second process data and the second result data to obtain third data; convert the third data into JOSN format and upload it to the database.
[0182] In this embodiment, the Codesys controller is responsible for obtaining raw data (first data) from various devices or sensors. These data come from sensors, machine equipment or other data sources on the production line. The first data obtained will be classified and processed based on the data type to generate second data. These classified data include first process data (reflecting the real-time status or parameters in the production process), first result data (data reflecting the production results or product quality) and first transit data (reflecting the flow of products between different production stages or sites). The OPCUA server is deployed in the Codesys controller and is responsible for reading the classified second data. OPCUA (Open Platform Communications Unified Architecture) is a modern, standard-based industrial automation communication protocol that can provide reliable and secure data communication. The 5G gateway, indoor antenna, RRU, BBU, UPF and core server are connected in sequence to form a communication link. The 5G gateway is connected to the OPC UA server to transmit the read second data to the core server. The acquisition unit reads the second data from the core server, and the unified data upload interface is deployed in the acquisition unit, which is responsible for defining the minimum data unit and processing the first process data to generate the second process data; at the same time, a universal format is established to process the first result data to generate the second result data; finally, the first transit data, the second process data and the second result data are encapsulated using the preset transit data format to obtain the third data. This process ensures the normalization and standardization of the data, which facilitates subsequent data analysis and application. The encapsulated third data is converted into JSON format and uploaded to the database. JSON (JavaScript Object Notation) is a lightweight data exchange format that is easy for people to read and write, and is also easy for machines to parse and generate. Uploading data in JSON format can improve the compatibility and readability of data.
[0183] In this embodiment, a unified data upload interface is used to upload the data of all equipment in the factory, which is convenient for development and maintenance and provides unlimited expansion space. The JSON format improves the flexibility and scalability of the system, especially for uploading scenarios with large amounts of data and dynamic changes. It provides data prerequisites for the realization of smart engineering and supports subsequent data analysis and decision-making. The data qualification formula and qualified mark are introduced to ensure the validity and accuracy of the data. Encapsulation of data ensures a complete record of the status of the production process, which is conducive to product traceability.
[0184] An embodiment of the present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any method described in the embodiments of the present application or the functions of any system described in the embodiments of the present application are implemented.
[0185] The embodiments of the present application also provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed, it implements the steps of the method in the embodiments of the present application or the functions of the system as described in any of the embodiments of the present application. The specific implementation method is consistent with the implementation method and the technical effect achieved recorded in the above-mentioned method embodiments, and some contents will not be repeated here.
[0186] In the present application, a readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. A program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0187] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or for use in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. The program code for performing the operation of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also conventional procedural programming languages such as C language or similar programming languages. The program code may be executed entirely on a user computing device, partially on an associated device, as an independent software package, partially on a user computing device, partially on a remote computing device, or entirely on a remote computing device or server. Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).
[0188] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, substitute and modify the above embodiments within the scope of the invention without departing from the principles and purpose of the present invention. All such changes should fall within the scope of protection of the claims of the present invention.< / resultvalue> < / secondvalue>
Claims
1. A data collection method, characterized in that: The method comprises: Acquire first data; Classifying the first data based on data types to obtain second data, wherein the second data includes first process data, first result data, and first transit data; defining a minimum data unit, and processing the first process data through the minimum data unit to generate second process data; Establishing a common format, wherein the common format is used to process the first result data; generating second result data; The first transit data, the second process data and the second result data are encapsulated by using a preset transit data format to obtain third data; The third data is converted into JSON format and uploaded to the database.
2. The method according to claim 1, characterized in that The minimum data unit includes a timestamp, a name, a value and a unit; and establishing a universal format includes: Expand the minimum data unit, add data qualification formula and qualification flag field based on the minimum data unit, and establish a common format.
3. The method according to claim 1, characterized in that: The converting the third data into JSON format and uploading it to the database includes: Determine the URL address of an http server; Pack the third data in JSON format and send it to the URL address in POST format; After receiving the uploaded data, the http server stores the data in a database.
4. The method according to claim 1, characterized in that: The first data is device data, and the method includes: Acquire first data of a corresponding device through a Codesys controller, and classify the first data based on data type to obtain second data; The second data is acquired by interacting with the OPC UA server through the acquisition unit; the acquisition unit is connected to the core server; the OPC UA server is connected and deployed on the Codesys controller.
5. The method according to claim 4, characterized in that The acquiring unit interacts with the OPC UA server to obtain the second data; including: Get device information from the database; Poll multiple OPCUA server nodes in the queue through the acquisition unit; Determine whether the device is connected. If the device is connected, obtain the data node information from the database; if the device is not connected, alarm according to the device information and obtain the next device information; Determine whether the data node is valid. If the data nodes corresponding to the server node are all valid, read the data of each data node corresponding to the server node; otherwise, return to obtain the next device information; Read the data of the data node and determine whether the data of the data node is abnormal. If not, save the data of the data node to the database and poll the next server node. If any data node data corresponding to the server node is abnormal, transfer the server node to the abnormal thread for processing, remove it from the inquiry team, and poll the next server node; If the device is not connected, any data node of the device is invalid, or the data of the data node is abnormal, the first data collection of the device is stopped, and the next data collection is performed after the interaction is completed.
6. The method according to claim 5, characterized in that The method of polling multiple OPC UA server nodes in the queue by the acquisition unit comprises: Determine the comprehensive score of the server node through the server node priority score, the success rate and response time of the interaction between the server node and the acquisition unit; Monitor the current network status, and adjust the polling order and single query duration of the server node according to the current network status, the timestamp of the most recent successful interaction between the server node and the collection unit, and the server node score.
7. The method according to claim 5, characterized in that If the data of any data node corresponding to the server node is abnormal, the server node is transferred to the abnormal thread for processing, including: If multiple reads are performed within the first time period, the server node data is abnormal; or, If the preset number of reads is exceeded and the server node data is abnormal, an alarm will be issued and the abnormality will be recorded; Obtain the cause of the exception and the processing result through terminal feedback.
8. The method according to claim 1, characterized in that Generate third data according to the second data through a unified data upload interface, and convert the third data into JSON format and upload it to the database. The method also includes: Through historical data, a variety of network conditions, data types and data volumes are obtained; Create test data of different data types, encapsulate the test data, and perform sending and receiving tests through a unified data upload interface; When the sent data is inconsistent with the received data, machine learning is used to locate the cause and specific link of the problem.
9. A data acquisition system, used to implement the method according to any one of claims 1 to 8, characterized in that: The system comprises: A Codesys controller, the Codesys controller is used to obtain first data; classify the first data based on data type to obtain second data, wherein the second data includes first process data, first result data and first transit data; An OPCUA server, wherein the OPCUA server is deployed in the Codesys controller and is used to read second data in the Codesys controller; A communication device, wherein the communication device is used to read the second data of the OPC UA server and send it to the core server; A collection unit, the collection unit is used to read the second data in the core server; A unified data upload interface, the unified data upload interface is connected to the acquisition unit, the unified data upload interface is used to define a minimum data unit, the minimum data unit is used to process the first process data and generate the second process data; establish a common format, the common format is used to process the first result data; generate the second result data; use a preset transit data format to encapsulate the first transit data, the second process data and the second result data to obtain third data; convert the third data into JOSN format and upload it to the database.
10. A data acquisition system according to claim 9, characterized in that: The communication equipment includes: a 5G gateway, a room antenna, an RRU, a BBU, a UPF, and a core server. The 5G gateway, the room antenna, the RRU, the BBU, the UPF, and the core server are connected in sequence. The 5G gateway is connected to the OPC UA server; the core server is connected to the acquisition unit.
