Data processing system and method

Abstract

The invention relates to a data processing system and method, and belongs to the technical field of data processing.The data processing system comprises a data acquisition end and a data receiving end, the data acquisition end is used for acquiring operation data of to-be-detected equipment, and the data receiving end is used for receiving the operation data of the to-be-detected equipment; the data receiving end is used for receiving operation data, packaging the operation data into a data packet in a specified protocol format according to a preset coding rule, and sending the data packet to the data receiving end; and the data receiving end is used for receiving the data packets, storing the data packets in a preset high-speed queue, combining the data packets in the high-speed queue into a to-be-analyzed file block according to the specified protocol format, and analyzing the to-be-analyzed file block to obtain the running data. According to the method, the data throughput can be improved, meanwhile, the data analysis speed is improved, the data processing efficiency can be improved to a relatively high degree, and the operation efficiency and the control timeliness of the system are ensured.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing system and method. Background Art

[0002] In industrial application scenarios, it is necessary to collect various operating data of equipment, connect with various industrial equipment through the data acquisition platform, realize data collection and status monitoring of various industrial equipment on site, and then control them.

[0003] Due to the large number and variety of equipment in industrial application scenarios, the collected operating data is usually large-scale concurrent data. The existing technology cannot well implement either the collection of large-scale concurrent data or the transmission of the collected large-scale concurrent data, especially the processing of data files. Because the data volume is large and the data throughput of the data processing system is small, the data collection and transmission are very slow, which affects the operating efficiency of the industrial system and the timeliness of control.

[0004] This shows that the existing technology is not fast enough in processing large concurrent data, which affects the operating efficiency and timeliness of control of industrial systems. Summary of the invention

[0005] In view of this, it is necessary to provide a data processing system and method to solve the problem that the existing technology is not fast enough to process large concurrent data, which affects the operating efficiency and timeliness of control of the industrial system.

[0006] In order to solve the above problems, the present invention provides a data processing system, comprising a data acquisition end and a data receiving end, wherein the data acquisition end is used to collect the operation data of the device to be detected, and package the operation data into a data packet of a specified protocol format according to a preset encoding rule, and send the data packet to the data receiving end; The data receiving end is used to receive data packets, store the data packets in a preset high-speed queue, combine the data packets in the high-speed queue into a file block to be parsed according to a specified protocol format, and parse the file block to be parsed to obtain running data.

[0007] In a possible implementation, the data collection end is further used to identify the data type of the operating data and adjust the sampling frequency of the operating data according to the data type.

[0008] In a possible implementation, the data type includes state data and event data, and the data acquisition end is further configured to adjust the sampling frequency of the event data to be greater than the sampling frequency of the state data when the operation data is identified as event data.

[0009] In a possible implementation manner, when the data acquisition end packages the operation data into a data packet of a specified protocol format according to a preset encoding rule, it is used to: The valid data, type, data length, and preset data packet header of the running data are encoded according to preset encoding rules to obtain a data packet in a specified protocol format.

[0010] In a possible implementation manner, after receiving the data packet, the data receiving end is further configured to: Verify the legitimacy of data packets according to preset configuration information, receive legal data packets, and reject illegal data packets.

[0011] In a possible implementation manner, when the data receiving end combines the data packets in the high-speed queue into the file blocks to be parsed according to the specified protocol format, it is configured to: The data packets corresponding to the same device to be detected are combined into a data block to be parsed according to the format of data length, unique identifier of the device to be detected, timestamp and sequence number.

[0012] In a possible implementation manner, when parsing the file block to be parsed, the data receiving end is used to: When the file block to be parsed is real-time data, determining the encoding rule of the real-time data file according to the unique identifier of the device to be detected in the real-time data file, and parsing the real-time data file based on the encoding rule; When the file block to be parsed is file data, the file block to be parsed is split into a plurality of file sub-blocks to be parsed of preset sizes, and the file sub-blocks to be parsed are parsed respectively according to the formats of the file block to be parsed.

[0013] In a possible implementation manner, the data receiving end is further configured to send the parsed operating data to a subscription service port in a subscription publishing manner.

[0014] In one possible implementation, the operating data is stored in a comma-separated value format.

[0015] The present invention further provides a data processing method, applicable to the data processing system of any of the above embodiments, comprising: The data acquisition terminal collects the operating data of the device to be detected, packages the operating data into a data packet in a specified protocol format according to a preset encoding rule, and sends the data packet to the data receiving terminal; The data receiving end receives the data packets and stores them in a preset high-speed queue, combines the data packets in the high-speed queue into a file block to be parsed according to a specified protocol format, and parses the file block to be parsed to obtain running data.

[0016] The beneficial effects of the present invention are as follows: the data processing system provided by the present invention packages the collected operating data of the device to be detected into a data packet in a specified protocol format according to a preset encoding rule, and transmits the data packet to a data receiving end in the form of a data packet. The data receiving end merges the data packets into file blocks to be parsed through a high-speed queue, and parses the file blocks to be parsed to obtain operating data, thereby improving data throughput and data parsing speed, thereby improving data processing efficiency to a high degree and ensuring the operating efficiency of the system and the timeliness of control. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic diagram of the structure of a data processing system provided by an embodiment of the present invention; Figure 2 A schematic diagram of a data packet provided by an embodiment of the present invention; Figure 3 A flowchart of a data processing method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0020] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0021] A specific embodiment of the present invention, as Figure 1 As shown, a data processing system is disclosed, including a data acquisition terminal 101 and a data receiving terminal 102, wherein the data acquisition terminal is used to collect the operation data of the device to be detected, and package the operation data into a data packet of a specified protocol format according to a preset encoding rule, and send the data packet to the data receiving terminal; The data receiving end is used to receive data packets, store the data packets in a preset high-speed queue, combine the data packets in the high-speed queue into a file block to be parsed according to a specified protocol format, and parse the file block to be parsed to obtain running data.

[0022] In an embodiment of the present invention, the provided data processing system can be used for processing the operating data of various equipment in an industrial system, including the collection and transmission of operating data, etc. Specifically, the industrial equipment may include PLC (Programmable Logic Controller), instruments, sensors, CNC (CNC lathe, numerically controlled machine tools), robots, etc. The operating data includes but is not limited to voltage, current, temperature, pressure, etc.

[0023] In an embodiment of the present invention, the data acquisition end may be a device that integrates a data acquisition sensor and a data processing module. The data acquisition sensor includes but is not limited to a voltmeter, an ammeter, a thermometer, a pressure gauge, etc., which may be configured according to the operation data to be collected. The data processing module may be a chip with data processing functions such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array). The data acquisition end may collect the operation data of various industrial equipment in an industrial system and package the collected operation data. The method for packaging the operation data and the packaging rules are described in detail later in the present invention. After completing the packaging of the operation data, the data acquisition end may send the packaged data packet to the data receiving end.

[0024] In an embodiment of the present invention, the data receiving end includes a data receiving interface and a data processing device. The data receiving interface is used to receive data packets sent by the data acquisition end. The data processing device can be a device or service platform with data processing functions such as a computer, a tablet, a server, etc., which can specifically realize functions such as data reception, parsing, storage, and publishing. After receiving the data packet, the data receiving end can store the data packet in a preset high-speed queue, and buffer the processing of high-frequency data according to a specifically designed file block in the high-speed queue, combine the data packets in the high-speed queue into a file block to be parsed according to a specified protocol format, and parse the file block to be parsed to obtain operation data. The composition method and parsing method of the file block to be parsed will be described in detail later in the present invention.

[0025] The data processing system provided by the present invention packages the collected operating data of the device to be detected into a data packet in a specified protocol format according to a preset coding rule, and transmits the data packet to a data receiving end in the form of a data packet. The data receiving end merges the data packets into file blocks to be parsed by means of a high-speed queue, and parses the file blocks to be parsed to obtain operating data, thereby improving data throughput and data parsing speed, thereby improving data processing efficiency to a high degree and ensuring the operating efficiency of the system and the timeliness of control.

[0026] As a possible implementation manner of the present invention, in this implementation manner, the data acquisition end is also used to identify the data type of the operating data and adjust the sampling frequency of the operating data according to the data type.

[0027] In the embodiment of the present invention, when the data acquisition terminal collects the operation data of the equipment to be detected in the industrial system, it can also identify the type of the operation data, wherein the type of operation data includes state data and event data, state data refers to data used to indicate the operation state of the equipment, such as current, voltage, temperature, pressure, etc., and event data refers to the working conditions of the equipment under the operation state, such as a picking action of a robot, a laminating action of a laminating device, etc. According to the type of operation data, the sampling frequency of the operation data can be adjusted. Specifically, for operation data that will not change in a short time, its sampling frequency can be reduced, while for operation data that may change in a short time, its sampling frequency needs to be increased.

[0028] Furthermore, the data type includes state data and event data, and the data acquisition end is further used to adjust the sampling frequency of the event data to be greater than the sampling frequency of the state data when the operation data is identified as event data.

[0029] In an embodiment of the present invention, as in the aforementioned embodiment, the data type of the operating data includes status data and event data. When the data acquisition end identifies the type of the operating data, when it identifies the operating data as event data, it can increase its sampling frequency so that the sampling frequency of the event data is greater than the sampling frequency of the status data. For example, the sampling frequency of the event data can be increased by 10 times, 100 times, etc., so that the event data can be accurately and detailedly collected, providing accurate data support for event analysis.

[0030] As a possible implementation of the present invention, in this implementation, when the data acquisition end packages the operation data into a data packet of a specified protocol format according to a preset encoding rule, it is used to: The valid data, type, data length, and preset data packet header of the running data are encoded according to preset encoding rules to obtain a data packet in a specified protocol format.

[0031] In the embodiment of the present invention, when the data acquisition end packages the operation data, it is necessary to package the operation data into a data packet of a specified protocol format according to a preset encoding rule. For example, the collected operation data includes temperature, pressure, current and voltage. Figure 2 The figure shows a design method of a data packet header. The data packet consists of a TCP / UDP (Transmission Control Protocol, User Datagram Protocol) header, a payload header, and collected valid data. The specific data packet design is shown in Table 1: Table 1: Data packet design standard table

[0032] Furthermore, the encoding rules of the valid data in the payload are shown in Table 2: Table 2: Encoding rules for valid data in the payload

[0033] Furthermore, the type coding of the running data can be shown in Table 3: Table 3: Encoding rules for running data types

[0034] The embodiment of the present invention packages the operation data according to a preset coding rule, forms a data packet and then sends it, thereby improving the data collection efficiency.

[0035] As a possible implementation manner of the present invention, in this implementation manner, after receiving the data packet, the data receiving end is further used to: Verify the legitimacy of data packets according to preset configuration information, receive legal data packets, and reject illegal data packets.

[0036] In an embodiment of the present invention, the data receiving end can use the native network socket API (Application Programming Interface) and IO (Input / Output) multiplexing technology to receive data packets and verify the legitimacy of the data frame through configuration information. The configuration information is stored in a local database. For example, the configuration information may be "device 1 identifier (device1) - encoding ID (1000)"; further, the encoding ID is associated with the packaging format of the running data. For example, the running data with the encoding ID of 1000 is configured with frame header information, as well as the type of each sensor data and its position in the entire packet.

[0037] Furthermore, when the data receiving end combines the data packets in the high-speed queue into the file blocks to be parsed according to the specified protocol format, it is used to: The data packets corresponding to the same device to be detected are combined into a data block to be parsed according to the format of data length, unique identifier of the device to be detected, timestamp and sequence number.

[0038] In the embodiment of the present invention, the data receiving thread immediately puts the network data packet into the high-speed queue after receiving it. The file cache thread obtains data from the high-speed queue and composes the file block to be parsed according to the format. For some data that need to be published in real time, it can be directly published to the outside after being parsed without entering the file cache.

[0039] Specifically, the file cache block is stored according to different devices and different file names. For example, the collected data of device 1 (device1) is named according to the device unique identifier + timestamp + serial number, such as: device1_xxyear_xxmonth_day_hh_mm_ss_indx_.bin. The size of the file block can be customized, or stored as one file per event. The format of the file block can be: frame length + encoding ID + timestamp + data packet (payload header + payload valid data). The length of each frame of cached data is the same, including frame length-encoding ID-timestamp-data packet. The file cache block consists of multiple frames of cached data. For each file cache block, its status is recorded when it is stored, as shown in Table 4: Table 4: File cache block record table

[0040] Through the above record table, the parsing process can perform data parsing work concurrently. For example, for the data in Table 5, three parsing processes can be used to perform parsing simultaneously.

[0041]

[0042] The embodiment of the present invention merges the data packets in the high-speed queue into the file blocks to be parsed. Each frame of data in the file blocks to be parsed consists of two parts: the frame header and the data. The frame header contains information such as timestamp, coding ID, length, etc. The data of the same device uses a file block named by an identifier, and the length of the data frame is consistent. Through the file cache block and pipeline architecture design, the total processing time of each module is shortened to the processing time of the fastest module, thereby improving data processing efficiency.

[0043] As a possible implementation manner of the present invention, in this implementation manner, when parsing the file block to be parsed, the data receiving end is used to: When the file block to be parsed is real-time data, determining the encoding rule of the real-time data file according to the unique identifier of the device to be detected in the real-time data file, and parsing the real-time data file based on the encoding rule; When the file block to be parsed is file data, the file block to be parsed is split into a plurality of file sub-blocks to be parsed of preset sizes, and the file sub-blocks to be parsed are parsed respectively according to the formats of the file block to be parsed.

[0044] In the embodiment of the present invention, it is necessary to determine the parsing method according to the type of the data block to be parsed. Specifically, when the file block to be parsed is real-time data, the encoding rule ID of the device is found in the configuration table through the unique identifier of the device. The encoding rule of the data and the frame header rule information are found through the encoding rule ID, the frame header is removed, and the acquisition data of the frame is obtained. The hexadecimal to decimal data conversion is performed according to the data length of each acquisition point, and the data is parsed. For example, for the one-pin data shown in Table 6, the first 8 bytes are the frame header, and the parsed decimal data is: {25 (C), 0.2 (MPa) 80 (%), 80 (mA), 80 (V)}, and the unit can be customized.

[0045]

[0046] Furthermore, when the file block to be parsed is file data, a file cache block whose file parsing flag is unparsed (flag is 0) in the database is read to obtain file identifier information and file name information, and the file is read into the memory in a manner of reading a certain size (such as 1MB) at a time. According to the structure of the file block storage defined in the aforementioned embodiment, steps 1 and 2 are repeated in the subsequent steps until the entire file is parsed.

[0047] Step 1: Parse the first 8 bytes to obtain the length, decoding ID, and timestamp information. Get the length to be parsed; Step 2: According to the encoding ID, find the parsing rules of the data packet and parse the original data using the previous parsing method.

[0048] Then, through multi-threading, multiple files are processed in parallel, and the cache files can be deleted as needed after processing.

[0049] The parsing method of the embodiment of the present invention first extracts the frame length, coding ID and timestamp from the file block, extracts the data of the subsequent frame through the length information, uses the coding ID information to find the coding rule of the data, and thus parses the data, and adopts a multi-threaded parallel method to parse different file blocks, thereby maximizing the concurrent parsing speed.

[0050] As a possible implementation manner of the present invention, in this implementation manner, the data receiving end is also used to send the parsed operating data to the subscription service port in a subscription publishing manner.

[0051] In the embodiment of the present invention, the parsed data adopts the subscription publishing method. The data can be packaged using the packaging tool of the search engine and published to the corresponding service port through the message system. The frames with the same timestamp are published as a unit, such as publishing the data of the four acquisition points in the above embodiment at one time. Further, the data file can be stored in the csv (Comma-Separated Values) format. The csv file format only needs to convert the parsed data into a string, and then add commas to separate each test point data, and store it in a row of one frame of data. The parsed data can be stored in a real-time database, using a real-time database commonly used on the market.

[0052] The embodiment of the present invention achieves the purpose of accurately collecting device data by dividing the collected operation data into state data and event data. By pre-defining the event content, when the collection device receives the event command, the sampling frequency is immediately increased, and the data is transmitted to the data receiving end through the protocol of the present invention, so as to provide a more accurate analysis and control basis for the analysis of the event. The collection receiving end avoids many problems of memory caching by caching file blocks, such as large memory occupation affecting program performance, data loss in memory after power failure, difficulty in concurrent parsing of multiple memory blocks, and inability to cache a large amount of data when high-frequency sampling data appears. By designing the storage format of the file block, the parsing process directly processes the local file, and the design that the data receiving and processing processes do not affect each other, the throughput rate of data collection is greatly improved, and the reliability of data collection is improved.

[0053] As a possible implementation of the present invention, in this implementation, a data processing method is also provided, which is applicable to the data processing system of any of the above embodiments. Figure 3 As shown, including: S301, the data collection end collects the operation data of the device to be detected, and packages the operation data into a data packet of a specified protocol format according to a preset encoding rule, and sends the data packet to the data receiving end; S302, the data receiving end receives data packets, and stores the data packets in a preset high-speed queue, and combines the data packets in the high-speed queue into a file block to be parsed according to a specified protocol format, and parses the file block to be parsed to obtain running data.

[0054] Furthermore, when the operation data is identified as event data, the sampling frequency of the event data is increased.

[0055] The data processing method provided in the embodiment of the present invention has the same data processing principle as the data processing system provided in the aforementioned embodiment, which will not be described in detail here.

[0056] The present invention packages the collected operating data of the device to be detected into a data packet in a specified protocol format according to a preset coding rule, and transmits the data packet to a data receiving end in the form of a data packet. The data receiving end merges the data packets into file blocks to be parsed by means of a high-speed queue, and parses the file blocks to be parsed to obtain operating data, thereby improving data throughput and data parsing speed, thereby improving data processing efficiency to a high degree and ensuring the operating efficiency of the system and the timeliness of control.

[0057] The above are only preferred specific implementations of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A data processing system, characterized in that: It includes a data acquisition end and a data receiving end, wherein the data acquisition end is used to collect the operation data of the device to be detected, and package the operation data into a data packet of a specified protocol format according to a preset coding rule, and send the data packet to the data receiving end; The data receiving end is used to receive the data packets, store the data packets in a preset high-speed queue, combine the data packets in the high-speed queue into a file block to be parsed according to the specified protocol format, and parse the file block to be parsed to obtain the running data.

2. The data processing system according to claim 1, characterized in that The data acquisition end is also used to identify the data type of the operating data and adjust the sampling frequency of the operating data according to the data type.

3. The data processing system according to claim 2, characterized in that: The data types include state data and event data. The data acquisition end is further used to adjust the sampling frequency of the event data to be greater than the sampling frequency of the state data when the operation data is identified as event data.

4. The data processing system according to claim 1, characterized in that: When the data acquisition terminal packages the operation data into a data packet of a specified protocol format according to a preset encoding rule, it is used to: The valid data, type, data length, and preset data packet header of the operating data are encoded according to preset encoding rules to obtain a data packet in a specified protocol format.

5. The data processing system according to claim 1, characterized in that: After receiving the data packet, the data receiving end is further used to: The legitimacy of the data packets is verified according to the preset configuration information, legal data packets are received, and illegal data packets are rejected.

6. The data processing system according to claim 1, characterized in that: When the data receiving end combines the data packets in the high-speed queue into the file blocks to be parsed according to the specified protocol format, it is used to: The data packets corresponding to the same device to be detected are combined into a data block to be parsed according to the format of data length, unique identifier of the device to be detected, timestamp and sequence number.

7. The data processing system according to claim 6, characterized in that: When parsing the file block to be parsed, the data receiving end is used to: When the file block to be parsed is real-time data, determining an encoding rule of the real-time data file according to a unique identifier of the device to be detected in the real-time data file, and parsing the real-time data file based on the encoding rule; When the file block to be parsed is file data, the file block to be parsed is split into a plurality of file sub-blocks to be parsed of preset sizes, and the file sub-blocks to be parsed are parsed respectively according to the formats of the file block to be parsed.

8. The data processing system according to claim 1, characterized in that: The data receiving end is also used to send the parsed operation data to a subscription service port in a subscription publishing manner.

9. The data processing system according to claim 1, characterized in that: The operation data is stored in a comma-separated value format.

10. A data processing method, applicable to the data processing system according to any one of claims 1 to 9, characterized in that: include: The data acquisition end collects the operation data of the device to be detected, packages the operation data into a data packet in a specified protocol format according to a preset coding rule, and sends the data packet to the data receiving end; The data receiving end receives the data packet, stores the data packet in a preset high-speed queue, combines the data packets in the high-speed queue into a file block to be parsed according to the specified protocol format, and parses the file block to be parsed to obtain the running data.

Images