Real-time processing method for frame data of non-stationary time sequence
By real-time processing of non-stationary time series test data in industrial systems, including de-network protocolization and parsing into single-parameter data, and storing it in a time series database, the problem of inefficient data processing is solved, and unified standardization and efficient storage of data is achieved.
Patent Information
- Application Number
- CN202411971330.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-05-02
AI Technical Summary
The non-stationary time series of test data in industrial systems leads to inefficient frame data processing and large differences in data structures, making it difficult to achieve unified management and efficient storage.
It provides a real-time processing method for frame data in a non-stationary time series, and realizes standardization and persistence storage of data by receiving test frame data, de-network protocol processing, parsing it into single-parameter data, and saving it in a time series database.
It realizes unified standardization of complex and variable heterogeneous test data, improves real-time data processing and storage efficiency, and ensures generalized data management and query efficiency.
Smart Images
Figure CN119917498A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of frame data processing, and in particular relates to a real-time processing method for frame data of a non-stationary time series. Background Art
[0002] The field test data generated by industrial systems during the testing process usually comes from different data sources and is transmitted via a variety of bus media. Due to the different technical standards and characteristics of various data sources and field buses, the test data generated by them have great differences in structure, rate, transmission cycle, etc., making this multi-source heterogeneous data present a non-stationary distribution in the time series. At the same time, with the gradual development of industrial system technology, the amount of data generated by test equipment is increasing, which also poses a challenge to the real-time processing efficiency of test frame data.
[0003] In order to meet the requirements of efficient processing and unified management of non-stationary time series test frame data of complex systems, it is necessary to provide a real-time processing method for non-stationary time series frame data to ensure that the test data is converted into standard format data for universal management and persistent storage after high real-time processing. Summary of the invention
[0004] In order to meet the requirements of industrial systems for test frame data processing, the inventors have conducted intensive research and provided a real-time processing method for frame data of non-stationary time series, which effectively solves the demand for real-time conversion of standard single parameters of non-stationary time series test frame data.
[0005] The technical solution provided by the present invention is as follows:
[0006] In a first aspect, a method for real-time processing of frame data of a non-stationary time series comprises:
[0007] Receive test frame data and save them to the network frame data queue in sequence according to the local time when they are received;
[0008] De-network protocol processing is performed on the test frame data in sequence to form original frame data, and the original frame data is saved in the original frame data queue in sequence;
[0009] Perform single-thread or multi-thread parsing on the original frame data, and save the single-parameter data obtained by parsing into a structured single-parameter queue;
[0010] Read the data in the single parameter queue. When the number of data reaches the specified number, classify the batch of data according to the single parameter type, and store all the classified single parameter data tables in the time series database. The time series database uses the local time as the index to sort the data in the single parameter data table by time.
[0011] In a second aspect, a real-time processing device for frame data of a non-stationary time series includes:
[0012] one or more processors;
[0013] A storage device for storing one or more programs;
[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement the real-time processing method for frame data of a non-stationary time series described in the first aspect.
[0015] In a third aspect, a readable storage medium stores a computer program, which, when executed by a processor, implements the real-time processing method for frame data of a non-stationary time series described in the first aspect.
[0016] In a fourth aspect, a computer program product is provided, comprising: a computer program (also referred to as code, or instruction), which, when executed, executes the method for real-time processing of frame data of a non-stationary time series as described in the first aspect.
[0017] A real-time processing method for frame data of a non-stationary time series provided by the present invention has the following characteristics:
[0018] Beneficial effects:
[0019] (1) The present invention provides a real-time processing method for frame data of a non-stationary time series, which normalizes complex and changeable heterogeneous test data into standard single-parameter data, so as to provide a unified data interface to the outside world and perform universal storage and management.
[0020] (2) The present invention provides a real-time processing method for frame data of a non-stationary time series. For the analysis of frame data, two analysis modes, sequential or parallel, are proposed to meet the optimal processing performance requirements in different operating environments. In most cases, the parallel analysis mode greatly improves the analysis processing performance while ensuring the time relationship of the data.
[0021] (3) The present invention provides a real-time processing method for frame data of a non-stationary time series, which adopts time series database modeling and performs persistent storage of single parameter data, which not only improves the real-time storage speed of data, but also improves the efficiency of later query and management of data. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A schematic diagram of a single parameter structure of frame data conversion provided by the design method of the present invention;
[0023] Figure 2 A flowchart of the serial-parallel real-time processing architecture provided by the design method of the present invention;
[0024] Figure 3 A schematic diagram of static attribute configuration of a data frame provided in an embodiment of the present invention;
[0025] Figure 4 A schematic diagram of structured single parameter content provided by an embodiment of the present invention;
[0026] Figure 5 A schematic diagram of the contents of a single parameter data table provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The following detailed description of the present invention will make the features and advantages of the present invention more clear and explicit.
[0028] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0029] The present invention provides a real-time processing method for frame data of a non-stationary time series, such as Figure 1 As shown, the following steps are included:
[0030] Step 1: The network frame data receiving service is a single-threaded service, which receives network test frame data sent externally and saves them in the network frame data queue in sequence.
[0031] Step 2: The network frame processing service is a single-threaded service, which can ensure the monotonicity of the frame data in time. The service performs de-network protocol processing on the network test frame data, including intercepting the frame data according to the protocol positioning, verifying the correctness of the frame data transmission, etc. After processing, the original frame data is formed and the data is saved in the original frame data queue in sequence.
[0032] Step 3: The frame data consumption service itself is a single-threaded service. The service manages the frame data parsing thread pool and can perform single-threaded or multi-threaded frame data parsing processing. The parsing function is completed by a single-parameter parser. By configuring parameters, the service provides two original frame data parsing modes: sequential flow operation and parallel efficient operation. Both modes maintain the time monotonicity when a single parameter is enqueued, ensuring the time authenticity of the data during real-time processing, and can meet real-time operation requirements such as real-time data display.
[0033] The sequential pipeline operation is to parse each single parameter of the original frame data in a loop in the same thread, and save the parsed single parameter data into a structured single parameter queue.
[0034] Parallel efficient operation is to parse all single parameters of the original frame data in parallel in n threads, and save them to the structured single parameter queue immediately after the parsing is completed. The end time of the original frame data processing is based on the completion time of the last single parameter parsing. The next frame can be processed only after the original frame data processing is completed. In this way, while the parallel parsing is in progress, the single parameters can be ensured to enter the single parameter queue in a time-monotonous order.
[0035] For the original frame data, the data attributes it carries include dynamic attributes and static attributes. The dynamic attributes include: data generation time, data project, data source. Static attributes include: parameter name, offset, length, data type, calculation formula, unit, data range, etc. Therefore, the structure of the standard single parameter is designed to be divided into three parts: dynamic attributes, static attributes, and data value. That is, each single parameter includes parameter name, data generation time, data project, data source, data type, calculation formula, unit, range, data value, etc.
[0036] (a) Data generation time is the time when the frame data is tested and generated by the test equipment, and its format is UTC time format. In order to avoid time asynchrony between test equipment, a local time received by the processing system is added to the data generation time, and the format is UTC time format, so that the time series relationship of each data can be clearly distinguished.
[0037] (b) The project to which the data belongs is the logical division attribute of the data at the test time, that is, the project range with actual physical meaning.
[0038] (c) The data source is the identifier of the test device that generates the frame data. To facilitate the uniqueness of the protocol for locating data in a complex system, the data source is defined using a data source ID.
[0039] (d) The parameter offset and length attributes are the basis for locating the starting position and byte length of a single parameter in the frame data. These two attributes can be used to extract a single parameter from the frame.
[0040] (e) Data type determines the data type of a single parameter. The single parameter can be assigned to a variable of a specific data type for subsequent processing.
[0041] (f) The calculation formula is optional, that is, whether further calculation processing is required after receiving a single parameter.
[0042] (g) The unit is the physical unit of a single parameter.
[0043] (h) Data range is optional, that is, whether a single parameter is within a certain data range.
[0044] The frame data is decomposed into single parameter processes:
[0045] Design a single parameter parser, whose input is the original frame data and parameter sequence number, and output is a single parameter with standard structure;
[0046] The internal workflow of the single parameter parser is to find the specific attribute configuration information of the original frame data in the static attribute through the information of the data source (data source ID) carried in the original frame data. Then the attribute configuration of the single parameter is determined through the parameter sequence number. Finally, through a series of operations such as shifting, intercepting, calculating and converting, adding units, and judging the range in the memory of the original frame data, the single parameter is structured and assigned. The decomposition process is shown in Figure 2 .
[0047] A single parameter parser completes the parsing of a single parameter. Therefore, the single parameter parser can be designed as a sequential flow operation or as a parallel and efficient operation. The design pattern is flexible and convenient.
[0048] Step 4: The single parameter warehousing service is a multi-threaded service. Each thread reads the data in the single parameter queue. When the number of data reaches a batch level, the batch of data is classified according to the single parameter type, and all the classified single parameter data tables are entered into the time series database at one time; the time series database uses the local time as the index to sort the data by time.
[0049] The time series database is used to store the parsed and standardized single-parameter data. The time series database design effectively solves the problem of low storage efficiency caused by the rapid expansion of data volume after the frame data is converted into a single parameter.
[0050] The time series database uses a single parameter to create a table, that is, one parameter has a parameter data table, and the table name is the parameter name. The specific design is shown in Table 1.
[0051] The columns of the parameter data table are: local time, original time, data value (different data types are designed as different columns, and one value is only stored in one column), unit, data range, and the label column is designed as the project and data source ID. The label column is used to classify and segment the data.
[0052] When single-parameter data is stored in the database, the local time ensures the uniqueness and monotonicity of the time and serves as the time index column of the time series database data table.
[0053] Table 1
[0054]
[0055] Example
[0056] This embodiment is described by taking two typical industrial field test standard protocol frames as examples, 1553B standard protocol format data frame and CAN bus standard protocol format data frame, wherein the 1553B data frame is a position measurement frame with a period of 100ms, and the CAN bus data frame is a power status data frame with a period of 1s.
[0057] The 1553B position measurement data frame structure is shown in Table 2 below:
[0058] Table 2
[0059]
[0060]
[0061] The CAN bus power status measurement data frame structure is shown in Table 3 below:
[0062] Table 3
[0063]
[0064] The test data is transmitted to this design project via the network. The network protocol encapsulation format is shown in Table 4 below:
[0065] Table 4
[0066] Frame Header Data Area check Frame end
[0067] The data source ID of 1553B data is defined as a command word composed of RT address, sub-address, message length, and send and receive flag. The data source ID of CAN bus data is defined as the frame ID of CAN bus. The design standards of the two data source IDs can uniquely determine the data protocol. For details on the static attribute configuration of the two frame data, see Figure 3 .
[0068] The parallel real-time processing architecture workflow in the design method of the present invention is used to process it, and the working process is as follows:
[0069] (1) The network frame data receiving service continuously receives the test data transmitted from the network and saves the network protocol frame shown in Table 4 into the network frame data queue.
[0070] (2) The network frame data processing service cyclically reads the data in the network frame data queue and performs network protocol removal processing on it, that is, removes the frame header and frame tail, and performs verification. If the verification is correct, the content of the data area is the original data frame that meets the processing requirements, as shown in Table 2 and Table 3. The original data frame is saved in the original frame data queue.
[0071] (3) The frame data consumption service loop reads the data in the original frame data queue, and determines the number of single parameters by configuring the index static properties corresponding to the data source ID. Taking 1553B data as an example, there are 3 single parameters in total, so 3 single parameter parsers are used to process the frame at the same time. After processing, structured single parameters position X, position Y, and position Z are formed respectively, and each parameter is accompanied by the relevant static and dynamic properties involved in this method, among which the local time attribute is the machine time currently processing the data. The three single parameters are saved in the single parameter data queue respectively. After all the data of this frame is processed, the next frame data is processed in a loop. The CAN bus data is processed in the same way according to its order in the frame data queue to form four structured single parameters: voltage 1, current 1, voltage 2, and current 2, which are saved in the single parameter data queue. Taking the position X parameter as an example, the standard single parameter structure after parsing is detailed in the following table. Figure 4 .
[0072] (4) In the multi-threaded single parameter storage service, each thread simultaneously loops and reads the data in the single parameter data queue. When the number of data reaches a specified batch, a batch of single parameter data is divided by type and stored in different data tables in the time series database (position X data table, position Y data table, position Z data table, voltage 1 data table, current 1 data table, voltage 2 data table, current 2 data table) at one time. The local time is the primary key column of the data table, and the time series database can sort the data based on this. Taking the position X parameter as an example, the schematic diagram of the data table storage content is shown in Figure 5 .
[0073] According to the above workflow, the two types of test data are continuously processed in real time and stored in the database.
[0074] The present invention processes industrial field test data in real time, ensures the time relationship between data, and ensures that the real-time processing performance and data storage performance of data are optimal. It provides a good foundation for real-time data subscription and post-data analysis and query, and is a feasible and effective overall solution.
[0075] The present invention also provides a real-time processing device for frame data of a non-stationary time series, comprising:
[0076] one or more processors;
[0077] A storage device for storing one or more programs;
[0078] When the one or more programs are executed by the one or more processors, the one or more processors implement the real-time processing method for frame data of a non-stationary time series described in the first aspect.
[0079] The present invention also provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the real-time processing method for frame data of a non-stationary time series described in the first aspect.
[0080] The readable storage medium includes, but is not limited to, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0081] The present invention also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed, executes the real-time processing method for frame data of a non-stationary time series described in the first aspect.
[0082] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, microwave, etc.) mode to another website site, computer, server or data center.
[0083] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0084] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0085] The present invention has been described in detail above in conjunction with specific implementations and exemplary examples, but these descriptions cannot be understood as limiting the present invention. Those skilled in the art understand that, without departing from the spirit and scope of the present invention, a variety of equivalent substitutions, modifications or improvements may be made to the technical solution of the present invention and its implementation methods, all of which fall within the scope of the present invention. The scope of protection of the present invention shall be subject to the attached claims.
[0086] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.
Claims
1. A real-time processing method for frame data of a non-stationary time series, characterized in that: include: Receive test frame data and save them to the network frame data queue in sequence according to the local time when they are received; De-network protocol processing is performed on the test frame data in sequence to form original frame data, and the original frame data is saved in the original frame data queue in sequence; Perform single-thread or multi-thread parsing on the original frame data, and save the single-parameter data obtained by parsing into a structured single-parameter queue; Read the data in the single parameter queue. When the number of data reaches the specified number, classify the batch of data according to the single parameter type, and store all the classified single parameter data tables in the time series database. The time series database uses the local time as the index to sort the data in the single parameter data table by time.
2. The real-time processing method for frame data of a non-stationary time series according to claim 1 is characterized in that: In the step of receiving the test frame data, the test frame data is received in a single-threaded manner.
3. The real-time processing method for frame data of a non-stationary time series according to claim 1 is characterized in that: In the step of sequentially removing network protocol processing from the test frame data, the test frame data is sequentially removed from network protocol processing in a single-threaded manner.
4. The real-time processing method for frame data of a non-stationary time series according to claim 1, characterized in that: In the step of performing single-thread or multi-thread parsing processing on the original frame data and saving the single-parameter data obtained by the parsing into a structured single-parameter queue, the original frame data includes dynamic attributes and static attributes, wherein the dynamic attributes include the data generation time, the project to which the data belongs, and the data source to which the data belongs, and the static attributes include the parameter name, offset, length, data type, calculation formula, unit, and data range; The structure of the single parameter obtained by parsing is designed as dynamic attributes, static attributes, and data value parts. Each single parameter includes parameter name, data generation time, data project, data source, data type, calculation formula, unit, data range, and data value.
5. The real-time processing method for frame data of a non-stationary time series according to claim 1 is characterized in that: The step of performing single-thread or multi-thread parsing processing on the original frame data and saving the single parameter data obtained by the parsing into a structured single parameter queue, wherein the thread parsing processing is to parse each single parameter of the original frame data in a loop in the same thread in turn, and save the parsed single parameter data into a structured single parameter queue.
6. The real-time processing method for frame data of a non-stationary time series according to claim 1, characterized in that: In the step of performing single-thread or multi-thread parsing processing on the original frame data and saving the single parameter data obtained by the parsing into a structured single parameter queue, the multi-thread parsing processing is to parse all single parameters of the original frame data in parallel in multiple threads, and save them into the structured single parameter queue immediately after the parsing is completed; the end time of the original frame data processing is based on the completion time of the last single parameter parsing; and the next frame processing is performed after the original frame data processing is completed.
7. The real-time processing method for frame data of a non-stationary time series according to claim 1, characterized in that: The step of performing single-thread or multi-thread parsing processing on the original frame data and saving the single parameter data obtained by the parsing into a structured single parameter queue, the parsing process is: through the information of the data source carried in the original frame data, find the attribute configuration information of the original frame data in the static attributes; then determine the attribute configuration of the single parameter through the parameter sequence number, and finally perform structured assignment on the single parameter through shifting, intercepting, calculating and converting, adding units, and judging the range operations in the memory of the original frame data.
8. The real-time processing method for frame data of a non-stationary time series according to claim 1, characterized in that: The time series database uses a single parameter to create a table, one parameter is a parameter data table, and the table name is parameter name; The design columns of the parameter data table include: local time, original time, data value, unit and data range, among which the design label columns are the project to which they belong and the data source ID.
9. A real-time processing device for frame data of a non-stationary time series, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the real-time processing method for frame data of a non-stationary time series as described in one of claims 1 to 8.
10. A computer program product, characterized in that The computer program product comprises: a computer program, which, when executed, executes the real-time processing method for frame data of a non-stationary time series according to any one of claims 1 to 8.