Real-time measurement and control accumulated data display analysis optimization method

Abstract

The invention belongs to the field of real-time measurement and control task auxiliary decision making, relates to a real-time measurement and control accumulated data display analysis optimization method, and can carry out real-time data efficient display rendering and decision analysis. According to the real-time data optimization strategy adopted by the invention, the advantages of a fixed-time-length point picking method and a data feature marking method are fully played, and the disadvantages are complemented, so that the data storage capacity is controllable, the algorithm is real-time and efficient, and the stable and reliable requirements of a real-time measurement and control task on the system are met. And performing feature marking on the received data by using a data feature perception function, formulating a data storage threshold by integrating real-time rendering capability and server performance, and realizing automatic optimization storage of the received data according to a fusion optimization algorithm. The composite data optimization strategy obtained by fusing the two methods can meet the real-time performance and information utilization efficiency requirements of the system at the same time, a method for efficiently screening mass real-time information is provided for auxiliary command decision making, and the analysis and utilization efficiency of high-frequency mass real-time data is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of auxiliary decision-making for real-time measurement and control tasks, and relates to a real-time measurement and control accumulated data display and analysis optimization method, which is an efficient display and analysis optimization method capable of intermittently discarding low-value measurement and control accumulated data according to time steps in real time. Background Art

[0002] In the real-time measurement and control tasks of various flight targets, the real-time and accuracy requirements for the data supporting command decisions are very high, and the measurement and control data is an important data source that affects command decisions. With the expansion of the functions and missions of the measurement and control system, the measurement and control data are increasingly showing the characteristics of large data volume, high transmission frequency, and long duration. If the curve data that needs to be accumulated and analyzed is not processed, it is easy to cause the memory container to expand and the pressure of front-end display rendering to increase. In the measurement and control tasks that require long-term operation and large data volume accumulation, it is difficult to meet the real-time display and analysis requirements of such auxiliary decision-making data. Researchers engaged in real-time data processing and analysis have also made many attempts. The solutions given either discard historical data or use intermittent point picking (fixed-time point picking method) to solve the problem that the large amount of data cannot meet the real-time update display. Obviously, both methods have cut useful information to some extent, making it impossible for command decision-makers to fully and accurately grasp the characteristics of the data. In serious cases, important information will be missed, affecting command decision-making judgments. Therefore, studying a real-time data optimization strategy to retain high-value data, discard low-value data when appropriate, and screen out useful information in real time to provide auxiliary decision-making is of great significance to improving the efficiency of real-time measurement and control accumulated data display and analysis, and enabling command decision makers to understand and grasp valuable information from massive data in real time. Summary of the invention

[0003] In order to support the auxiliary decision-making of measurement and control tasks with long-term operation and large data accumulation, the real-time data optimization strategy is studied. This paper proposes an efficient optimization method for real-time cumulative data display and analysis that integrates the fixed-time point selection method and the data feature marking method. This method combines the fixed-time point selection method that can control the data volume and has a small amount of calculation with the data feature marking method that can retain high-value data and meet real-time requirements, and constructs a composite real-time data optimization strategy suitable for real-time measurement and control accumulated data display and analysis. The composite real-time data optimization strategy can give full play to the advantages of the fixed-time point selection method and the data feature marking method, and can complement each other in terms of shortcomings. While realizing point selection based on data features, it prevents the point selection from being too sparse in the data interval that is in a slow change for a long time, resulting in serious discontinuity of the curve. The composite real-time data optimization strategy constructs a point selection time step that is automatically adjusted with the amount of data accumulation according to the fixed-time point selection principle, and designs a data feature perception function at the same time. Finally, the time step of the fixed-time point selection method is sparse several times, and the data feature marking method determines the point selection in the sparse area, realizing the combination of the two methods. The algorithm time complexity is O(M), which meets the requirements of real-time data display and analysis optimization efficiency. Real-time measurement and control accumulated data display and analysis for decision support based on a composite real-time data optimization strategy can meet the needs of efficient display and analysis of massive real-time information, greatly improving the ability and efficiency of the decision support system in accurately selecting and utilizing massive real-time information.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] Real-time measurement and control of cumulative data display and analysis optimization method, first, according to the container structure and quantity of the accumulated data and the system hardware and software performance, set the maximum data storage capacity of each data container, and according to the real-time data reception situation, give the time step calculation method of the fixed-time picking method. Secondly, design the picking point feature perception function of the feature marking method, and perform feature calculation and mark storage on the change amplitude of each set of data received in real time. Then, design a fusion algorithm to realize the combination of the fixed-time picking method and the data feature marking method to form a data optimization strategy. Finally, the accumulated data is optimized in real time according to the optimization strategy to form an auxiliary decision-making information source that can be efficiently displayed and has high analytical value. The following steps are included:

[0006] Step 1: Set the maximum data storage capacity of each data container according to the container structure and quantity of the accumulated data and the system hardware and software performance. According to the real-time data reception situation, give the time step calculation method of the fixed-time point picking method.

[0007] Step 1-1: Set basic data

[0008] To ensure the stable and reliable performance of the system, the maximum data storage capacity M of each data container is set according to the container unit structure size A of the stored and accumulated data, the number of containers N, the front-end real-time rendering ability, and the server hardware performance, such that does not exceed the safe ratio P of the server memory configuration, and the front-end can render M groups of data smoothly without lag when ensuring the real-time update frequency K, where A i is the unit structure size of the i-th container. The real-time received data accumulation amount is Q, the data reception frequency is H, and the time step for selective storage is L.

[0009] Step 1-2: Set the calculation method of the time step L for the fixed-duration selective sampling method

[0010] The time step L needs to be adjusted according to the data accumulation amount Q; when Q < M, all the data can be stored, so the step length L = 1 / H is adopted to make the storage frequency consistent with the reception frequency; when Q ≥ M, recalculate the step length according to to make the amount of data stored by selective sampling always less than M, which can effectively control the storage volume of each task process, where represents rounding up.

[0011] Step 2: Design the selective sampling feature perception function of the feature marking method to perform feature calculation and marked storage on the change amplitude of each group of real-time received data.

[0012] Step 2-1: Design the selective sampling feature perception function of the data feature marking method

[0013] Each group of data is marked with the change amplitude between adjacent data. The calculation formula for the change amplitude tz is as follows:

[0014]

[0015] where n represents the number of elements to be analyzed in each group of data, and x i represents the current value of the i-th element, represents the previous value of the i-th element.

[0016] The feature value calculated according to formula (1) can perceive the data change situation of each element. The larger the amplitude, the easier it is to be retained during the selective sampling process, and the smaller the amplitude, the easier it is to be discarded. Therefore, formula (1) is used as the selective sampling feature perception function.

[0017] Step 2-2: Receive data in real time and synchronously perform feature calculation and marked storage

[0018] Create several data receiving threads according to the real-time data sending groups, and use the data-driven method to receive real-time measurement and control data in parallel. Calculate the features of the newly received data that need to be accumulated and stored according to formula (1), and append the feature value tz as a marker value to the corresponding container of the original data for storage. It should be noted that since the first group of data of each type of data has no previous data for comparison and should not be excluded as the reference data for subsequent comparison, the tz value of the first group of data is directly set to 0 and does not participate in data screening subsequently to prevent it from being excluded during the feature screening process.

[0019] Step 3: Design a fusion algorithm to form a data optimization strategy.

[0020] Step 3-1: Design a fusion algorithm

[0021] When Q < M, there is no need to optimize storage. Store all the feature values solved by the time step of the fixed-duration sampling method in Step 1-2 and the original data in the corresponding containers according to the data feature marking method in Step 2-1.

[0022] When Q ≥ M, storage optimization is required. At this time, according to Step 1-2, the time step of the fixed-duration sampling method is thinned by D times, and the thinning points are determined by the data feature marking method in Step 2-1 to realize the combination of the two methods. The implementation steps are as follows: perform feature marking and storage on the real-time received data. When Q ≥ M, Update the time step in real time, and use the combination of the two methods to find low-value data in the historical data, that is, start traversing from the second group of data to find data that satisfies the minimum feature value and the time step with the previous point is less than L. Analyze that the time complexity of this algorithm is O(M), which can be used for real-time applications.

[0023] Step 3-2: Design a data optimization strategy

[0024] Perform marker storage on the real-time received data and detect the growth of the Q value. When Q ≥ M, find the data that satisfies the step length requirement and has the smallest change amplitude in the historical data according to the fusion algorithm, and exclude it from the data container as the data to be discarded, so as to retain the high-value data and the latest data in the historical data to the greatest extent under the condition that the container storage capacity does not exceed M, and effectively control the situation that the memory occupancy increases infinitely with time.

[0025] Step 4: Optimize the accumulated data in real time according to the data optimization strategy to form an auxiliary decision-making information source that can be efficiently displayed and has high analysis value.

[0026] Step 4-1: Prepare data

[0027] Wait for the data to arrive, parse and store it in a temporary structure according to the data type definition, update the Q value. When Q < M, take L = 1 / H; when Q ≥ M, update the step size L according to the time step calculation formula given in step 3-1.

[0028] Step 4-2: Real-time optimization

[0029] Detect the size relationship between the Q value and the M value. When Q < M, mark and store the real-time received data according to step 2; when Q ≥ M, continue to mark and store the new data, and find the low-value data in the historical data that meets the step size requirements and has the smallest change amplitude according to the fusion algorithm given in step 3-1. Discard the low-value data intermittently (at intervals of the time step) according to the optimization strategy given in step 3-2. Finally, while keeping the storage capacity of each container not exceeding M, update and store a set of continuous data that can best represent the data characteristics in real time, which can be called by the front-end display analysis interface to achieve fast rendering and analysis of curve data.

[0030] Step 4-3: Termination condition

[0031] If the job is not finished, go to step 4-1; otherwise, terminate the operation.

[0032] The beneficial effects of the present invention are as follows:

[0033] The present invention provides a real-time cumulative data optimization method that combines the fixed-duration sampling method and the data feature marking method for efficient display rendering and decision analysis of massive real-time data. It can implement the storage and display of cumulative data according to local conditions based on the system's software and hardware conditions, making the data storage capacity controllable, the algorithm real-time and efficient, and meeting the requirements of the real-time measurement and control task for the stability and reliability of the system. The data feature perception function algorithm constructed in this method is simple, easy to implement, has good calculation effect and real-time performance. When the data reception volume exceeds the storage threshold, the cumulative data can be automatically optimized and stored according to the real-time optimization strategy. For each type of measurement and control data, a set of continuous data that can best represent the data change situation can be selected in real time, and finally an auxiliary decision information source that can be efficiently displayed and has high analysis value is formed, effectively improving the decision information screening efficiency for high-frequency massive real-time data. Brief description of the drawings

[0034] Figure 1 It is the basic flowchart of the present invention.

[0035] Figure 2 It is the data display effect diagram after processing by the optimization strategy in the embodiment of the present invention.

[0036] Figure 3 It is the original data display result diagram without using the optimization strategy of the present invention. Detailed implementation manners

[0037] The following further illustrates the specific implementation manners of the present invention in conjunction with the accompanying drawings and technical solutions.

[0038] Considering the general effect of the algorithm, the curve data is rendered on the web front end, the interface refresh frequency is 2 Hz, and the number of data containers N = 160 is created according to the real-time information classification display requirements. The data reception frequency H i (i = 1,..., N) is initialized to 1 / 2 / 4 / 20 Hz according to the actual situation.

[0039] As Figure 1 shown, the real-time measurement and control cumulative data display analysis and optimization method of the present invention includes the following steps:

[0040] Step 1: Set the maximum data storage capacity of each data container according to the container structure, quantity of the stored and accumulated data, and the system software and hardware performance. According to the real-time data reception situation, give the time step calculation method of the fixed-duration sampling method.

[0041] Step 1-1: Set the basic data

[0042] Combining the real-time rendering ability of the front end and the server performance, set the maximum storage capacity M = 1000 of each data container, and the real-time received data accumulation amount Q i (i = 1,..., N) is initialized to 0.

[0043] Step 1-2: Set the calculation method of the time step L of the fixed-duration sampling method

[0044] The time step L needs to be adjusted with the data accumulation amount Q. When Q < M, all the data can be stored, so the step length L = 1 / H is adopted to make the storage frequency consistent with the reception frequency; when Q ≥ M, recalculate the step length according to The result is shown in Table 1. It can be seen that in this way, the amount of data m sampled and stored for each type of data is always less than M.

[0045] Table 1 Variation of the step length corresponding to different H and Q values and the actual storage amount m obtained

[0046]

[0047] Step 2: Design a sampling feature perception function of the feature marking method to calculate and mark and store the change amplitude of each group of real-time received data.

[0048] Step 2-1: Design a sampling feature perception function of the data feature marking method

[0049] Take formula (1) as the sampling feature perception function to calculate the change amplitude of adjacent groups of each type of data.

[0050] Step 2-2: Receive data in real time, and simultaneously perform feature calculation and marked storage

[0051] Create 9 data receiving threads according to the group of the real-time data sent, and realize synchronous reception of data from 9 multicast addresses. The threads adopt a data-driven method to receive real-time measurement and control data in parallel. For the first group of data of each type, set the tz value to 0. For the subsequent data that needs to be accumulated and stored, perform feature calculation according to formula (1), and append the feature value tz as the marked value to the corresponding container together with the original data.

[0052] Step 3: Design a fusion algorithm to combine the two to form a data optimization strategy

[0053] Step 3-1: Design a fusion algorithm

[0054] When Q < M, there is no need to optimize storage. Store all the feature values solved by the feature marking method and the original data into the corresponding container according to the time step of the fixed-duration sampling method. At this time, the amount of data sampled and stored for each type of data m = Q, that is, store all received data, ensuring the maximum storage of m < M.

[0055] When Q ≥ M, storage optimization is required. At this time, sparse the time step of the fixed-duration sampling method by 10 times, and the sparse places are determined by the data feature marking method for sampling, in order to Update the time step in real time, and start from the second group of data in the historical data, traverse and find the data that satisfies the minimum feature value and the time step with the previous point less than L.

[0056] Step 3-2: Give a data optimization strategy

[0057] Design the data optimization strategy as follows: Mark and store the real-time received data, and detect the growth of the Q value. When Q ≥ M, find the data that satisfies the step length requirement and has the smallest change amplitude in the historical data according to the fusion algorithm, and remove it as low-value data from the data container, so that the actual storage amount m of the container remains at 1000 and does not increase with the growth of data. Thus, when the container storage amount m does not exceed M, the high-value data and the latest data in the historical data are retained to the greatest extent, effectively controlling the situation where the memory occupancy grows infinitely with time.

[0058] Update L and m in Table 1 to Table 2 according to this method.

[0059] Table 2 Variation of L and m values obtained by the fusion algorithm with Q and H

[0060]

[0061] Step 4: Optimize the accumulated data in real time according to the optimization strategy to form an auxiliary decision-making information source that can be efficiently displayed and has high analysis value

[0062] Step 4-1: Prepare data

[0063] Wait for the data to arrive, parse and store it in a temporary structure according to the data type definition, and update the Q value at the same time. When Q < M, take L = 1 / H; when Q ≥ M, update the step size L according to the time step calculation formula given in Step 3-1.

[0064] Step 4-2: Real-time optimization

[0065] Detect the size relationship between the Q value and the M value.

[0066] When Q < M, mark and store the real-time received data according to Step 2. The marking and storage process is as follows: expand the original data structure by tz elements and store them in the marked data structure, and then add them to the corresponding container in an append manner.

[0067] After Q ≥ M, continue to mark and store the new data, and find the low-value data that meets the step size requirements and has the smallest change amplitude in the historical data according to the fusion algorithm given in Step 3-1, and perform intermittent discard processing on the low-value data according to the optimization strategy given in Step 3-2.

[0068] Finally, while keeping the storage capacity of each container not exceeding M, a set of continuous data that best represents the data characteristics is updated in real time, which can be called by the front-end display and analysis interface. For example, Figure 2 For the curve display effect of a set of analysis data (X, Y, Z, V, dT, zt) changing with time, where Figure a shows the curve display result of the measured data (X, Y, Z, V) after quantization analysis, and Figure b shows the analysis data display result of the corresponding time delay dT and state zt. The graphics rendering time is less than 0.2s, meeting the 2Hz update frequency and analysis requirements of the curve data.

[0069] Step 4-3: Termination condition

[0070] If the job is not finished, go to Step 4-1; otherwise, terminate the operation.

[0071] To verify the screening and optimization efficiency of the algorithm for the cumulative curve data, compare the data display effects before and after using the optimization strategy. As mentioned above, Figure 2 For the data result using the optimization algorithm, the curve elements are drawn from 1000 groups of data after optimization; Figure 3 For the display result without optimization, the curve elements in the figure are drawn from 17,000 groups of data before optimization. By comparison, it can be seen that the large data jump points after optimization are retained, and the curve is not distorted, which does not affect the real-time analysis of the curve changes, indicating that the algorithm is feasible; at the same time, due to the small amount of data after optimization, the real-time rendering pressure on the front-end is greatly reduced, which is very suitable for real-time applications.

[0072] Starting from receiving simulated data transmission, the system runs for 1000 seconds. Table 3 records the resource usage before and after optimization. As can be seen from Table 3, with the increase of time, the CPU usage after optimization increases slightly and stabilizes within the performance index range. In the early stage of operation, the memory changes before and after optimization are not much different. At this time, the memory pressure is not great and no optimization is required. In the middle and late stages, when the accumulated data exceeds the maximum value threshold, point storage will occur, that is, low-value data will be intermittently discarded according to the time step. At this time, by comparing the memory growth rate, it can be seen that the memory growth rate after optimization is significantly lower than that before optimization. The experimental results are consistent with the algorithm logic, indicating that the optimization algorithm proposed in the present invention has better control over the container capacity and is suitable for executing measurement and control tasks with a longer duration.

[0073] Table 3 Curve screening optimization algorithm performance test results

[0074]

[0075] In summary, the present invention proposes a real-time cumulative data optimization method that integrates the fixed-duration point selection method and the data feature marking method. Based on this method, efficient display rendering and decision analysis of real-time data can be carried out. The adopted real-time data optimization strategy gives full play to the advantages of the fixed-duration point selection method and the data feature marking method, and complements each other in their disadvantages, so that the data storage capacity is controllable, the algorithm is real-time and efficient, and meets the stable and reliable requirements of the system put forward by the real-time measurement and control task. The received data is feature-marked using the data feature perception function, and the data storage threshold is formulated based on the real-time rendering capability and server performance, and the automatic optimization storage of the received data is realized according to the fusion optimization algorithm. The composite data optimization strategy obtained by integrating the two methods can simultaneously meet the system real-time and information utilization efficiency requirements, and provides a method for efficiently screening massive real-time information for auxiliary command decision-making, and effectively improves the analysis and utilization efficiency of high-frequency massive real-time data.

Claims

1. Real-time measurement and control cumulative data display analysis and optimization method, characterized in that: It includes the following steps: Step 1: Set the maximum data storage capacity of each data container according to the container structure, quantity for storing and accumulating data, and the performance of the system's software and hardware. According to the real-time data reception situation, give the calculation method of the time step of the fixed-duration sampling method. Step 1-1: Set the basic data To ensure stable and reliable system performance, the maximum data storage capacity of each data container is set to M according to the container unit structure size A for storing accumulated data, the number of containers N, the front-end real-time rendering capability, and the server hardware performance, so that The safe ratio P of the server memory configuration is not exceeded, and the front end renders M sets of data smoothly without lag while ensuring the real-time update frequency K, where A i is the unit structure size of the i-th container; the real-time received data accumulation amount is Q, the data receiving frequency is H, and the time step of the selected point storage is L; Step 1-2: Set the calculation method of the time step L of the fixed-duration sampling method The time step L needs to be adjusted according to the data accumulation amount Q; when Q < M, all data can be stored, so the step length L = 1 / H is adopted to make the storage frequency consistent with the receiving frequency; when Q ≥ M, recalculate the step length according to to ensure that the amount of data stored by sampling points is always less than M, where represents rounding up; Step 2: Design the sampling feature perception function of the feature marking method, and perform feature calculation and marked storage on the change amplitude of each group of real-time received data. Step 2-1: Design the sampling feature perception function of the data feature marking method Mark each group of data with the change amplitude between adjacent data. The calculation formula of the change amplitude tz is as follows: Among them, n represents the number of elements to be analyzed in each set of data, x i represents the current value of the i-th element, Represents the previous value of the i-th element; Step 2-2: Receive data in real time, and synchronously carry out feature calculation and marked storage Create several data receiving threads according to the real-time data sending group. Adopt the data-driven method to receive real-time measurement and control data in parallel. Perform feature calculation on the newly received data that needs to be accumulated and stored according to formula (1). Use the feature value tz as the marked value and append it to the corresponding container together with the original data for storage; and set the tz value of the first group of data to 0, and it will not participate in data screening subsequently. Step 3: Design a fusion algorithm to form a data optimization strategy. Step 3-1: Design a fusion algorithm When Q < M, there is no need to optimize storage. Store all the feature values solved by the time step of the fixed-duration sampling method in Step 1-2 and the original data in the corresponding container through the data feature marking method in Step 2-1. When Q≥M, storage optimization is required. At this time, according to step 1-2, the time step of the fixed-time point selection method is sparse by D times. The sparse places are determined by the data feature marking method in step 2-1 to combine the two methods. The implementation steps are: feature marking and storage of the data received in real time. When Q≥M, Update the time step in real time, and use two methods to combine and find low-value data in historical data, that is, traverse from the second set of data to find data that satisfies the minimum eigenvalue and has a time step less than L from the previous point; the time complexity of the algorithm is O(M), which is used for real-time applications; Step 3-2: Design a data optimization strategy Perform marked storage on the real-time received data, and detect the growth of the Q value; when Q ≥ M, find the data in the historical data that meets the step length requirement and has the smallest change amplitude according to the fusion algorithm, and remove it from the data container as the data to be discarded. Step 4: Perform real-time optimization on the accumulated data according to the data optimization strategy to form an auxiliary decision-making information source that can be efficiently displayed and has high analysis value. Step 4-1: Prepare data Wait for the data to arrive, parse and store it in the temporary structure according to the data type definition, update the Q value. When Q < M, take L = 1 / H. When Q ≥ M, update the step length L according to the time step calculation formula given in Step 3-1. Step 4-2: Real-time optimization Detect the size relationship between the Q value and the M value. When Q < M, perform marked storage on the real-time received data according to Step 2; when Q ≥ M, continue to perform marked storage on the new data, and find the low-value data in the historical data that meets the step length requirement and has the smallest change amplitude according to the fusion algorithm given in Step 3-1. Intermittently discard the low-value data at intervals of the time step according to the optimization strategy given in Step 3-2. Finally, while keeping the storage capacity of each container not exceeding M, update and store a set of continuous data that can best represent the data features in real time for the front-end display and analysis interface to call, realizing fast rendering and analysis of curve data; Step 4-3: Termination condition If the job is not finished, go to Step 4-1; otherwise, terminate the operation.

Images