Method and system for recovering streaming performance data in an interruption scenario

Abstract

The application discloses a kind of streaming performance data recovery method and system under interrupt scene, it is related to data security field.The steps of the method include: forming session information with performance data, session information includes session performance code and its performance value;After transmission channel is recovered, by nonlinear discrete sampling mode, remove the performance value of the performance code that does not reflect the change characteristics of performance data.The application removes the performance value that does not reflect the change characteristics of performance data to streaming transmission data after transmission channel interruption, so as to reduce the amount of data transmission on the basis of not losing the change characteristics of performance data, and then solve the problem that performance data during interruption cannot be recovered after transmission session interruption recovery in prior art, and the problem that data amount is too large after recovery of traditional cache technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data security, in particular to a method and system for recovering streaming performance data (data continuously generated by multiple data sources, with the properties of large amount, continuity, rapidity and irreproducibility) in an interruption scenario. BACKGROUND

[0002] In engineering applications, a control system often needs to collect performance data of a device in real time through Telemetry technology (a technology for remotely collecting data at a high speed from physical devices or virtual devices) to determine the current running state of the device and make decisions on whether to optimize and adjust the network according to the changes in the corresponding performance indicators.

[0003] When the network transmission is unstable or the control system software is upgraded, the transmission session will be interrupted, and the performance data pushed by the device during the interruption period will be lost, resulting in discontinuity of performance monitoring and loss of some key performance indicators such as latency and packet loss rate, which affects the correctness of the running state evaluation of the control system for the device and the timely network tuning.

[0004] For the loss of performance data, the traditional solution is to cache the performance data and send it after the session is restored. Since the subscription period of streaming performance data is short, if a simple caching solution is used, a large number of messages will be stored in a short period of time. One is to occupy the memory resources of the device, and the other is that a large number of message sending after the session is restored will cause network congestion. SUMMARY

[0005] In view of the defects in the prior art, the technical problem solved by the present application is how to reduce the amount of data transmitted without losing the change characteristics of performance data after recovering the interrupted transmission performance data.

[0006] To achieve the above purpose, in a first aspect, the present application provides a method for recovering streaming performance data in an interruption scenario, comprising the following steps: forming performance data into session information, the session information including session performance codes and their performance values; after the transmission channel is restored, removing the performance values in the performance codes that do not reflect the change characteristics of the performance data by means of nonlinear discrete sampling.

[0007] In combination with the first aspect, in an implementation, the performance value of the session performance code comprises a current value, a minimum value and a maximum value at different time points; the session performance code is defined by three groups of performance values: one group of all current values, one group of all minimum values, and one group of all maximum values; and the process of removing performance values that do not reflect the change characteristics of performance data in the performance code by means of nonlinear discrete sampling comprises: in any group of performance values, removing performance values that do not reflect the change characteristics of performance data in the performance code.

[0008] In combination with the first aspect, in an implementation, the process of removing performance values that do not reflect the change characteristics of performance data in the performance code by means of nonlinear discrete sampling comprises: in each group of performance values, removing performance values that do not reflect the change characteristics of performance data in the performance code.

[0009] In combination with the first aspect, in an implementation, the process of removing performance values that do not reflect the change characteristics of performance data in the performance code in the group of performance values comprises: arranging the performance values according to the continuity of time stamps to form a plurality of data points, and retaining the data points of the head and tail nodes.

[0010] For each data point A i , the performance value of which is represented as V i , the slope of the data point is represented as K i , and the inclination angle is represented as θ i . If K i *K i+1 <0, it is determined whether the data point A i corresponds to an oscillation scenario, if yes, the data point A i is removed, otherwise the data point A i is retained; the determination criterion of the oscillation scenario is that there is a fluctuation error β of the performance value, and |V i -V i-1 |≤β, and |V i+1 -V i |≤β, and |V i+2 -V i+1 |≤β, and K i+1 *K i+2 <0; if K i =K i+1 or the data point A i is removed; wherein is an initial value of the fluctuation angle coefficient, and Δt is the interruption duration of the session; if the data point A i is retained.

[0011] In combination with the first aspect, in an implementation, the method further comprises the following steps after removing the performance values that do not represent the change characteristics of the performance data in the performance code: forming the performance data into a transmission data packet in a specified format, the transmission data packet comprising a sampling type corresponding to a traditional path, an object name, and a performance code total code corresponding to a traditional performance code string.

[0012] In combination with the first aspect, in an implementation, the transmission data packet further comprises a plurality of current performance values, a plurality of minimum performance values, and a plurality of maximum performance values of the performance code total code.

[0013] In combination with the first aspect, in an implementation, the method further comprises the following steps after forming the performance data into a transmission data packet in a specified format: determining whether the transmission channel of the current session is normal, and if so, sending the transmission data packet, otherwise, buffering the transmission data packet and increasing the number of sending failures by 1; when the number of sending failures exceeds a specified threshold, clearing the buffered transmission data packet.

[0014] In combination with the first aspect, in an implementation, the method further comprises the following steps: monitoring the disk occupancy rate c, defining a first threshold value a and a second threshold value b, b > a; when c rises to b and c ≥ b, prohibiting data buffering; when c falls to a and c ≤ a, allowing data buffering.

[0015] In combination with the first aspect, in an implementation, the method further comprises the following steps: storing the session information into a database, the session information association table of the database comprising a session information table and a performance code data table; the session information table comprising a remote IP, a port, and a session label; and the performance code data table comprising an object name, a performance code, the performance value information of the performance code, and a session reference corresponding to the session label.

[0016] The second aspect provides a streaming performance data recovery system in an interrupt scenario, which is used to implement the above method.

[0017] Compared with the prior art, the present application has the following advantages:

[0018] (1) The present application removes the performance values that do not represent the change characteristics of the performance data from the streaming transmission data after the transmission channel is interrupted, thereby reducing the amount of data to be transmitted on the basis of not losing the performance values that represent the change characteristics of the performance data, and solving the problems of the prior art that the performance data during the interruption period cannot be recovered after the transmission session is recovered, and the amount of data is too large after the traditional buffering technology is recovered.

[0019] (2) The present application can transmit a plurality of performance values of one performance code at a time through the self-defined transmission data packet, thereby reducing the number of transmissions and improving the transmission efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative effort based on these drawings are within the scope of the present application.

[0021] Figure 1 Linear diagram for calculating the current performance value of the input optical power in the embodiments of the present application;

[0022] Figure 2 Linear diagram for calculating the current performance value of the input optical power in the embodiments of the present application;

[0023] Figure 3 Architecture diagram for transmitting data packets in the embodiments of the present application;

[0024] Figure 4 System framework diagram of the streaming performance data recovery system in the interruption scenario in the embodiments of the present application. DETAILED DESCRIPTION

[0025] In order to make the purposes, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of the present application.

[0026] The flowcharts shown in the drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the described order. For example, some operations / steps can be further decomposed, combined or partially merged, so the actual execution order may be changed according to the actual situation.

[0027] The application scenario of the stream performance data recovery method in the interrupt scenario in the embodiment of the application is as follows: a GRPC transmission channel (the transmission channel is mainly used to bear the performance data pushed by the device in real time, and is a standard channel) between the device and the remote management and control system is successfully built and a performance data session is carried out, and the performance data is transmitted when the transmission channel is normal; after the transmission channel state monitored is abnormal (that is, interrupted), the performance data (Protobuf format of binary stream) to be sent is deserialized and parsed, session information is formed and cached (that is, sent to the session content of a session), and the session information includes the session tag, session data, session performance code and performance values of each session performance code at different times identified after parsing.

[0028] Therefore, the performance values not reflecting the change characteristics of the performance data are removed through the deserialization, caching and "feature fitting" of the stream transmission data, so that the data amount transmitted is reduced on the basis of not losing the change characteristics of the performance data, and the problems that the performance data during the interruption cannot be recovered after the transmission session is interrupted and recovered, and the data amount is too large after the recovery of the traditional caching technology are solved.

[0029] In actual use, the session information is stored in the interrupt area database, and two session information association tables, that is, a session information table and a performance code data table, are designed.

[0030] Table 1, session information table

[0031]

[0032] As shown in Table 1, the session information table mainly stores the remote IP, port and session tag; the session tag is generated with the remote IP and port as the key, and is sequentially assigned by the system, and the value range of the session tag in the embodiment is [1, 32], that is, 32 sessions are supported at most.

[0033] As shown in Table 2, the performance code data table includes the object name, performance code, performance value information of the performance code and session reference of the corresponding session tag; the performance value information of the performance code includes the current value (current performance value), minimum value and maximum value at different times. The session reference field is designed as a decimal number of "bitwise or". For example, the value 1 identifies the session tag 1, and the value 3 identifies the session tags "1 and 2". On the one hand, the data is simplified, and on the other hand, the subsequent data extraction and statistics are facilitated.

[0034] Table 2, performance code data table

[0035]

[0036] Specifically, each performance code includes 3 groups of performance values, all current values mentioned above are a group, all minimum values are a group, and all maximum values are a group; the process of removing performance values that do not reflect the change characteristics of performance data in each performance code by means of nonlinear discrete sampling above can be: removing performance values that do not reflect the change characteristics of performance data in any one group of performance values of each performance code, and the specific steps include:

[0037] According to the continuity of the timestamp (from large to small or from small to large), the performance values are arranged to form data points A(i-n…i-2, i-1, i, i+1, i+2…i+m), the performance values of the data points are represented as V(i-n…i-2, i-1, i, i+1, i+2…i+m), the slope of the data points is represented as K(i-n…i-2, i-1, i, i+1, i+2…i+m), and the inclination angle of the data points is represented as θ i ,

[0038] The data points that meet the following conditions are retained:

[0039] (1) The data points of the head and tail nodes;

[0040] (2) For each non-head and tail node data point A i :

[0041] When K i *K i+1 <0, it means that it is an inflection point of rising and falling, and it is necessary to determine whether the data point A i corresponds to a shock scene, if so, remove the data point A i , otherwise, it represents that the data point A i corresponds to a normal scene, and the data point A i is retained.

[0042] The judgment standard of the shock scene is: a fluctuation error β of the performance value is set in advance (generally, only the input optical power and the output optical power are set as β=0.05 by default, and other performance codes are not set), and |V i -V i-1 |≤β, and |V i+1 -V i |≤β, and |V i+2 -V i+1 |≤β, and K i+1 *K i+2 <0, remove the data point A i , because at this time it is a shock scene, and the inflection point should be discarded; in actual application, the accuracy of some performance codes is higher, and repeated shocks may occur. For example, repeated rising and falling in the 0.01 or 0.02 interval. At this time, it should not be stored.

[0043] When K i = K i+1 , remove data point A i， . At this time, data point A i is the middle point on a straight line, that is, the point to be discarded.

[0044] When it is considered that the fluctuation trend is not obvious, remove data point A i , where is the initial value of the fluctuation angle coefficient (in this embodiment ), and Δt is the interruption time length of the session, in seconds. As the interruption time length increases, the coefficient adjustment of the fluctuation angle is greater, and the discarded points tend to be more, such as when the interruption time length is 1-59 seconds, the fluctuation angle is 5°; when the interruption time length is 60-119 seconds, the fluctuation angle is 6°. The upper limit of is the fluctuation angle threshold, and the value range is 0°≤μ≤90°, and different fluctuation angle thresholds can be set for each performance code; the smaller the fluctuation angle threshold, the higher the recovery accuracy, and the fluctuation angle threshold can be flexibly set according to the customer's demand.

[0045] When , it means that the fluctuation amplitude is large, and data point A i is retained at this time.

[0046] Next, all current values of the input optical power of a certain port are taken as an example, as shown in Table 3, the data points before calculation are shown in Table 3 and / or Figure 1 , and the data points after calculation are shown in Table 4 and / or Figure 2 .

[0047] Table 3, current performance values of input optical power before calculation

[0048]

[0049] Table 4, current performance values of input optical power after calculation

[0050]

[0051] Through comparison of the number of data points and / or graphical features, it can be obviously obtained that after processing any group of performance values in the above-mentioned manner, the number of data points can be reduced as much as possible without affecting the change characteristics of the performance data. On this basis, the more the number of groups of processed performance values, the more the number of reduced data points, and in this embodiment, each group of performance values is removed according to the above-mentioned manner.

[0052] Preferably, the process of transmitting the performance data includes: forming the performance data into a transmission data packet in a specified format, as shown in Figure 3As shown, the transmission data packet includes the sampling type corresponding to the traditional path, the object name, and the performance code total code corresponding to the traditional performance code string, that is, the simple sampling type and performance code total code of the present application replace the relatively complex sampling path and performance code string of the traditional method,

[0053] Preferably, referring to Figure 3 As shown, the transmission data packet also includes a plurality of current performance values, a plurality of minimum performance values, and a plurality of maximum performance values of the performance code total code. The performance value field is defined as the repeated type, which has the effect of transmitting a plurality of performance values of a performance code at a time, reducing the transmission times, and improving the transmission efficiency. In the existing standard GNMI-Proto model, the "sampling path" and "performance code string" need to be transmitted each time, and only one current value, one maximum value, and one maximum value of the performance code can be transmitted at a time. When the performance code has multiple values, that is, multiple points, it needs to be transmitted multiple times, and the transmission efficiency is low.

[0054] Preferably, after forming the performance data into a transmission data packet in a specified format, the following steps are further included: judging whether the transmission channel of the current session is normal. If yes, the transmission data packet is sent. Otherwise, it represents that the transmission channel is abnormal, indicating that the session has not stopped flashing (that is, the session is interrupted and recovered, and then interrupted again). At this time, the transmission data packet is buffered, and the number of sending failures is increased by 1. When the number of sending failures exceeds a specified threshold (5 times in this embodiment), the buffered transmission data packet is cleared.

[0055] Preferably, the data recovery method further includes the following steps (which can be executed in real time): monitoring the disk occupancy rate c, defining the first threshold value as a (70% in this embodiment) and the second threshold value as b (80% in this embodiment); in the initial state, the buffer switch is set to ON (that is, data buffering is allowed) by default; when c rises to b and c≥b, the buffer switch is turned off (that is, data buffering is prohibited); when c decreases to a and c≤a, the buffer switch is turned on (that is, data buffering is allowed); the values between a and b are floating spaces, which can avoid frequent switching of the switch.

[0056] Therefore, according to the specific embodiments described above, the following effects can be achieved:

[0057] (1) Deserialization of performance data: the binary stream data that needs to be buffered is parsed to the performance code granularity and then stored in the database, which is convenient for subsequent data statistics and merging.

[0058] (2) Provide a "non-linear discrete sampling" method. Based on the "angle of fluctuation, interruption duration, and feature judgment" of the performance code, multiple point data can be abstracted and cropped. This method can effectively reduce the transmission data volume while not affecting the change characteristics reflected by the performance data.

[0059] (3) extend the Proto model, add a performance data transmission structure, used for the transmission of performance data after the interruption is recovered, reduce the amount of redundant data and transmission times, and improve the data transmission efficiency.

[0060] Referring to Figure 4 As shown in the figure, the stream performance data recovery system in the interruption scenario in the embodiment of the application includes a data caching unit, a computing unit, and a plurality of parsing units corresponding to each session.

[0061] The parsing unit is configured to, after the transmission channel state is abnormal, perform deserialization and analysis on the performance data to be sent according to the above process, form session information, and cache the session information to the interruption area database.

[0062] The data caching unit is configured to create the interruption area database and the recovery area database.

[0063] The computing unit is configured to:

[0064] (1) after the transmission channel is recovered, the performance data is transmitted to the control system by removing the performance values that do not reflect the change characteristics of the performance data in each performance code in a non-linear discrete sampling manner according to the process described above.

[0065] (2) determine whether the transmission channel of the current session is normal, if yes, send the transmission data packet, otherwise, it means that the transmission channel is abnormal, cache the transmission data packet to the recovery area database, and add 1 to the number of sending failures, when the number of sending failures exceeds a specified threshold, clear the cached transmission data packet.

[0066] Referring to Figure 4 As shown in the figure, the system further includes a session management module and a disk space monitoring module.

[0067] The session management module is configured to monitor the state of the transmission channel, and send a work signal to the parsing unit when the transmission channel is interrupted, and send a work signal to the computing unit when the transmission channel is recovered after being interrupted.

[0068] The disk space monitoring module is configured to monitor the disk occupancy rate c, and when c rises to b and c is greater than or equal to b, the data caching is prohibited, and when c decreases to a and c is less than or equal to a, the data caching is allowed.

[0069] The embodiment of the application also provides a storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the above method. It should be noted that the storage medium includes a U disk, a mobile hard disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0070] The embodiment of the present application further provides an electronic device, comprising a memory and a processor, the memory storing a computer program which is run on the processor, and the processor implements the above method when executing the computer program.

[0071] Those skilled in the art can understand that all or some steps in the method disclosed above, and the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware or appropriate combination thereof. In the hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components in cooperation. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer readable storage medium, which can include computer readable storage media (or non-transitory media) and communication media (or transitory media).

[0072] As known to those skilled in the art, the term computer readable storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and can be accessed by a computer. In addition, it is known to those skilled in the art that communication media typically includes computer readable instructions, data structures, program modules or other data in modulated data signals such as carrier waves or other transport mechanisms, and can include any information delivery medium.

[0073] For example, the computer readable storage medium can be an internal storage unit of the electronic device of the foregoing embodiments, such as a hard disk or a memory of the electronic device. The computer readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card and the like.

[0074] The above merely describes specific implementation of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the embodiments of the present application, and these modifications or replacements should be covered in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

1. A method for recovering streaming performance data in an interrupt scenario, the method comprising: The method comprises the following steps: forming performance data into session information, the session information comprising session performance codes and performance values thereof; after the transmission channel is recovered, removing the performance values of the performance codes which do not reflect the change characteristics of the performance data by means of nonlinear discrete sampling; The performance values of the session performance codes comprise current values, minimum values and maximum values at different time points; the session performance codes are defined to comprise three groups of performance values: one group of all current values, one group of all minimum values and one group of all maximum values; the process of removing the performance values of the performance codes which do not reflect the change characteristics of the performance data by means of nonlinear discrete sampling comprises: removing the performance values of the performance codes which do not reflect the change characteristics of the performance data in any group of performance values; The process of removing the performance values of the performance codes which do not reflect the change characteristics of the performance data in one group of performance values comprises: arranging the performance values according to the continuity of time stamps to form a plurality of data points, and retaining the data points of the head and tail nodes; For each non-head-tail node data point A i , its performance value is represented as V i , the slope of the data point is represented as K i , and the inclination angle is represented as θ i , If K i *K i+1 <0, it is determined whether the data point A i corresponds to a shock scene, if yes, the data point A i is removed, otherwise the data point A i is retained; the determination criterion of the shock scene is that there is a fluctuation error β of the performance value, and |V i -V i-1 |≤β, and |V i+1 -V i |≤β, and |V i+2 -V i+1 |≤β, and K i+1 *K i+2 <0; if K i =K i+1 or , the data point A i is removed; wherein is the initial value of the fluctuation angle coefficient, and Δt is the interruption duration of the session; the data point A i is retained.

2. The method of claim 1, wherein: The process of removing the performance values of the performance codes which do not reflect the change characteristics of the performance data by means of nonlinear discrete sampling comprises: removing the performance values of the performance codes which do not reflect the change characteristics of the performance data in each group of performance values.

3. The method of claim 1, wherein: After the performance values of the performance codes which do not reflect the change characteristics of the performance data are removed, the method further comprises the following steps: forming the performance data into a transmission data packet of a specified format, the transmission data packet comprising a sampling type corresponding to a conventional path, an object name and a total performance code corresponding to a conventional performance code string.

4. The method of claim 3, wherein: The transmission data packet further comprises a plurality of current performance values, a plurality of minimum performance values and a plurality of maximum performance values of the total performance code.

5. The method of recovering stream performance data in an interrupted scenario of claim 3 or 4, wherein: After the performance data is formed into the transmission data packet of the specified format, the method further comprises the following steps: judging whether the transmission channel of the current session is normal, and if yes, sending the transmission data packet, otherwise, buffering the transmission data packet and increasing the number of sending failures by 1; when the number of sending failures exceeds a specified threshold, clearing the buffered transmission data packet.

6. The method of claim 1, wherein: The method further comprises the following steps: monitoring a disk occupancy rate c, defining a first threshold value as a and a second threshold value as b, and b>a; when c rises to b and c≥b, prohibiting data buffering; when c falls to a and c≤a, allowing data buffering.

7. The method of claim 1, wherein: The method further comprises the following steps: storing the session information into a database, and a session information association table of the database comprising a session information table and a performance code data table; the session information table comprising a remote IP, a port and a session label; and the performance code data table comprising an object name, a performance code, performance value information of the performance code and a session reference corresponding to the session label.

8. A system for recovery of streaming performance data in the event of an outage, the system comprising: The system is used for implementing the method of any one of claims 1-7; The system comprises a data buffering unit, a calculation unit and a plurality of parsing units corresponding to each session; The parsing unit is used for, after the transmission channel state is abnormal, deserializing and parsing the performance data to be sent, forming session information and buffering the session information into an interruption area database; The data buffering unit is used for creating the interruption area database and a recovery area database; The data buffering unit is used for creating the interruption area database and a recovery area database; The computing unit is configured to transmit the performance data to the management system after removing the performance values that do not reflect the change characteristics of the performance data in each performance code by means of nonlinear discrete sampling after the transmission channel is recovered.

Images