Streaming performance data recovery method and system in interruption scene

Abstract

The invention discloses a streaming performance data recovery method and system in an interruption scene, and relates to the field of data security. The method comprises the following steps of: forming session information from performance data, wherein the session information comprises a session performance code and a performance value thereof; and after the transmission channel is recovered, removing the performance values which do not reflect the change characteristics of the performance data in the performance codes in a nonlinear discrete sampling manner. According to the method, after the transmission channel is interrupted, the performance value which does not reflect the change characteristic of the performance data is removed from the streaming transmission data, so that the transmitted data volume is reduced on the basis of not losing the change characteristic which reflects the performance data, and the problems that in the prior art, after transmission session interruption is recovered, the performance data in the interruption period cannot be recovered, and the transmission efficiency is low are solved. And the problem of overlarge data volume after recovery of a traditional caching technology is solved.

Description

Technical Field

[0001] The present invention relates to the field of data security, and in particular to a method and system for recovering streaming performance data (data continuously generated by multiple data sources, with the properties of being large, continuous, fast and non-reproducible) in an interruption scenario. Background Art

[0002] In engineering applications, management and control systems often need to collect device performance data in real time through Telemetry (network telemetry technology, which is a technology for remotely collecting data from physical or virtual devices at high speed) to determine the current operating status of the device and decide whether network optimization and adjustment are needed based on changes in corresponding performance indicators.

[0003] When the network transmission is unstable or the management and control system software is upgraded, the transmission session will be interrupted, and the performance data pushed by the device during the interruption will be lost. This will lead to discontinuity in performance monitoring and the loss of some key performance indicators, such as latency and packet loss rate. This affects the correctness of the management and control system's assessment of the device's operating status and makes it impossible to perform network optimization in a timely manner.

[0004] The traditional solution to performance data loss is to cache the performance data and send it after the session is restored. Since the subscription cycle of streaming performance data is short, if a simple cache solution is used, a large number of messages will be stored in a short period of time. First, it occupies the memory resources of the device, and second, sending a large number of messages after the session is restored will cause network congestion. Summary of the invention

[0005] In view of the defects existing in the prior art, the technical problem solved by the present invention is: after the transmission of performance data is interrupted and the transmission is resumed, how to reduce the amount of transmitted data without losing the changing characteristics of the performance data.

[0006] To achieve the above objectives, in the first aspect, an embodiment of the present application provides a method for recovering streaming performance data in an interruption scenario, comprising the following steps: converting performance data into session information, the session information including a session performance code and its performance value; after the transmission channel is restored, removing the performance values ​​in the performance code that do not reflect the changing characteristics of the performance data by nonlinear discrete sampling.

[0007] In combination with the first aspect, in one embodiment, the performance value of the session performance code includes the current value, the minimum value and the maximum value at different times; the session performance code is defined to include 3 groups of performance values: all current values ​​as a group, all minimum values ​​as a group, and all maximum values ​​as a group; the process of removing the performance values ​​that do not reflect the changing characteristics of the performance data in the performance code by nonlinear discrete sampling includes: in any group of performance values, removing the performance values ​​that do not reflect the changing characteristics of the performance data in the performance code.

[0008] In combination with the first aspect, in one embodiment, the process of removing performance values ​​in the performance code that do not reflect the changing characteristics of the performance data by nonlinear discrete sampling includes: in each group of performance values, removing the performance values ​​in the performance code that do not reflect the changing characteristics of the performance data.

[0009] In conjunction with the first aspect, in one implementation, the process of removing performance values ​​in a group of performance values ​​that do not reflect the change characteristics of performance data in the performance code includes: arranging the performance values ​​to form a plurality of data points according to the continuity of timestamps, and retaining the data points of the first and last nodes;

[0010] For each non-first and last node data point A i , whose performance value is expressed as V i , the slope of the data point is denoted as K i , the tilt angle is denoted as θ, If K i *K i+1 <0, judge data point A i Does it correspond to the shock scenario? If so, remove data point A i , otherwise keep data point A i ; The criteria for judging the oscillation scenario are: the fluctuation error β of the performance value is pre-set, and |V i -V i-1 |≤β, and |V i+1 -V i |≤β, and |V i+2 -V i+1 |≤β, and K i+1 *K i+2 <0; if K i =K i+1 Or |θ i+1 -θ i |≤ε+Δt / 60, remove data point A i ; where ε is the initial value of the fluctuation angle coefficient, Δt is the duration of the session interruption; if |θ i+1 -θ i |>ε+Δt / 60, retain data point A i .

[0011] In combination with the first aspect, in one embodiment, after removing the performance values ​​in the performance code that do not reflect the changing characteristics of the performance data, the following steps are also included: the performance data is formed into a transmission data packet in a specified format, and the transmission data packet includes a sampling type corresponding to the traditional adopted path, an object name, and a total performance code corresponding to the traditional performance code string.

[0012] In combination with the first aspect, in one implementation, the transmission data packet also includes several current performance values, several minimum performance values ​​and several maximum performance values ​​of the performance code total code.

[0013] In combination with the first aspect, in one embodiment, after the performance data is formed into a transmission data packet of a specified format, the following steps are also included: determining whether the transmission channel of the current session is normal, and if so, sending the transmission data packet; otherwise, caching the transmission data packet and increasing the number of sending failures by 1; when the number of sending failures exceeds a specified threshold, clearing the cached transmission data packet.

[0014] In combination with the first aspect, in one embodiment, the method also includes the following steps: monitor the disk usage c, define the first-level threshold as a, the second-level threshold as b, b>a; when c rises to b and c≥b, prohibit data caching; when c decreases to a and c≤a, allow data caching.

[0015] In combination with the first aspect, in one embodiment, the method also includes the following steps: storing the session information into a database, the session information association table of the database includes a session information table and a performance code data table; the session information table includes a remote IP, a port and a session tag; the performance code data table includes an object name, a performance code, the performance value information of the performance code, and a session reference corresponding to the session tag.

[0016] In a second aspect, an embodiment of the present application provides a streaming performance data recovery system in an interruption scenario, which is used to implement the above method.

[0017] Compared with the prior art, the advantages of the present invention are:

[0018] (1) After the transmission channel is interrupted, the present invention removes the performance values ​​that do not reflect the change characteristics of the performance data from the streaming transmission data, thereby reducing the amount of data transmitted without losing the change characteristics of the performance data. This solves the problem in the prior art that after the transmission session is interrupted and restored, the performance data during the interruption period cannot be restored, and the problem that the amount of data is too large after the traditional caching technology is restored.

[0019] (2) The present invention can transmit multiple performance values ​​of a performance code at one time through a customized transmission data packet, thereby reducing the number of transmissions and improving transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is a linear schematic diagram of the current performance value of the input optical power before calculation in an embodiment of the present invention;

[0022] Figure 2 It is a linear schematic diagram of the current performance value of the calculated input optical power in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of an architecture for transmitting data packets in an embodiment of the present invention;

[0024] Figure 4 It is a system framework diagram of a streaming performance data recovery system in an interruption scenario in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0026] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0027] The application scenario of the streaming performance data recovery method in the interruption scenario in the embodiment of the present invention is: the GRPC transmission channel between the device and the remote control system (the transmission channel is mainly used to carry the performance data pushed by the device in real time, which is a standard channel) is successfully established and the performance data session is carried out, and the performance data is transmitted when the transmission channel is normal; when the monitored transmission channel state is abnormal (i.e., interrupted), the performance data to be sent (Protobuf format of the binary stream) is deserialized and parsed to form session information and cache (i.e., the session content sent to a certain session), and the session information includes the session tag, session data, session performance code, and performance values ​​of each session performance code at different times identified after parsing. After the transmission channel is restored, the performance data is transmitted after removing the performance values ​​that do not reflect the change characteristics of the performance data in each performance code through nonlinear discrete sampling.

[0028] It can be seen from this that the present invention removes performance values ​​that do not reflect the changing characteristics of performance data by deserializing, caching and "feature fitting" the streaming data, thereby reducing the amount of data transmitted without losing the changing characteristics of the performance data, thereby solving the problem in the prior art that after the transmission session is interrupted and restored, the performance data during the interruption period cannot be restored, and the problem of excessive data volume after recovery using traditional caching technology.

[0029] In actual use, this embodiment stores the session information in the interruption area database, and designs two session information association tables: a session information table and a performance code data table.

[0030] Table 1. Session information table

[0031]

[0032] As shown in Table 1, the session information table mainly stores the remote IP, port and session tag; the session tag is generated with the remote IP and port as keywords and is assigned sequentially by the system. In this embodiment, the value range of the session tag is [1,32], that is, a maximum of 32 sessions are supported.

[0033] As shown in Table 2, the performance code data table includes the object name, performance code, performance value information of the performance code, and the session reference of the corresponding session tag; the performance value information of the performance code includes the current value (current performance value), minimum value and maximum value at different times. The session reference field is designed as a "bitwise OR" decimal number. For example, the value 1 identifies the session tag 1, and the value 3 identifies the session tags "1 and 2". On the one hand, it simplifies the data, and on the other hand, it facilitates subsequent data extraction and statistics.

[0034] Table 2. Performance code data table

[0035]

[0036] Specifically, each performance code includes three groups of performance values, all current values ​​mentioned above are one group, all minimum values ​​are one group, and all maximum values ​​are one group; the process of removing the performance values ​​that do not reflect the change characteristics of the performance data in each performance code by nonlinear discrete sampling can be: in any group of performance values ​​of each performance code, the performance values ​​that do not reflect the change characteristics of the performance data are removed, and the specific steps include:

[0037] According to the continuity of timestamps (from large to small or from small to large), the performance values ​​are arranged to form a number of data points A (in…i-2, i-1, i, i+1, i+2…i+m). The performance value of the data point is expressed as V (in…i-2, i-1, i, i+1, i+2…i+m), the slope of the data point is expressed as K (in…i-2, i-1, i, i+1, i+2…i+m), and the inclination angle of the data point is expressed as θ.

[0038] Keep data points that meet the following criteria:

[0039] (1) Data points of the first and last nodes;

[0040] (2) For each data point A that is not the first or last node i :

[0041] When K i *K i+1 <0, it indicates an inflection point of rising and falling, and it is necessary to judge data point A i Does it correspond to the shock scenario? If so, remove data point A i , otherwise it represents data point A i Corresponding to the normal scene, retain data point A i .

[0042] The criteria for determining the oscillation scenario are: the fluctuation error β of the performance value is preset (generally, only β = 0.05 is set for the input optical power and output optical power by default, and β is not set for other performance codes), and |V i -V i-1 |≤β, and |V i+1 -V i |≤β, and |V i+2 -V i+1 |≤β, and K i+1 *K i+2 <0, remove data point A i , because in this case, the inflection point should be discarded in the oscillation scenario; in actual applications, some performance codes have high precision and may oscillate repeatedly. For example, it rises and falls repeatedly in the range of 0.01 or 0.02. At this time, it should not be stored.

[0043] When Ki =K i+1 When data point A is removed i , which means that data point A i The middle point on a straight line is the point that must be discarded.

[0044] when It is believed that the fluctuation trend is not obvious, so data point A is removed i ,in is the initial value of the fluctuation angle coefficient (in this embodiment ), Δt is the interruption duration of the session, in seconds. As the interruption duration increases, the greater the adjustment of the fluctuation angle coefficient, the more points will be discarded. For example, when the interruption duration is 1-59 seconds, the fluctuation angle is 5°; when the interruption duration is 60-119 seconds, the fluctuation angle is 6°. The upper limit is the fluctuation angle threshold, and the value range is 0°≤μ≤90°. Each performance code can set a different fluctuation angle threshold. The smaller the fluctuation angle threshold, the higher the recovery accuracy. It can be flexibly set according to customer needs.

[0045] when This indicates that the fluctuation is large, so keep data point A. i .

[0046] The following takes all current values ​​of the input optical power of a certain port as an example, as shown in Table 3. The data points before calculation are shown in Table 3 and / or Figure 1 As shown, the calculated data points are shown in Table 4 and / or Figure 2 shown.

[0047] Table 3. Current performance values ​​of input optical power before calculation

[0048]

[0049] Table 4. Current performance values ​​of calculated input optical power

[0050]

[0051] By comparing the number of data points and / or graphic features, it can be clearly concluded that after processing any group of performance values ​​in the above manner, the number of data points can be reduced as much as possible without affecting the changing characteristics reflected in the performance data; on this basis, the more performance value groups are processed, the more data points are reduced. This embodiment will remove "unobvious" performance values ​​for each group of performance values ​​in the above manner.

[0052] Preferably, the process of transmitting the performance data includes: forming the performance data into a transmission data packet in a specified format, see Figure 3As shown, the transmission data packet includes a sampling type corresponding to the traditional adopted path, an object name, and a performance code total code corresponding to the traditional performance code string, that is, the present invention uses a simple sampling type and a performance code total code to replace the traditional relatively complex sampling path and performance code string, respectively.

[0053] Preferably, see Figure 3 As shown, the transmission data packet also includes several current performance values, several minimum performance values ​​and several maximum performance values ​​of the total performance code. The performance value field is defined as a repeated type, which has the effect of transmitting multiple performance values ​​of a performance code at one time, reducing the number of transmissions and improving transmission efficiency. In the existing standard GNMI-Proto model, the "sampling path" and "performance code string" need to be transmitted each time, and only one current value, one maximum value and one maximum value of the performance code can be transmitted at one time. When the performance code has multiple sets of values, that is, multiple points, it needs to be transmitted multiple times, and the transmission efficiency is low.

[0054] Preferably, after the performance data is formed into a transmission data packet of a specified format, the following steps are also included: determining whether the transmission channel of the current session is normal, and if so, sending the transmission data packet; otherwise, it means that the transmission channel is abnormal, indicating that the session is constantly interrupted (i.e., the session is interrupted and then interrupted again immediately after recovery). At this time, the transmission data packet is cached and the number of sending failures is increased by 1. When the number of sending failures exceeds the specified threshold (5 times in this embodiment), the cached transmission data packet is cleared.

[0055] Preferably, the above data recovery method also includes the following steps (which may be executed in real time): monitor the disk occupancy c, define the first-level threshold as a (70% in this embodiment) and the second-level threshold as b (80% in this embodiment); in the initial state, the cache switch defaults to ON (i.e., data caching is allowed); when c rises to b and c≥b, turn off the cache switch (i.e., prohibit data caching); when c decreases to a and c≤a, turn on the cache switch (i.e., allow data caching); the value between a and b is a floating space, which can avoid frequent switching of the switch.

[0056] It can be seen that the following effects can be achieved according to the above specific implementation methods:

[0057] (1) Deserialize performance data: Parse the binary stream data to be cached into performance code granularity and store it in the database to facilitate subsequent data statistics and merging.

[0058] (2) Provide a "nonlinear discrete sampling" method, based on the "fluctuation angle, interruption duration, feature judgment" of the performance code, which can abstractly cut multiple point data. It can effectively reduce the amount of transmitted data without affecting the change characteristics reflected by the performance data.

[0059] (3) Expand the Proto model and add a performance data transmission structure. This is used to transmit performance data after interruption recovery. This reduces the amount of redundant data and the number of transmissions, thus improving data transmission efficiency.

[0060] See also Figure 4 As shown, the streaming performance data recovery system in an interruption scenario in an embodiment of the present invention includes a data cache unit, a calculation unit and a plurality of parsing units corresponding to each session.

[0061] The parsing unit is used to: when the transmission channel state is abnormal, after deserializing and parsing the performance data to be sent according to the above process, form session information and cache it in the interruption area database;

[0062] The data cache unit is used to: create an interruption area database and a recovery area database;

[0063] The computing unit is used to:

[0064] (1) After the transmission channel is restored, the performance data is transmitted to the control system according to the above process, by removing the performance values ​​that do not reflect the change characteristics of the performance data in each performance code through nonlinear discrete sampling.

[0065] (2) Determine whether the transmission channel of the current session is normal. If so, send the transmission data packet. Otherwise, it means that the transmission channel is abnormal. The transmission data packet is cached in the recovery area database and the number of transmission failures is increased by 1. When the number of transmission failures exceeds the specified threshold, the cached transmission data packet is cleared.

[0066] See also Figure 4 As shown, the system also includes a session management module and a disk space monitoring module.

[0067] The session management module is used to: monitor the status of the transmission channel, and when the transmission channel is interrupted, send a working signal to the parsing unit; when the transmission channel is restored after interruption, send a working signal to the computing unit.

[0068] The disk space monitoring module is used to monitor the disk usage c. When c rises to b and c≥b, data caching is prohibited; when c decreases to a and c≤a, data caching is allowed.

[0069] The embodiment of the present invention further provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It should be noted that the storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk or an optical disk.

[0070] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that runs on the processor, and the processor implements the above method when executing the computer program.

[0071] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer-readable storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0072] As is known to those of ordinary skill in the art, the term computer-readable storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0073] Exemplarily, the computer-readable storage medium may be an internal storage unit of the electronic device of the aforementioned embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.

[0074] The above are only specific implementations of the embodiments of the present invention, but the protection scope of the embodiments of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the embodiments of the present invention, and these modifications or replacements should be included in the protection scope of the embodiments of the present invention. Therefore, the protection scope of the embodiments of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for recovering streaming performance data in an interruption scenario, characterized in that: The method comprises the following steps: converting performance data into session information, the session information comprising a session performance code and its performance value; after the transmission channel is restored, removing the performance value in the performance code that does not reflect the change characteristics of the performance data by nonlinear discrete sampling.

2. The method for recovering streaming performance data in an interruption scenario according to claim 1, characterized in that: The performance value of the session performance code includes the current value, minimum value and maximum value at different times; the session performance code is defined to include 3 groups of performance values: all current values ​​are one group, all minimum values ​​are one group, and all maximum values ​​are one group; The process of removing the performance values ​​that do not reflect the change characteristics of the performance data in the performance code by nonlinear discrete sampling includes: in any group of performance values, removing the performance values ​​that do not reflect the change characteristics of the performance data in the performance code.

3. The method for recovering streaming performance data in an interruption scenario according to claim 2, characterized in that: The process of removing the performance values ​​that do not reflect the change characteristics of the performance data in the performance code by nonlinear discrete sampling includes: in each group of performance values, removing the performance values ​​that do not reflect the change characteristics of the performance data in the performance code.

4. The method for recovering streaming performance data in an interruption scenario according to claim 2 or 3, characterized in that: The process of removing the performance values ​​that do not reflect the change characteristics of the performance data in the performance code from a group of performance values ​​includes: arranging the performance values ​​to form a number of data points according to the continuity of the timestamps, and retaining the data points of the first and last nodes; For each non-first and last node data point A i , whose performance value is expressed as V i , the slope of the data point is denoted as K i , the tilt angle is denoted as θ, If K i *K i+1 <0, judge data point A i Does it correspond to the shock scenario? If so, remove data point A i， Otherwise, keep data point A i ; The criteria for judging the oscillation scenario are: the fluctuation error β of the performance value is pre-set, and |V i -V i-1 |≤β, and |V i+1 -V i |≤β, and |V i+2 -V i+1 |≤β, and K i+1 *K i+2 <0; if K i =K i+1 or Remove data point A i； in is the initial value of the fluctuation angle coefficient, Δt is the interruption duration of the session; if |θ i+1 -θ i |>ε+Δt / 60, retain data point A i .

5. The method for recovering streaming performance data in an interruption scenario according to claim 1, characterized in that: After removing the performance values ​​in the performance code that do not reflect the changing characteristics of the performance data, the following steps are also included: the performance data is formed into a transmission data packet of a specified format, and the transmission data packet includes a sampling type corresponding to the traditional adopted path, an object name and a total performance code corresponding to the traditional performance code string.

6. The method for recovering streaming performance data in an interruption scenario according to claim 5, characterized in that: The transmission data packet also includes several current performance values, several minimum performance values ​​and several maximum performance values ​​of the performance code total code.

7. The method for recovering streaming performance data in an interruption scenario according to claim 5 or 6, characterized in that: After the performance data is formed into a transmission data packet of a specified format, the following steps are also included: determining whether the transmission channel of the current session is normal, if so, sending the transmission data packet, otherwise caching the transmission data packet and adding 1 to the number of transmission failures; when the number of transmission failures exceeds a specified threshold, clearing the cached transmission data packet.

8. The method for recovering streaming performance data in an interruption scenario according to claim 1, characterized in that: The method also includes the following steps: monitoring the disk occupancy rate c, defining the first-level threshold as a, the second-level threshold as b, b>a; when c rises to b and c≥b, prohibiting data caching; when c decreases to a and c≤a, allowing data caching.

9. The method for recovering streaming performance data in an interruption scenario according to claim 1, characterized in that: The method also includes the following steps: storing the session information in a database, wherein the session information association table of the database includes a session information table and a performance code data table; the session information table includes a remote IP, a port and a session tag; the performance code data table includes an object name, a performance code, the performance value information of the performance code, and a session reference corresponding to the session tag.

10. A streaming performance data recovery system in an interruption scenario, characterized in that: The system is used to implement the method described in any one of claims 1 to 9.

Images