A data monitoring method, device, storage medium and equipment

Abstract

The application discloses a data monitoring method and device, a storage medium and equipment. Based on compressed forwarding data and second forwarding data, a forwarding data packet is constructed. For each target forwarding data, the target forwarding data is disassembled to obtain disassembled target forwarding data. The disassembled target forwarding data is associated with cloud resource information to obtain associated target forwarding data. The associated target forwarding data is associated with an evaluation index to obtain an alarm strategy. Compared with the prior art, the problem of difficult discovery of fault signs in the existing operation and maintenance scene of a cloud network is solved, and the fault troubleshooting efficiency of cloud network operation and maintenance is improved.

Description

Technical Field

[0001] This application relates to the field of data monitoring, and in particular to a data monitoring method, apparatus, storage medium and device. Background Technology

[0002] In network operations and maintenance scenarios, due to the massive traffic and packet volume of forwarded data, and the high requirements of upper-layer applications for forwarding performance, it is almost impossible to retain all forwarded data packet by packet for a long time without affecting application forwarding.

[0003] In traditional physical networks, hardware optical splitters combined with network probe technology can collect data from some nodes, mirroring and storing the full data flowing through the corresponding nodes. However, this method can only collect data from switches and forwarded data, and cannot collect data from virtualized resources. Therefore, when a fault occurs, maintenance personnel cannot locate the cause of the fault based on the existing information, and cannot resolve the fault, resulting in low efficiency in network maintenance and troubleshooting. Summary of the Invention

[0004] This application provides a data monitoring method, apparatus, storage medium, and device, with the aim of improving the efficiency of cloud network operation and maintenance troubleshooting.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] A data monitoring method, comprising:

[0007] A forwarding data packet is constructed based on the compressed forwarding data and the second forwarding data; the compressed forwarding data is obtained by preprocessing the first forwarding data; the first forwarding data is obtained from a first preset device in advance; the second forwarding data is data pre-collected from a second preset device; the forwarding data packet includes at least each target forwarding data.

[0008] For each of the target forwarding data, the target forwarding data is decomposed to obtain the decomposed target forwarding data;

[0009] The disassembled target forwarding data is associated with cloud resource information to obtain associated target forwarding data;

[0010] The associated target forwarding data is then linked with evaluation metrics to obtain an alarm strategy.

[0011] Optionally, the process of preprocessing the first forwarding data to obtain the compressed forwarding data includes:

[0012] The first forwarded data is truncated to obtain truncated forwarded data;

[0013] The truncated forwarded data is fragmented to obtain individual fragments.

[0014] For each of the data fragments, the data fragments are compressed to obtain the compressed forwarding data.

[0015] Optionally, the process of pre-obtaining the first forwarding data from a first preset device includes:

[0016] The first forwarding data of the first preset device is collected using the collection points preset on the first preset device.

[0017] Optional, also includes:

[0018] The forwarded data packets are stored in the database.

[0019] A data monitoring device, comprising:

[0020] A construction unit is used to construct a forwarding data packet based on compressed forwarding data and second forwarding data; the compressed forwarding data is obtained by preprocessing the first forwarding data; the first forwarding data is obtained from a first preset device in advance; the second forwarding data is data pre-collected from a second preset device; the forwarding data packet includes at least each target forwarding data.

[0021] The disassembly unit is used to disassemble the target forwarding data for each target forwarding data to obtain the disassembled target forwarding data.

[0022] The first association unit is used to associate the disassembled target forwarding data with cloud resource information to obtain the associated target forwarding data.

[0023] The second association unit is used to associate the associated target forwarding data with the evaluation indicators to obtain an alarm strategy.

[0024] Optionally, the building unit is specifically used for:

[0025] The first forwarded data is truncated to obtain truncated forwarded data;

[0026] The truncated forwarded data is fragmented to obtain individual fragments.

[0027] For each of the data fragments, the data fragments are compressed to obtain the compressed forwarding data.

[0028] Optionally, the building unit is specifically used for:

[0029] The first forwarding data of the first preset device is collected using the collection points preset on the first preset device.

[0030] Optional, also includes:

[0031] The forwarded data packets are stored in the database.

[0032] A computer-readable storage medium includes a stored program, wherein the program is executed by a processor to perform the data monitoring method.

[0033] A data monitoring device includes: a processor, a memory, and a bus; the processor and the memory are connected via the bus.

[0034] The memory is used to store a program, and the processor is used to run the program, wherein the program is executed by the processor to perform the data monitoring method.

[0035] The technical solution provided in this application constructs forwarding data packets based on compressed forwarding data and second forwarding data; the compressed forwarding data is preprocessed from the first forwarding data; for each target forwarding data, the target forwarding data is decomposed to obtain decomposed target forwarding data; the decomposed target forwarding data is associated with cloud resource information to obtain associated target forwarding data; the associated target forwarding data is associated with evaluation indicators to obtain alarm strategies. Compared with existing technologies, this solution solves the problem of difficulty in detecting fault symptoms in existing cloud network operation and maintenance scenarios, and improves the efficiency of cloud network operation and maintenance troubleshooting. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1a A flowchart illustrating a data monitoring method provided in this application embodiment;

[0038] Figure 1b A flowchart illustrating a data monitoring method provided in this application embodiment;

[0039] Figure 2 A flowchart illustrating another data monitoring method provided in this application embodiment;

[0040] Figure 3This is a schematic diagram of the architecture of a data monitoring device provided in an embodiment of this application;

[0041] Figure 4 This is a schematic diagram of the architecture of a data monitoring device provided in an embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0044] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0045] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0046] like Figure 1a , Figure 1b The diagram shown is a flowchart of a data monitoring method provided in this embodiment, applied to a cloud network traffic observation system, including:

[0047] S101: Collect the first forwarding data of the first preset device using the collection points preset on the first preset device.

[0048] When executing S101, S105 can be executed concurrently.

[0049] The first preset device includes, but is not limited to, cloud servers and host machines, and forwards network data packets (such as network data packets of cloud servers and host machines) to the corresponding nodes of the data indication.

[0050] Optionally, the first forwarding data of the first preset device can be collected by using the extended Berkeley Packet Filter (eBPF).

[0051] It should be noted that the specific implementation method of collecting the first forwarding data of the first preset device through eBPF technology is common knowledge known to those in the field, and will not be elaborated here.

[0052] S102: Truncate the first forwarded data to obtain the truncated forwarded data.

[0053] The specific implementation method for truncating the first forwarded data is common knowledge known to those in the field.

[0054] It should be noted that, since there is invalid data in the forwarded data (i.e., it occupies a large amount of space but does not affect subsequent data analysis), the first forwarded data needs to be truncated to remove the invalid data.

[0055] S103: Perform fragmentation on the truncated forwarded data to obtain individual fragment data.

[0056] Optionally, a fragmentation algorithm can be used to fragment the truncated forwarded data.

[0057] It should be noted that the specific implementation of using the fragmentation algorithm to fragment the truncated forwarded data is common knowledge to those in the field, and will not be elaborated here.

[0058] S104: For each data fragment, compress the data fragment to obtain compressed forwarding data.

[0059] Optionally, data compression techniques can be used to compress the fragmented data, including but not limited to: transform coding and predictive coding.

[0060] It should be noted that the specific implementation of data compression technology to compress fragmented data is common knowledge to those in the field, and will not be elaborated here.

[0061] It is important to emphasize that the purpose of compressing fragmented data is to reduce the amount of data and thus reduce storage space without losing useful information.

[0062] S105: Collect the second forwarding data from the second preset device.

[0063] The second preset device includes, but is not limited to, a cloud gateway server.

[0064] Optionally, the second forwarding data of the second preset device can be collected using a hardware splitter or switch mirroring.

[0065] It should be noted that the second forwarding data of the second preset device is collected by using a hardware splitter or switch mirroring. Since the second forwarding data is collected by bypass, no preprocessing is required.

[0066] S106: Construct a forwarding data packet based on the compressed forwarding data and the second forwarding data.

[0067] The forwarded data packet includes at least the forwarded data for each target.

[0068] S107: Store the forwarded data packets in the database.

[0069] When executing S107, S108 can be executed concurrently.

[0070] It should be noted that the forwarded data packets are stored in the database to form the conditions for backtracking historical forwarding details. If real-time analysis and processing are not possible or if there are occasional failure scenarios, the historical forwarded data packet details can be further analyzed based on the corresponding time period and the path of the forwarded data to obtain a solution.

[0071] S108: For each target forwarding data, the target forwarding data is decomposed to obtain the decomposed target forwarding data.

[0072] The target forwarding data includes, but is not limited to, the following: protocol header, outer message, inner message, and request.

[0073] It should be noted that the specific implementation process of disassembling the target forwarded data is as follows: the header of the target forwarded data is identified to obtain the byte length; the protocol type is determined based on the byte length; and relevant information is determined based on the protocol type. Among these relevant information, at least the outer packet and the inner packet are included.

[0074] It is important to emphasize that disassembling the target forwarding data essentially eliminates the analysis barriers caused by the overlapping of IP addresses and the frequent switching of various non-standard network protocols in the cloud network. This helps to form a full-path cloud network traffic topology, making it easier to quickly locate the fault location when a fault occurs.

[0075] S109: Associate the disassembled target forwarding data with cloud resource information to obtain the associated target forwarding data.

[0076] Cloud resource information includes, but is not limited to: cloud servers, cloud storage, and cloud databases.

[0077] It should be noted that the specific implementation process of associating the disassembled target forwarding data with cloud resource information is as follows: the IP information shown in the disassembled target forwarding data is associated with the cloud resource information to obtain the associated target forwarding data.

[0078] It is important to emphasize that associating the disassembled target forwarding data with cloud resource information helps to form a cloud network knowledge graph, making cloud network operation and maintenance scenarios more closely aligned with actual cloud applications, so as to quickly locate the scope of the fault when it occurs.

[0079] S110: Associate the associated target forwarding data with the evaluation metrics to obtain the alarm policy.

[0080] The evaluation metrics include, but are not limited to: retransmission rate, packet loss rate, and latency.

[0081] It should be noted that the specific implementation method of associating the target forwarding data with the evaluation indicators is common knowledge to those in the field, and will not be elaborated here.

[0082] It is important to emphasize that by associating the target forwarding data with evaluation metrics to obtain alarm strategies, the forwarding quality of the target forwarding data can be observed in real time, which is beneficial for backtracking the forwarding quality of the target forwarding data within historical time periods.

[0083] Optionally, alarm thresholds can be set according to the alarm policy, and alarm information can be sent to the user when the alarm threshold is exceeded.

[0084] In summary, based on the compressed forwarding data and the second forwarding data, forwarding data packets are constructed. For each target forwarding data, the target forwarding data is decomposed to obtain the decomposed target forwarding data. The decomposed target forwarding data is associated with cloud resource information to obtain the associated target forwarding data. The associated target forwarding data is associated with evaluation indicators to obtain alarm strategies. Compared with existing technologies, this solves the problem of difficulty in detecting fault symptoms in existing cloud network operation and maintenance scenarios, and improves the efficiency of cloud network operation and maintenance troubleshooting.

[0085] like Figure 2 The flowchart shown is another data monitoring method provided in this application embodiment, including:

[0086] S201: Construct a forwarding data packet based on the compressed forwarding data and the second forwarding data.

[0087] The compressed forwarding data is obtained by preprocessing the first forwarding data; the first forwarding data is obtained from the first preset device; the second forwarding data is data collected from the second preset device in advance; the forwarding data packet includes at least the forwarding data of each target.

[0088] S202: For each target forwarding data, the target forwarding data is decomposed to obtain the decomposed target forwarding data.

[0089] S203: Associate the disassembled target forwarding data with cloud resource information to obtain the associated target forwarding data.

[0090] S204: Associate the associated target forwarding data with the evaluation metrics to obtain the alarm policy.

[0091] In summary, based on the compressed forwarding data and the second forwarding data, forwarding data packets are constructed. For each target forwarding data, the target forwarding data is decomposed to obtain the decomposed target forwarding data. The decomposed target forwarding data is associated with cloud resource information to obtain the associated target forwarding data. The associated target forwarding data is associated with evaluation indicators to obtain alarm strategies. Compared with existing technologies, this solves the problem of difficulty in detecting fault symptoms in existing cloud network operation and maintenance scenarios, and improves the efficiency of cloud network operation and maintenance troubleshooting.

[0092] like Figure 3 The diagram shown is a schematic representation of the architecture of a data monitoring device provided in an embodiment of this application, comprising:

[0093] The construction unit 100 is used to construct a forwarding data packet based on the compressed forwarding data and the second forwarding data; the compressed forwarding data is obtained by preprocessing the first forwarding data; the first forwarding data is obtained from the first preset device in advance; the second forwarding data is data collected from the second preset device in advance; the forwarding data packet includes at least the forwarding data of each target.

[0094] The construction unit 100 is specifically used for: truncating the first forwarding data to obtain truncated forwarding data; fragmenting the truncated forwarding data to obtain individual fragments; and compressing each fragment to obtain compressed forwarding data.

[0095] The construction unit 100 is specifically used to: collect the first forwarding data of the first preset device using the collection points preset on the first preset device.

[0096] The disassembly unit 200 is used to disassemble the target forwarding data for each target to obtain the disassembled target forwarding data.

[0097] The first association unit 300 is used to associate the disassembled target forwarding data with cloud resource information to obtain the associated target forwarding data.

[0098] The second association unit 400 is used to associate the associated target forwarding data with the evaluation indicators to obtain the alarm strategy.

[0099] The second association unit 400 is specifically used to store forwarded data packets in the database.

[0100] In summary, based on the compressed forwarding data and the second forwarding data, forwarding data packets are constructed. For each target forwarding data, the target forwarding data is decomposed to obtain the decomposed target forwarding data. The decomposed target forwarding data is associated with cloud resource information to obtain the associated target forwarding data. The associated target forwarding data is associated with evaluation indicators to obtain alarm strategies. Compared with existing technologies, this solves the problem of difficulty in detecting fault symptoms in existing cloud network operation and maintenance scenarios, and improves the efficiency of cloud network operation and maintenance troubleshooting.

[0101] This application also provides a computer-readable storage medium including a stored program, wherein the program executes the data monitoring method provided in this application.

[0102] like Figure 4 As shown, this application also provides a data monitoring device, including: a processor 401, a memory 402, and a bus 403. The processor 401 and the memory 402 are connected via the bus 403. The memory 402 is used to store programs, and the processor 401 is used to run the programs. When the programs run, they execute the data monitoring method provided in this application, including the following steps:

[0103] A forwarding data packet is constructed based on the compressed forwarding data and the second forwarding data; the compressed forwarding data is obtained by preprocessing the first forwarding data; the first forwarding data is obtained from a first preset device in advance; the second forwarding data is data pre-collected from a second preset device; the forwarding data packet includes at least each target forwarding data.

[0104] For each of the target forwarding data, the target forwarding data is decomposed to obtain the decomposed target forwarding data;

[0105] The disassembled target forwarding data is associated with cloud resource information to obtain associated target forwarding data;

[0106] The associated target forwarding data is then linked with evaluation metrics to obtain an alarm strategy.

[0107] Optionally, the process of preprocessing the first forwarding data to obtain the compressed forwarding data includes:

[0108] The first forwarded data is truncated to obtain truncated forwarded data;

[0109] The truncated forwarded data is fragmented to obtain individual fragments.

[0110] For each of the data fragments, the data fragments are compressed to obtain the compressed forwarding data.

[0111] Optionally, the process of pre-obtaining the first forwarding data from a first preset device includes:

[0112] The first forwarding data of the first preset device is collected using the collection points preset on the first preset device.

[0113] Optional, also includes:

[0114] The forwarded data packets are stored in the database.

[0115] If the functions described in the embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in a computing device readable storage medium. Based on this understanding, the parts of the embodiments of this application that contribute to the prior art or the technical solutions can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computing device (which may be a personal computer, server, mobile computing device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0116] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0117] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data monitoring method, characterized in that, Applications include cloud network traffic monitoring systems, including: A forwarding data packet is constructed based on the compressed forwarding data and the second forwarding data; the compressed forwarding data is obtained by preprocessing the first forwarding data; the first forwarding data is obtained from the first preset device in advance; the second forwarding data is data collected from the second preset device in advance; the forwarding data packet includes at least each target forwarding data; the target forwarding data is the forwarding data in the forwarding data packet that originates from the compressed forwarding data and the second forwarding data; For each of the target forwarding data, the target forwarding data is decomposed to obtain the decomposed target forwarding data; The disassembled target forwarding data is associated with cloud resource information to obtain associated target forwarding data; The associated target forwarding data is correlated with evaluation metrics to obtain an alarm strategy; the forwarding quality of the target forwarding data is observed through the alarm strategy.

2. The method according to claim 1, characterized in that, The process of preprocessing the first forwarded data to obtain the compressed forwarded data includes: The first forwarded data is truncated to obtain truncated forwarded data; The truncated forwarded data is fragmented to obtain individual fragments. For each of the data fragments, the data fragments are compressed to obtain the compressed forwarding data.

3. The method according to claim 1, characterized in that, The process of obtaining the first forwarding data from the first preset device includes: The first forwarding data of the first preset device is collected using the collection points preset on the first preset device.

4. The method according to claim 1, characterized in that, Also includes: The forwarded data packets are stored in the database.

5. A data monitoring device, characterized in that, Applications include cloud network traffic monitoring systems, including: A construction unit is used to construct a forwarding data packet based on compressed forwarding data and second forwarding data; the compressed forwarding data is obtained by preprocessing the first forwarding data; the first forwarding data is obtained from a first preset device in advance; the second forwarding data is data pre-collected from the second preset device; the forwarding data packet includes at least each target forwarding data; the target forwarding data is forwarding data in the forwarding data packet derived from the compressed forwarding data and the second forwarding data; The disassembly unit is used to disassemble the target forwarding data for each target forwarding data to obtain the disassembled target forwarding data. The first association unit is used to associate the disassembled target forwarding data with cloud resource information to obtain the associated target forwarding data. The second association unit is used to associate the associated target forwarding data with evaluation indicators to obtain an alarm strategy; and to observe the forwarding quality of the target forwarding data through the alarm strategy.

6. The apparatus according to claim 5, characterized in that, The building unit is specifically used for: The first forwarded data is truncated to obtain truncated forwarded data; The truncated forwarded data is fragmented to obtain individual fragments. For each of the data fragments, the data fragments are compressed to obtain the compressed forwarding data.

7. The apparatus according to claim 5, characterized in that, The building unit is specifically used for: The first forwarding data of the first preset device is collected using the collection points preset on the first preset device.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed by a processor, performs the data monitoring method according to any one of claims 1-4.

9. A data monitoring device, characterized in that, include: Processor, memory, and bus; The processor and the memory are connected via the bus; The memory is used to store a program, and the processor is used to run the program, wherein the program is executed by the processor to perform the data monitoring method according to any one of claims 1-4.

Images