Monitoring data processing method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a monitoring data processing method and device, electronic equipment and a readable storage medium, and relates to the technical field of network communication. The method comprises the following steps: acquiring a target network transmission data packet of a predefined network port in the distributed system; analyzing the target network transmission data packet by using the network transmission protocol analysis program to obtain an analysis result; determining the transmission type of the target network transmission data packet based on the source address and the destination address in the analysis result; based on the transmission type of the target network transmission data packet, target transmission identification information corresponding to the target network transmission data packet is determined, and the target transmission identification information is used for identifying the transmission process of the target network transmission data packet; and sending the target transmission identification information to a distributed message system for persistent storage. Through the method and the device, the technical problem that full-link monitoring cannot be carried out on the network transmission data packet in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of network communication technology, and in particular to a monitoring data processing method, device, electronic device and readable storage medium. Background Art

[0002] At present, in the field of cloud computing and distributed technology, full-link monitoring technology has developed into a key tool to ensure system stability and performance. Traditional full-link monitoring methods usually rely on embedding specific monitoring components or codes in specific application layer services to collect link data. These components can collect link information with low performance overhead without adjusting the business code. The collected data is uniformly managed and distributed through servers or other message queue systems to support a wider range of monitoring needs. However, this method encounters limitations in practical applications, especially when dealing with non-mainstream applications or external components, which are usually not directly monitored by mainstream monitoring frameworks, resulting in breakpoints in the monitoring link. There is a technical problem that it is impossible to perform full-link monitoring of network transmission data packets.

[0003] With regard to the technical problem in related technologies that it is impossible to perform full-link monitoring of network transmission data packets, no effective solution has been proposed so far. Summary of the invention

[0004] The main purpose of the present application is to provide a monitoring data processing method, device, electronic device and readable storage medium to solve the technical problem in the related art that it is impossible to perform full-link monitoring of network transmission data packets.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for processing monitoring data is provided. The method includes: collecting target network transmission data packets of pre-defined network ports in a distributed system; parsing the target network transmission data packets using a network transmission protocol parsing program to obtain a parsing result, wherein the parsing result at least includes the source address and destination address of the target network transmission data packet; determining the transmission type of the target network transmission data packet based on the source address and destination address in the parsing result; determining the target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet, wherein the target transmission identification information is used to identify the transmission process of the target network transmission data packet; and sending the target transmission identification information to a distributed message system for persistent storage.

[0006] Optionally, based on the source address and destination address in the parsing result, the transmission type of the target network transmission data packet is determined, including: in response to the source address being a component address outside the distributed system and the destination address being a service node address within the distributed system, determining that the transmission type of the target network transmission data packet is a receiving type; in response to the source address being a service node address within the distributed system, determining that the transmission type of the target network transmission data packet is a sending type.

[0007] Optionally, based on the transmission type of the target network transmission data packet, the target transmission identification information corresponding to the target network transmission data packet is determined, including: in response to the transmission type of the target network transmission data packet being a receiving type and the target transmission identification information not existing in the target network transmission data packet, generating new transmission identification information; determining the new transmission identification information as the target transmission identification information corresponding to the target network transmission data packet; the method also includes: storing the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0008] Optionally, the monitoring data processing method also includes: in response to the transmission type of the target network transmission data packet being a receiving type and the target network transmission data packet containing target transmission identification information, storing the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0009] Optionally, based on the transmission type of the target network transmission data packet, target transmission identification information corresponding to the target network transmission data packet is determined, including: in response to the transmission type of the target network transmission data packet being a sending type, determining the similarity between the target network transmission data packet and multiple network transmission data packets stored in the memory of the distributed system; based on the similarity, determining a source network transmission data packet associated with the target network transmission data packet from the multiple network transmission data packets; based on the transmission identification information corresponding to the source network transmission data packet, determining the target transmission identification information corresponding to the target network transmission data packet.

[0010] Optionally, the monitoring data processing method further includes: determining a transmission link of the target network transmission data packet based on transmission identification information corresponding to the source network transmission data packet and target transmission identification information corresponding to the target network transmission data packet.

[0011] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a monitoring data processing device is provided. The device includes: a collection unit, which is used to collect target network transmission data packets of a pre-defined network port in a distributed system; a parsing unit, which is used to parse the target network transmission data packets using a network transmission protocol parsing program to obtain a parsing result, wherein the parsing result at least includes the source address and destination address of the target network transmission data packet; a first determination unit, which is used to determine the transmission type of the target network transmission data packet based on the source address and destination address in the parsing result; a second determination unit, which is used to determine the target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet, wherein the target transmission identification information is used to identify the transmission process of the target network transmission data packet; a storage unit, which is used to send the target transmission identification information to a distributed message system for persistent storage.

[0012] Optionally, the first determination unit is also used to: in response to the source address being a component address outside the distributed system and the destination address being a service node address within the distributed system, determine that the transmission type of the network transmission data is a receiving type; in response to the source address being a service node address within the distributed system, determine that the transmission type of the network transmission data is a sending type.

[0013] Optionally, the second determination unit is also used to: generate new transmission identification information in response to the transmission type of the target network transmission data packet being a receiving type, and the target network transmission data packet does not have target transmission identification information; determine the new transmission identification information as the target transmission identification information corresponding to the target network transmission data packet; the method also includes: storing the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0014] Optionally, the monitoring data processing device is also used to: in response to the transmission type of the target network transmission data packet being a receiving type and the target network transmission data packet containing target transmission identification information, store the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0015] Optionally, the second determination unit is also used to: in response to the transmission type of the target network transmission data packet being a sending type, determine the similarity between the target network transmission data packet and multiple network transmission data packets stored in the memory of the distributed system based on a similarity algorithm; determine a source network transmission data packet associated with the target network transmission data packet from multiple network transmission data packets based on the similarity; and determine the target transmission identification information corresponding to the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet.

[0016] Optionally, the monitoring data processing device is also used to determine the transmission link of the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet and the target transmission identification information corresponding to the target network transmission data packet.

[0017] To achieve the above object, according to another aspect of the present application, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above method embodiments.

[0018] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the steps in any one of the above-mentioned method embodiments.

[0019] In order to achieve the above objective, according to another aspect of the present application, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0020] In an embodiment of the present application, a target network transmission data packet of a predefined network port in a distributed system is collected; the target network transmission data packet is parsed by a network transmission protocol parsing program to obtain a parsing result, wherein the parsing result at least includes the source address and the destination address of the target network transmission data packet; based on the source address and the destination address in the parsing result, the transmission type of the target network transmission data packet is determined; based on the transmission type of the target network transmission data packet, the target transmission identification information corresponding to the target network transmission data packet is determined, wherein the target transmission identification information is used to identify the transmission process of the target network transmission data packet; the target transmission identification information is sent to a distributed message system for persistent storage. That is to say, in an embodiment of the present application, after the target network transmission data packet is parsed by a network transmission protocol parsing program to obtain a parsing result, the source address and the destination address in the target network transmission data packet included in the parsing result can be used to determine the transmission type of the target network transmission data packet, and then determine the target transmission identification information corresponding to the target network transmission data packet, and the transmission process of the target network transmission data packet can be monitored according to the target transmission identification information, thereby realizing full-link monitoring of the target network transmission data packet when it is transmitted in a distributed system, thereby solving the technical problem that the network transmission data packet cannot be fully monitored in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 is a hardware structure block diagram of a computer terminal for implementing a method for processing monitoring data according to an embodiment of the present application;

[0023] Figure 2 is a flow chart of a method for processing monitoring data according to an embodiment of the present application;

[0024] Figure 3 This is an architecture diagram of full-link monitoring in a distributed system according to an embodiment of the present application;

[0025] Figure 4 is a schematic diagram of a monitoring data processing device according to an embodiment of the present application;

[0026] Figure 5 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] It should be noted that the collected information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with relevant laws, regulations and standards, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation entrances for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered.

[0030] Example 1

[0031] According to an embodiment of the present application, an embodiment of a method for processing monitoring data is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for processing monitoring data according to an embodiment of the present application. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (shown in the figure as 102a, 102b, ..., 102n) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, abbreviated as MCU) or a programmable logic device (Field-Programmable Gate Array, abbreviated as FPGA)), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (Input / Output, abbreviated as I / O), a universal serial bus (Universal Serial Bus, abbreviated as USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0033] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0034] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the monitoring data processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned monitoring data processing method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0035] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0036] The display may be, for example, a touch screen liquid crystal display (LCD), which enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0037] Under the above operating environment, this application provides Figure 2 The monitoring data processing method shown. Figure 2 It is a flowchart of the method for processing monitoring data according to Example 1 of the present application.

[0038] Step S201 , collecting target network transmission data packets of a predefined network port in a distributed system.

[0039] In the technical solution provided in the above step S201 of the present application, in a distributed system, communication between service nodes and interaction with external components are usually carried out through a network, and data packets are transmitted in the network. In order to achieve full-link monitoring, a trace component can be designed on each key service node in the distributed system, or a trace component can be designed in the critical path of the network to monitor and capture network data packets of the target network port. These ports are usually common ports for communication between services, or ports specially defined when interacting with external components.

[0040] In this embodiment, the target network transmission data packets refer to the data packets that the Trace component pays special attention to in the distributed system monitoring. These data packets include but are not limited to the data of communication between service nodes, the data of external component requests and responses, and any key data transmission that may affect the operation and performance of the system.

[0041] Optionally, in order to collect all relevant network transmission data, the Trace component needs to listen to the network ports pre-defined in the system architecture. These ports may include, but are not limited to: ports for internal service communication, and target network ports for interaction with external components or systems. These ports may be specially set for full-link monitoring so that the Trace component can identify and process requests and responses from the outside.

[0042] Optionally, the target network transmission data packets of the pre-defined network ports in the distributed system are collected to ensure that the Trace component can capture all important data flows and provide a comprehensive data foundation for full-link monitoring. Through the subsequent processing of the collected data, true end-to-end full-link monitoring can be achieved, improving the efficiency and effectiveness of system monitoring.

[0043] Step S202, using a network transmission protocol analysis program to analyze the target network transmission data packet to obtain an analysis result.

[0044] In the technical solution provided in the above step S202 of the present application, the parsing result includes at least the source address and destination address of the target network transmission data packet. The source address and the destination address are two basic elements of network communication. The source address indicates the sender of the data packet, while the destination address indicates the receiver of the data packet.

[0045] In this embodiment, the network transmission protocol parser is software specially designed to understand and parse network data packets. A parser capable of processing multiple application layer protocols is used, which ensures that the monitoring system can identify and parse the communication protocols widely used in the distributed system, thereby obtaining detailed information of the data packets.

[0046] Optionally, the collected target network transmission data packets, that is, the data captured through the pre-defined network port, will be sent to the network transmission protocol parser for parsing. The parsing process involves reading the header information of the data packet, including but not limited to the source address, destination address, port number, protocol type, etc., as well as the payload content of the data packet. The parsing of the payload content is particularly important because the payload content contains the specific information of the request and the data of the response, and is the key data source for full-link monitoring.

[0047] Optionally, after parsing the target network transmission data packet through the network transmission protocol parsing program, a parsing result can be obtained, wherein the parsing result at least includes the source address and destination address of the target network transmission data packet. In addition, the parsing result can also include the specific content of the transmission content.

[0048] In this step, the target network transmission data packet is parsed using the network transmission protocol parser to obtain key information including the source address and destination address, which can provide basic data for subsequent full-link tracking and analysis, ensuring the accuracy and effectiveness of monitoring the target network data packet.

[0049] Step S203, based on the source address and the destination address in the parsing result, determine the transmission type of the target network transmission data packet.

[0050] In the technical solution provided in the above step S203 of the present application, after obtaining the parsing result, the transmission type of the target network transmission data packet can be determined according to the source address and destination address in the parsing result, wherein the transmission type includes: sending type and receiving type.

[0051] In this embodiment, if the source address of the data packet corresponds to a service node within the distributed system, then the data packet can be determined to be of the sending type. This means that a service node is sending data to another service node or an external component. For such data packets, an intelligent algorithm such as a similarity algorithm can be used to determine its similarity with previously received data packets to determine or generate Trace information, thereby tracking the source and destination of the request.

[0052] Optionally, if the destination address of the data packet corresponds to a service node inside the distributed system, then the data packet can be determined to be of the receive type. This means that the data packet is entering the system, possibly from another service node or an external component. For a receive type data packet, if the data packet does not contain trace information, this usually means that the data packet comes from outside or enters the monitoring link for the first time. The system will generate a new trace information and store it in the thread context for subsequent tracking. If the data packet already contains trace information, the system will record this information for subsequent comparison with other send type data packets to determine the relationship between the data packets.

[0053] In this step, the transmission type of the data packet transmitted by the target network is determined by parsing the source address and the destination address. This can intelligently distinguish whether the data packet is flowing within the system or coming from / going to the outside. This is important for establishing full-link monitoring and ensuring the integrity and consistency of each service call link.

[0054] Step S204: determining target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet.

[0055] In the technical solution provided in the above step S204 of the present application, the transmission type of the target network transmission data packet is identified. If it is a received data packet, it can be checked whether the target network data packet already has the target transmission identification information, wherein the target transmission identification information can also be called Trace information. If the target network transmission data packet originates from outside the system, that is, a data packet sent from an external component or other external system, it generally does not carry the pre-existing target transmission identification information. At this time, the Trace component will generate a new target transmission identification information and store it in the thread context so that this information can be referenced when a subsequent request data packet related to the data packet is sent. If the data packet already carries the target transmission identification information, this indicates that the data packet is sent from inside the system or from an external system that has been identified by Trace. The Trace component will record this information to provide a basis for the subsequent data packet comparison and the association of the target transmission identification information.

[0056] In this embodiment, if it is a data packet sent out, the Trace component will use a similarity algorithm to analyze the content of the data packet to determine its source or the related received data packet, so as to obtain or generate the correct target transmission identification information. Target transmission identification information, usually in the form of target transmission identification information, is a key identifier in full-link monitoring, which is used to identify and track the complete transmission process of the data packet in the distributed system. For the data packet sent out, the Trace component will use the similarity algorithm to determine its similarity with the recorded data packet based on the content, source address, destination address and other information of the data packet, and find the possible source data packet, so as to determine or generate the correct target transmission identification information. For the received data packet, if the target transmission identification information is already included in the data packet, the Trace component will use it directly; if not, the component will generate a new target transmission identification information.

[0057] Step S205: Send the target transmission identification information to the distributed messaging system for persistent storage.

[0058] In the technical solution provided in the above step S205 of the present application, after determining the target transmission identification information, the target identification information and other key information of the target network transmission data packet (such as source address, destination address, timestamp, etc.) can be sent to the distributed message system for persistent storage. In the subsequent monitoring and analysis, the operation and maintenance personnel can retrieve the relevant data packet records from the distributed message system through the Trace information, and then build a complete request-response chain, track the data flow in the system, locate the fault point, analyze the performance bottleneck, and optimize the system performance.

[0059] In this step, the target transmission identification information is persistently stored in the distributed messaging system and can be retrieved and used by multiple backend services or analysis tools through the subscription mechanism. This information can be aggregated and analyzed to identify performance bottlenecks, abnormal behaviors, and system failures, helping operation and maintenance personnel to quickly locate problems and take appropriate measures.

[0060] In the above steps S201 to S205, after the target network transmission data packet is parsed by the network transmission protocol parsing program and the parsing result is obtained, the source address and destination address in the target network transmission data packet included in the parsing result can be used to determine the transmission type of the target network transmission data packet, and then determine the target transmission identification information corresponding to the target network transmission data packet. According to the target transmission identification information, the transmission process of the target network transmission data packet can be monitored, thereby realizing full-link monitoring of the target network transmission data packet when it is transmitted in a distributed system, thereby solving the technical problem in the related technology that the network transmission data packet cannot be fully monitored.

[0061] The embodiment of the present invention is described in detail below in combination with the above steps.

[0062] As an optional implementation, step S203, based on the source address and destination address in the parsing result, determines the transmission type of the target network transmission data packet, including: in response to the source address being a component address outside the distributed system and the destination address being a service node address within the distributed system, determining that the transmission type of the network transmission data is a receiving type; in response to the source address being a service node address within the distributed system, determining that the transmission type of the network transmission data is a sending type.

[0063] In this embodiment, when the source address of the parsed target network transmission data packet corresponds to a component address outside the distributed system, and the destination address corresponds to a service node address inside the system, the system determines the transmission type of the data packet as a receive type. This means that the data packet is being sent from the external environment to a service node within the distributed system. It may be a request initiated by a user through a web page (Web) or an application (Application, referred to as APP), or it may be data sent to the system by other external systems or components. For a receive type data packet, the system needs to create a new Trace information to subsequently track the processing path of the request. This Trace information is stored in the thread context for comparison and association with subsequent send type data packets.

[0064] Optionally, if the source address of the target network transmission data packet corresponds to the service node address inside the system, and the destination address may be another service node address or the address of an external component, the system determines the transmission type of the data packet as the send type. This means that the data packet is being sent from one service node to another service node or the external environment, which may be a response to a request or a call between services. For data packets of the send type, the system uses intelligent algorithms such as similarity algorithms to determine the similarity of the content of the data packet with the previously received data packet, so as to determine or generate the correct Trace information and ensure the integrity and consistency of the request-response chain.

[0065] In this step, by determining the transmission type of the data packet based on the source address and the destination address, the flow direction of the data packet can be clearly distinguished, ensuring that the transmission process of each data packet is accurately recorded and tracked.

[0066] As an optional implementation, step S204, based on the transmission type of the target network transmission data packet, determines the target transmission identification information corresponding to the target network transmission data packet, including: in response to the transmission type of the target network transmission data packet being a receiving type, and the target transmission identification information does not exist in the target network transmission data packet, generates new transmission identification information; determines the new transmission identification information as the target transmission identification information corresponding to the target network transmission data packet; the monitoring data processing method also includes: storing the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0067] In this embodiment, when the transmission type of the target network transmission data packet monitored by the Trace component is determined to be a receiving type, and the data packet does not carry the target transmission identification information (e.g., Trace ID), this usually means that the data packet originates from outside the system or the source of the data packet has not yet been identified by the monitoring system. At this time, the Trace component will automatically generate a new transmission identification information (e.g., a new Trace ID) for the data packet to ensure that every data packet entering the system can be monitored and tracked. The generated transmission identification information will be associated with the data packet, marking it as the starting point of a new request or service call.

[0068] Optionally, once new transmission identification information is generated, or if transmission identification information already exists in a data packet, the Trace component will identify this information as the target transmission identification information corresponding to the data packet. The target transmission identification information is the core of full-link monitoring, which is used to uniquely identify the complete link of a request or service call, ensuring that every forwarding, processing, and response of a data packet in the system can be tracked and associated.

[0069] Optionally, after generating or determining the target transmission identification information, the Trace component temporarily stores the target network transmission data packet and the corresponding transmission identification information in the memory of the distributed system. The purpose of memory storage is to achieve fast access and processing of data, especially when it needs to be associated with subsequently sent data packets, the Trace information stored in the memory can be quickly retrieved and used to achieve fast comparison of data packets and full-link tracking.

[0070] In this step, in response to the transmission type of the target network transmission data packet being a receiving type and the target network transmission data packet not having target transmission identification information, new transmission identification information is generated; the new transmission identification information is determined as the target transmission identification information corresponding to the target network transmission data packet, and then the target transmission identification information is stored in the memory of the distributed system, which can ensure the traceability and rapid processing of the target network transmission data packet in full-link monitoring.

[0071] As an optional implementation, the monitoring data processing method also includes: in response to the transmission type of the target network transmission data packet being a receiving type, and the target network transmission data packet containing target transmission identification information, storing the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0072] In this embodiment, when the transmission type of the target network transmission data packet is the receiving type, it means that the target network transmission data packet enters the current service node from an external or another internal service node. If the data packet carries target transmission identification information, this generally indicates that the data packet arrives as part of a service call chain, and in the entire call chain, the transmission process of the data packet has been identified (for example, through Trace information of one or more service nodes).

[0073] Optionally, in order to track and analyze the complete transmission process of the target network transmission data packet of the receiving type, the target network transmission data packet and its target transmission identification information are stored in the distributed system memory.

[0074] In this step, in full-link monitoring, for receiving type data packets, if the data packets carry target transmission identification information, the system will temporarily store these data packets and their identification information in memory. This mechanism ensures fast access to data packet information and matching association with subsequent sent data packets, and is an important part of building a real-time and accurate full-link monitoring system. By storing data packets and identification information in memory, the real-time and response speed of monitoring can be improved, providing strong support for the operation, maintenance and optimization of distributed systems.

[0075] As an optional implementation, step S204, based on the transmission type of the target network transmission data packet, determines the target transmission identification information corresponding to the target network transmission data packet, including: in response to the transmission type of the target network transmission data packet being a sending type, determining the similarity between the target network transmission data packet and multiple network transmission data packets stored in the memory of the distributed system based on a similarity algorithm; determining a source network transmission data packet associated with the target network transmission data packet from multiple network transmission data packets based on the similarity; and determining the target transmission identification information corresponding to the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet.

[0076] In this embodiment, when the Trace component identifies that the transmission type of the target network transmission data packet is a sending type, it means that these data packets are sent by a service node inside the distributed system. For this type of data packet, a similarity algorithm can be used to analyze the content of the sending type data packet and compare it with multiple network transmission data packets stored in the memory of the distributed system to determine which receiving type data packets these data packets are most related to. Through similarity algorithm calculation, the Trace component can find the receiving type data packet that is most related to the target sending type data packet from the multiple network transmission data packets stored in the memory, that is, the source network transmission data packet. The source data packet is a request initiated by the system inside or outside to the target service node before the target data packet is sent, which may already contain Trace information.

[0077] Optionally, after determining the source network transmission data packet that is most relevant to the target send type data packet, the Trace information of the source data packet will be checked. If the Trace information of the source data packet exists, the target send type data packet will be associated with this Trace information to achieve the transmission of Trace information and the continuity of the link. If the Trace information of the source data packet does not exist, this usually means that the source data packet is a request to enter the monitoring link for the first time. At this time, a new Trace information will be generated to mark and track the transmission process of the target data packet.

[0078] In this step, by responding to the sending type of the target network transmission data packet, using the similarity algorithm to determine the similarity between the target data packet and the source data packet, and then associating or generating the target transmission identification information (Trace information), it is the key to achieve end-to-end tracking in full-link monitoring. This process ensures that the transmission link of each data packet can be accurately identified and recorded, providing an important means for the operation and maintenance and performance optimization of distributed systems.

[0079] As an optional implementation, the monitoring data processing method further includes: determining a transmission link of a target network transmission data packet based on transmission identification information corresponding to a source network transmission data packet and target transmission identification information corresponding to a target network transmission data packet.

[0080] In this embodiment, the source network transmission data packet refers to the data packet initially sent in a request or service call link. For example, when a user initiates a request through the Web or APP, the request is transmitted through the network to the entry point of the system. The first data packet generated in this process is regarded as the source network transmission data packet. After the source data packet is captured and parsed by the Trace component, a unique transmission identification information, such as a Trace ID, will be generated. This ID will be embedded in the source data packet as the starting point for subsequent transmission link tracing.

[0081] Optionally, the target network transmission data packet is a data packet that starts from the source data packet in the service call link, passes through a series of service nodes, and finally reaches the target service node or external system. When the Trace component captures the target data packet, it checks whether it contains transmission identification information (usually Trace ID). If the transmission type of the target data packet is a receiving type and does not contain target transmission identification information, the Trace component will generate a new target transmission identification information for it; for the sending type data packet, the Trace component will use a similarity algorithm to determine its association with the source data packet, and obtain or generate the correct target transmission identification information.

[0082] Optionally, once the source network transmission data packet and the target network transmission data packet are respectively assigned transmission identification information, the next task of the Trace component is to determine the transmission link of the data packet based on these IDs. Specifically, the Trace component compares the target transmission identification information with the source transmission identification information. If the target transmission identification information matches the source transmission identification information, it indicates that the target data packet and the source data packet belong to the same service call link. By matching the transmission identification information between different data packets, the Trace component can reconstruct the complete path of the request, from the source to the target, across multiple service nodes and possible external systems.

[0083] In this step, by comparing the transmission identification information of the source network transmission data packet and the target network transmission data packet, the transmission link of the data packet is determined, and the monitoring and tracking of each node on the service call link is realized.

[0084] Another optional specific implementation is described in detail below.

[0085] Related technologies rely on embedded deployment of collection components when monitoring network transmission data packets, and often collect and analyze traffic data for specific application layer service protocols. External communication components or various gadgets cannot be included in the information transmission link, so it is impossible to truly achieve end-to-end link monitoring and achieve a closed loop.

[0086] However, the embodiment of the present application proposes a method for processing monitoring data, which uses a network data capture tool to monitor and collect data packets in the link in real time. Whether it is a received or sent data packet, its content similarity can be analyzed through an intelligent algorithm, and Trace information can be dynamically generated or matched to ensure the traceability and relevance of each data packet in the entire link. Especially for external communication components and non-mainstream applications, by adjusting and optimizing the data parsing process, these difficult-to-monitor links are also included in the full-link monitoring system, realizing true end-to-end holographic monitoring. Moreover, by regularly uploading Trace information to the distributed messaging system for persistent storage, as well as querying and analyzing the stored data, a comprehensive, real-time, and accurate monitoring view is provided for operation and maintenance personnel, which significantly improves the monitoring efficiency and troubleshooting capabilities of the distributed system, and lays a solid foundation for optimizing system performance and improving service quality.

[0087] In this embodiment, a device is provided that realizes full-link monitoring based on a trace component for collecting port communication data. In the port data collection stage, the existing port data capture gadget is used to capture network transmission data; in the data parsing stage, a packet analysis program for multiple application layer protocols is built in, and after the complete data analysis, the data is adjusted to a certain extent. For example, for the received request data, if there is no trace information, indicating that the request data is external data, then the trace information is newly recorded and stored in the thread context. If there is trace information, it is recorded in the memory, and the message to be sent is compared, and the trace information is added to the message to be sent. For the request data sent out (generally without trace information), after the text conversion processing, the similarity algorithm in natural language processing is used to judge the message similarity, find its source, and determine the trace information. The recorded trace information is periodically sent to the distributed message collection system for persistent storage.

[0088] Figure 3 is an architecture diagram of a full-link monitoring in a distributed system according to an embodiment of the present application, such as Figure 3As shown, it includes front-end applications (such as Web, APP), back-end services (such as Web servers, application programming interfaces (Application Programming Interface, referred to as API) gateways, application plug-ins, data security services (Data Security Service, referred to as DSF) services) and external software or components. Among them, the front-end application is the interface for direct user interaction, which can be a Web application, a mobile application, etc., and is the starting point of the data link. The back-end service is used to process various service nodes of the front-end request, including: Web servers, API gateways, application plug-ins, DSF services, etc., and these components exchange data through the network; the dedicated fixed port trace component is located in the monitoring module between the external software or components and the back-end service, which is responsible for collecting and parsing network port communication data, identifying and associating requests and responses through similarity algorithms, and realizing full-link monitoring. The distributed message system plays the role of data collection and persistent storage, receives data from various service nodes and monitoring components, ensures reliable data transmission and storage, and provides data support for subsequent monitoring and analysis. The holographic monitoring interface is used for the final display layer of monitoring data. By querying and analyzing the persistent data in the distributed message system, it provides visual monitoring results to help operation and maintenance personnel quickly locate and solve problems.

[0089] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0090] Example 2

[0091] The embodiment of the present application also provides a monitoring data processing device. It should be noted that the monitoring data processing device of the embodiment of the present application can be used to execute the monitoring data processing method provided in the embodiment of the present application. The monitoring data processing device provided in the embodiment of the present application is introduced below.

[0092] According to an embodiment of the present application, a device for implementing the above monitoring data processing method is also provided. Figure 4 is a monitoring data processing device according to an embodiment of the present application. Figure 4 As shown, the device includes: a collection unit 401, a parsing unit 402, a first determination unit 403, a second determination unit 404 and a storage unit 405.

[0093] The collection unit 401 is used to collect target network transmission data packets of a predefined network port in the distributed system;

[0094] The parsing unit 402 is used to parse the target network transmission data packet using a network transmission protocol parsing program to obtain a parsing result, wherein the parsing result at least includes a source address and a destination address of the target network transmission data packet;

[0095] A first determining unit 403 is used to determine the transmission type of the target network transmission data packet based on the source address and the destination address in the parsing result;

[0096] A second determining unit 404 is used to determine target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet, wherein the target transmission identification information is used to identify the transmission process of the target network transmission data packet;

[0097] The storage unit 405 is used to send the target transmission identification information to the distributed message system for persistent storage.

[0098] Optionally, the first determination unit 403 is also used to: in response to the source address being a component address outside the distributed system and the destination address being a service node address within the distributed system, determine that the transmission type of the network transmission data is a receiving type; in response to the source address being a service node address within the distributed system, determine that the transmission type of the network transmission data is a sending type.

[0099] Optionally, the second determination unit 404 is also used to: generate new transmission identification information in response to the transmission type of the target network transmission data packet being a receiving type, and the target network transmission data packet does not have target transmission identification information; determine the new transmission identification information as the target transmission identification information corresponding to the target network transmission data packet; the method also includes: storing the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0100] Optionally, the monitoring data processing device 400 is also used to: in response to the transmission type of the target network transmission data packet being a receiving type and the target network transmission data packet containing target transmission identification information, store the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0101] Optionally, the second determination unit 404 is also used to: in response to the transmission type of the target network transmission data packet being a sending type, determine the similarity between the target network transmission data packet and multiple network transmission data packets stored in the memory of the distributed system based on a similarity algorithm; determine a source network transmission data packet associated with the target network transmission data packet from multiple network transmission data packets based on the similarity; and determine the target transmission identification information corresponding to the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet.

[0102] Optionally, the monitoring data processing device 400 is further used to determine the transmission link of the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet and the target transmission identification information corresponding to the target network transmission data packet.

[0103] The monitoring data processing device provided in the embodiment of the present application can parse the target network transmission data packet through a network transmission protocol parsing program to obtain the parsing result. Then, the source address and destination address in the target network transmission data packet included in the parsing result can be used to determine the transmission type of the target network transmission data packet, and then determine the target transmission identification information corresponding to the target network transmission data packet. The transmission process of the target network transmission data packet can be monitored according to the target transmission identification information, thereby realizing full-link monitoring of the target network transmission data packet when it is transmitted in a distributed system, thereby solving the technical problem in the related technology that the network transmission data packet cannot be fully monitored.

[0104] It should be noted that the above-mentioned acquisition unit 401, analysis unit 402, first determination unit 403, second determination unit 404 and storage unit 405 correspond to steps S201 to S205 in Example 1, and the five modules and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

[0105] Example 3

[0106] An embodiment of the present application may provide an electronic device, Figure 5 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 (only one is shown) processor 502, memory 504, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0107] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0108] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: collect the target network transmission data packet of the pre-defined network port in the distributed system; use the network transmission protocol parsing program to parse the target network transmission data packet to obtain the parsing result, wherein the parsing result at least includes the source address and destination address of the target network transmission data packet; based on the source address and destination address in the parsing result, determine the transmission type of the target network transmission data packet; based on the transmission type of the target network transmission data packet, determine the target transmission identification information corresponding to the target network transmission data packet, wherein the target transmission identification information is used to identify the transmission process of the target network transmission data packet; send the target transmission identification information to the distributed message system for persistent storage.

[0109] The processor can also call the information and applications stored in the memory through the transmission device to perform the following steps: in response to the source address being a component address outside the distributed system and the destination address being a service node address within the distributed system, determining that the transmission type of the network transmission data is a receiving type; in response to the source address being a service node address within the distributed system, determining that the transmission type of the network transmission data is a sending type.

[0110] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: in response to the transmission type of the target network transmission data packet being a receiving type, and the target network transmission data packet not having target transmission identification information, generate new transmission identification information; determine the new transmission identification information as the target transmission identification information corresponding to the target network transmission data packet; the monitoring data processing device is also used to: store the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0111] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: in response to the transmission type of the target network transmission data packet being a receiving type, and the target network transmission data packet containing target transmission identification information, the target network transmission data packet and the target transmission identification information are stored in the memory of the distributed system.

[0112] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: in response to the transmission type of the target network transmission data packet being a sending type, determine the similarity between the target network transmission data packet and multiple network transmission data packets stored in the memory of the distributed system based on a similarity algorithm; determine the source network transmission data packet associated with the target network transmission data packet from the multiple network transmission data packets based on the similarity; and determine the target transmission identification information corresponding to the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet.

[0113] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: determine the transmission link of the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet and the target transmission identification information corresponding to the target network transmission data packet.

[0114] By adopting the embodiment of the present application, a method for processing monitoring data is provided. After parsing the target network transmission data packet through a network transmission protocol parsing program and obtaining the parsing result, the source address and destination address in the target network transmission data packet included in the parsing result can be used to determine the transmission type of the target network transmission data packet, and then determine the target transmission identification information corresponding to the target network transmission data packet. According to the target transmission identification information, the transmission process of the target network transmission data packet can be monitored, thereby realizing full-link monitoring of the target network transmission data packet when it is transmitted in a distributed system, thereby solving the technical problem in the related technology that the network transmission data packet cannot be fully monitored.

[0115] It can be understood by those skilled in the art that Figure 5 The structure shown is for illustration only, and the electronic device may also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, abbreviated as MID), a personal access device (Personal Access Device, abbreviated as PAD), etc. Figure 5 The structure of the electronic device is not limited. Figure 5 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 5Different configurations are shown.

[0116] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0117] Example 4

[0118] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the monitoring data processing method provided in the first embodiment.

[0119] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0120] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the program steps of the method for processing monitoring data.

[0121] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0122] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0124] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0125] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0126] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0127] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for processing monitoring data, characterized in that: include: Collect target network transmission data packets of predefined network ports in the distributed system; Analyzing the target network transmission data packet using a network transmission protocol analysis program to obtain an analysis result, wherein the analysis result at least includes a source address and a destination address of the target network transmission data packet; Determine the transmission type of the target network transmission data packet based on the source address and the destination address in the parsing result; Based on the transmission type of the target network transmission data packet, determine the target transmission identification information corresponding to the target network transmission data packet, wherein the target transmission identification information is used to identify the transmission process of the target network transmission data packet; The target transmission identification information is sent to a distributed messaging system for persistent storage.

2. The method according to claim 1, characterized in that Determining the transmission type of the target network transmission data packet based on the source address and the destination address in the parsing result includes: In response to the source address being a component address outside the distributed system and the destination address being a service node address within the distributed system, determining that a transmission type of the target network transmission data packet is a receiving type; In response to the source address being a service node address in the distributed system, determining that the transmission type of the target network transmission data packet is a sending type.

3. The method according to claim 2, characterized in that Determining target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet includes: In response to the transmission type of the target network transmission data packet being the receiving type, and the target network transmission data packet not containing the target transmission identification information, generating new transmission identification information; Determine the new transmission identification information as the target transmission identification information corresponding to the target network transmission data packet; The method further includes: storing the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

4. The method according to claim 3, characterized in that The method further comprises: In response to the transmission type of the target network transmission data packet being the receiving type and the target transmission identification information existing in the target network transmission data packet, the target network transmission data packet and the target transmission identification information are stored in the memory of the distributed system.

5. The method according to claim 2, characterized in that: Determining target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet includes: In response to the transmission type of the target network transmission data packet being the sending type, determining a similarity between the target network transmission data packet and a plurality of network transmission data packets stored in a memory of the distributed system; Based on the similarity, determining a source network transmission data packet associated with the target network transmission data packet from the plurality of network transmission data packets; Based on the transmission identification information corresponding to the source network transmission data packet, the target transmission identification information corresponding to the target network transmission data packet is determined.

6. The method according to claim 5, characterized in that The method further comprises: Based on the transmission identification information corresponding to the source network transmission data packet and the target transmission identification information corresponding to the target network transmission data packet, a transmission link of the target network transmission data packet is determined.

7. A monitoring data processing device, characterized in that: include: A collection unit, used for collecting target network transmission data packets of a predefined network port in the distributed system; A parsing unit, used to parse the target network transmission data packet using a network transmission protocol parsing program to obtain a parsing result, wherein the parsing result at least includes a source address and a destination address of the target network transmission data packet; A first determining unit, configured to determine a transmission type of the target network transmission data packet based on the source address and the destination address in the parsing result; A second determining unit, configured to determine target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet, wherein the target transmission identification information is used to identify the transmission process of the target network transmission data packet; The storage unit is used to send the target transmission identification information to the distributed message system for persistent storage.

8. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 6 when running.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images