Method and apparatus for processing monitoring data, electronic device and readable storage medium

Abstract

This application discloses a method, apparatus, electronic device, and readable storage medium for processing monitoring data, relating to the field of network communication technology. The method includes: collecting target network transmission data packets from predefined network ports in a distributed system; parsing the target network transmission data packets using a network transmission protocol parsing program to obtain parsing results; determining the transmission type of the target network transmission data packets based on the source and destination addresses in the parsing results; determining target transmission identification information corresponding to the target network transmission data packets based on the transmission type of the target network transmission data packets, wherein the target transmission identification information is used to identify the transmission process of the target network transmission data packets; and sending the target transmission identification information to a distributed message system for persistent storage. This application solves the technical problem in related technologies where full-link monitoring of network transmission data packets is impossible.

Description

Technical Field

[0001] This application relates to the field of network communication technology, and more specifically, to a method, apparatus, electronic device, and readable storage medium for processing monitoring data. Background Technology

[0002] Currently, in the fields of cloud computing and distributed technologies, end-to-end monitoring technology has become a key tool for ensuring system stability and performance. Traditional end-to-end monitoring methods typically rely on embedding specific monitoring components or code into specific application-layer services to collect link data. These components can collect link information with low performance overhead without requiring adjustments to business code. The collected data is uniformly managed and distributed through servers or other message queue systems to support a wider range of monitoring needs. However, this approach has encountered limitations in practical applications, especially when dealing with non-mainstream applications or external components, which often cannot be directly monitored by mainstream monitoring frameworks, resulting in breakpoints in the monitoring link. There is also a technical problem of not being able to perform end-to-end monitoring of network transmission data packets.

[0003] There is currently no effective solution to the technical problem of not being able to monitor network data packets across the entire transmission chain in related technologies. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, electronic device, and readable storage medium for processing monitoring data, so as to solve the technical problem in the related art that it is impossible to perform full-link monitoring of network transmission data packets.

[0005] To achieve the above objectives, according to one aspect of this application, a method for processing monitoring data is provided. The method includes: collecting target network transmission data packets from predefined network ports in a distributed system; parsing the target network transmission data packets using a network transmission protocol parsing program to obtain a parsing result, wherein the parsing result includes at least the source address and destination address of the target network transmission data packet; determining the transmission type of the target network transmission data packet based on the source address and destination address in the parsing result; determining target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet, wherein the target transmission identification information is used to identify the transmission process of the target network transmission data packet; and sending the target transmission identification information to a distributed message system for persistent storage.

[0006] Optionally, based on the source address and destination address in the resolution result, the transmission type of the target network transmission data packet is determined, including: in response to the source address being the address of a component outside the distributed system and the destination address being the address of a service node within the distributed system, the transmission type of the target network transmission data packet is determined to be a receiving type; in response to the source address being the address of a service node within the distributed system, the transmission type of the target network transmission data packet is determined to be a sending type.

[0007] Optionally, determining the target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet includes: generating new transmission identification information in response to the target network transmission data packet being of the receiving type and the target network transmission data packet not having target transmission identification information; determining the new transmission identification information as the target transmission identification information corresponding to the target network transmission data packet; the method further includes: storing the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0008] Optionally, the method for processing monitoring data further includes: in response to the target network transmission data packet being of the receiving type and containing target transmission identification information, storing the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0009] Optionally, based on the transmission type of the target network transmission data packet, the target transmission identification information corresponding to the target network transmission data packet is determined, including: in response to the transmission type of the target network transmission data packet being a sending type, determining the similarity between the target network transmission data packet and multiple network transmission data packets stored in the memory of the distributed system; based on the similarity, determining the source network transmission data packet associated with the target network transmission data packet from the multiple network transmission data packets; and based on the transmission identification information corresponding to the source network transmission data packet, determining the target transmission identification information corresponding to the target network transmission data packet.

[0010] Optionally, the method for processing monitoring data further includes: determining the transmission link of the target network data packets based on the transmission identification information corresponding to the source network data packets and the target transmission identification information corresponding to the target network data packets.

[0011] To achieve the above objectives, according to another aspect of this application, a monitoring data processing apparatus is provided. The apparatus includes: a collection unit for collecting target network transmission data packets from predefined network ports in a distributed system; a parsing unit for parsing the target network transmission data packets using a network transmission protocol parsing program to obtain a parsing result, wherein the parsing result includes at least the source address and destination address of the target network transmission data packet; a first determining unit for determining the transmission type of the target network transmission data packet based on the source address and destination address in the parsing result; a second determining unit for determining target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet, wherein the target transmission identification information is used to identify the transmission process of the target network transmission data packet; and a storage unit for sending the target transmission identification information to a distributed message system for persistent storage.

[0012] Optionally, the first determining unit is further configured to: in response to the source address being a component address outside the distributed system and the destination address being a service node address within the distributed system, determine the transmission type of the network transmission data as a receiving type; and in response to the source address being a service node address within the distributed system, determine the transmission type of the network transmission data as a sending type.

[0013] Optionally, the second determining unit is further configured to: generate new transmission identification information in response to the target network transmission data packet having a receiving type and the target network transmission data packet not having target transmission identification information; determine the new transmission identification information as the target transmission identification information corresponding to the target network transmission data packet; the method further includes: storing the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0014] Optionally, the monitoring data processing device is further configured to: in response to the target network transmission data packet being of the receiving type and containing target transmission identification information, store the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0015] Optionally, the second determining unit is further configured to: in response to the transmission type of the target network transmission data packet being a sending type, determine the similarity between the target network transmission data packet and multiple network transmission data packets stored in the memory of the distributed system based on a similarity algorithm; determine the source network transmission data packet associated with the target network transmission data packet from the multiple network transmission data packets based on the similarity; and determine the target transmission identification information corresponding to the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet.

[0016] Optionally, the monitoring data processing device is also used to: determine the transmission link of the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet and the target transmission identification information corresponding to the target network transmission data packet.

[0017] To achieve the above objectives, according to another aspect of this application, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0018] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is provided, which stores a plurality of instructions adapted for loading by a processor and executing the steps in any of the above method embodiments.

[0019] To achieve the above objectives, according to another aspect of this application, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0020] In this embodiment, target network transmission data packets from predefined network ports in a distributed system are collected. A network transmission protocol parsing program is used to parse the target network transmission data packets to obtain a parsing result, which includes at least the source and destination addresses of the target network transmission data packets. Based on the source and destination addresses in the parsing result, the transmission type of the target network transmission data packets is determined. Based on the transmission type of the target network transmission data packets, target transmission identification information corresponding to the target network transmission data packets is determined, whereby the target transmission identification information is used to identify the transmission process of the target network transmission data packets. The target transmission identification information is then sent to a distributed message system for persistent storage. In other words, in this embodiment, after parsing the target network transmission data packets using a network transmission protocol parsing program and obtaining the parsing result, the source and destination addresses of the target network transmission data packets included in the parsing result can be used to determine the transmission type of the target network transmission data packets, thereby determining the target transmission identification information corresponding to the target network transmission data packets. Based on the target transmission identification information, the transmission process of the target network transmission data packets can be monitored, thus achieving full-link monitoring of the target network transmission data packets during transmission in the distributed system, thereby solving the technical problem in related technologies that cannot perform full-link monitoring of network transmission data packets. Attached Figure Description

[0021] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing a method for processing monitoring data according to an embodiment of this application;

[0023] Figure 2 This is a flowchart of a monitoring data processing method according to an embodiment of this application;

[0024] Figure 3 This is an architecture diagram of end-to-end monitoring in a distributed system according to an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of a monitoring data processing device according to an embodiment of this application;

[0026] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0030] Example 1

[0031] According to an embodiment of this application, a method for processing monitoring data is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for processing monitoring data, according to an embodiment of this application. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor (MCU) or a field-programmable gate array (FPGA), etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0033] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0034] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the monitoring data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned monitoring data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0035] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0036] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0037] Under the aforementioned operating environment, this application provides the following: Figure 2 The method for processing monitoring data is shown. Figure 2 This is a flowchart of a monitoring data processing method according to Embodiment 1 of this application.

[0038] Step S201: Collect target network transmission data packets for predefined network ports in the distributed system.

[0039] In the technical solution provided by step S201 of this application, in a distributed system, communication between service nodes and interaction with external components are typically conducted through a network, with data packets transmitted within the network. To achieve end-to-end monitoring, a tracing component can be designed on each critical service node in the distributed system, or on the critical path of the network, to listen for and capture network data packets on target network ports. These ports are typically common ports for inter-service communication, or ports specifically defined for interaction with external components.

[0040] In this embodiment, the target network transmission data packets refer to the data packets that the Trace component pays special attention to in distributed system monitoring. These data packets include, but are not limited to, data from communication between service nodes, data from requests and responses from external components, and any critical data transmissions that may affect system operation and performance.

[0041] Optionally, in order to collect all relevant network transmission data, the Trace component needs to listen on network ports predefined in the system architecture. These ports may include, but are not limited to, ports for internal inter-service communication and target network ports for interaction with external components or systems. These ports may be specifically configured for end-to-end monitoring so that the Trace component can identify and process requests and responses from the outside.

[0042] Optionally, target network transmission data packets on predefined network ports in the distributed system are collected, ensuring that the Trace component can capture all important data flows, providing a comprehensive data foundation for end-to-end monitoring. Subsequent processing of the collected data enables true end-to-end end-to-end monitoring, improving the efficiency and effectiveness of system monitoring.

[0043] Step S202: Use a network transmission protocol parser to parse the target network transmission data packets and obtain the parsing results.

[0044] In the technical solution provided by step S202 of this application, the parsing result includes at least the source address and destination address of the target network transmission data packet. The source address and destination address are two basic elements of network communication; the source address indicates the sender of the data packet, while the destination address indicates the receiver of the data packet.

[0045] In this embodiment, the network transport protocol parser is software specifically designed to understand and parse network data packets. The use of a parser capable of handling multiple application layer protocols ensures that the monitoring system can identify and parse widely used communication protocols in the distributed system, thereby obtaining detailed information about the data packets.

[0046] Optionally, the collected target network transmission data packets, i.e., data captured through predefined network ports, will be sent to a network transmission protocol parser for analysis. The parsing process involves reading the packet header information, including but not limited to source address, destination address, port number, protocol type, etc., as well as the packet payload content. The parsing of the payload content is particularly important because it contains specific request information and response data, making it a key data source for end-to-end monitoring.

[0047] Optionally, after parsing the target network transmission data packet through a network transmission protocol parsing program, a parsing result can be obtained. This parsing result includes at least the source address and destination address of the target network transmission data packet. In addition, the parsing result can also include the specific content of the transmission content.

[0048] In this step, a network transmission protocol parser is used to parse the target network transmission data packets to obtain key information, including the source address and destination address. This provides basic data for subsequent end-to-end tracking and analysis, ensuring the accuracy and effectiveness of monitoring the target network data packets.

[0049] Step S203: Based on the source address and destination address in the parsing result, determine the transmission type of the data packet transmitted to the target network.

[0050] In the technical solution provided by step S203 of this application, after obtaining the parsing result, the transmission type of the target network data packet can be determined based on the source address and destination address in the parsing result. The transmission type includes: sending type and receiving type.

[0051] In this embodiment, if the source address of a data packet corresponds to a service node within the distributed system, then the data packet can be determined to be a transmission type. This means that a service node is sending data to another service node or an external component. For such data packets, intelligent algorithms such as similarity algorithms can be used to determine their similarity to previously received data packets, in order to identify or generate trace information, thereby tracing the source and destination of the request.

[0052] Optionally, if the destination address of the data packet corresponds to a service node within the distributed system, then the data packet can be identified as a receive packet. This means the data packet is entering the system and may originate from another service node or an external component. For receive packets, if the packet does not contain trace information, this typically indicates that the packet is from outside or is entering the monitoring link for the first time. The system will generate a new trace and store it in the thread context for subsequent tracing. If the packet already contains trace information, the system will record this information for later comparison with other send packets to determine the relationships between the packets.

[0053] In this step, the transmission type of the data packets transmitted over the target network is determined by resolving the source and destination addresses. This enables intelligent differentiation of whether the data packets are flowing within the system or originating from / going to the outside. This is crucial for establishing end-to-end monitoring and ensuring the integrity and consistency of each service call link.

[0054] Step S204: Based on the transmission type of the target network transmission data packet, determine the target transmission identification information corresponding to the target network transmission data packet.

[0055] In the technical solution provided in step S204 of this application, the transmission type of the target network transmission data packet is identified. If it is a received data packet, it can be checked whether the target network data packet already has target transmission identification information, which can also be called Trace information. If the target network transmission data packet originates from outside the system, that is, a data packet sent from an external component or other external system, it generally does not carry pre-existing target transmission identification information. In this case, the Trace component will generate a new target transmission identification information and store it in the thread context so that this information can be referenced when sending request data packets related to this data packet. If the data packet already carries target transmission identification information, it indicates that the data packet was sent from inside the system or from an external system that has been marked by Trace. The Trace component will record this information to provide a basis for subsequent data packet comparison and association of target transmission identification information.

[0056] In this embodiment, if it is a sent data packet, the Trace component uses a similarity algorithm to analyze the packet content to determine its source or related received data packets, thereby obtaining or generating the correct target transmission identification information. Target transmission identification information, typically existing in the form of a target transmission identifier, is a key identifier in end-to-end monitoring, used to identify and track the complete transmission process of data packets in the distributed system. For sent data packets, the Trace component uses a similarity algorithm to determine their similarity to already recorded data packets based on the packet content, source address, and destination address, finding possible source data packets and thus determining or generating the correct target transmission identification information. For received data packets, if the packet already contains target transmission identification information, the Trace component uses it directly; otherwise, the component generates a new target transmission identification information.

[0057] Step S205: Send the target transmission identification information to the distributed messaging system for persistent storage.

[0058] In the technical solution provided in step S205 of this application, after determining the target transmission identification information, this target identification information, along with other key information of the target network transmission data packet (such as source address, destination address, timestamp, etc.), can be sent to a distributed message system for persistent storage. In subsequent monitoring and analysis, operations and maintenance personnel can retrieve relevant data packet records from the distributed message system through trace information, thereby constructing a complete request-response chain, tracing the data flow in the system, locating fault points, analyzing performance bottlenecks, and optimizing system performance.

[0059] In this step, the target transmission identification information, persistently stored in the distributed messaging system, can be retrieved and used by multiple backend services or analysis tools through a subscription mechanism. This information can be aggregated and analyzed to identify performance bottlenecks, abnormal behavior, and system failures, helping operations personnel quickly locate problems and take appropriate measures.

[0060] In steps S201 to S205 above, after parsing the target network transmission data packet through the network transmission protocol parsing program and obtaining the parsing result, the transmission type of the target network transmission data packet can be determined by using the source address and destination address in the target network transmission data packet included in the parsing result. Then, the target transmission identification information corresponding to the target network transmission data packet can be determined. Based on the target transmission identification information, the transmission process of the target network transmission data packet can be monitored, thereby realizing full-link monitoring of the target network transmission data packet during transmission in the distributed system, and thus solving the technical problem in related technologies that it is impossible to perform full-link monitoring of network transmission data packets.

[0061] The embodiments of the present invention will now be described in detail with reference to the steps described above.

[0062] As an optional implementation, step S203, based on the source address and destination address in the parsing result, determines the transmission type of the target network transmission data packet, including: in response to the source address being the address of a component outside the distributed system and the destination address being the address of a service node within the distributed system, determining the transmission type of the network transmission data as a receiving type; in response to the source address being the address of a service node within the distributed system, determining the transmission type of the network transmission data as a sending type.

[0063] In this embodiment, when the source address of the parsed target network transmission data packet corresponds to the address of a component outside the distributed system, and the destination address corresponds to the address of a service node inside the system, the system determines the transmission type of the data packet as a receive type. This means that the data packet is being sent from an external environment to a service node within the distributed system. This could be a request initiated by a user through a webpage or application (APP), or data sent from other external systems or components to the system. For receive type data packets, the system needs to create a new trace to track the processing path of the request. This trace is stored in the thread context and used for comparison and association with subsequent send type data packets.

[0064] Optionally, if the source address of the target network transmission data packet corresponds to the address of a service node within the system, while the destination address may be the address of another service node or an external component, the system determines the transmission type of the data packet as a "send" type. This means that the data packet is being sent from one service node to another or to an external environment, which could be a response to a request or a call between services. For "send" type data packets, the system uses intelligent algorithms such as similarity algorithms to determine the similarity between the content of the data packet and previously received data packets, thereby determining or generating correct trace information to ensure the integrity and consistency of the request-response chain.

[0065] In this step, by determining the transmission type of data packets based on the source and destination addresses, the flow direction of data packets can be clearly distinguished, ensuring that the transmission process of each data packet is accurately recorded and tracked.

[0066] As an optional implementation, step S204, determining the target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet, includes: generating new transmission identification information in response to the target network transmission data packet being a receiving type and the target network transmission data packet not having target transmission identification information; determining the new transmission identification information as the target transmission identification information corresponding to the target network transmission data packet; the method for processing the monitoring data further includes: storing the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0067] In this embodiment, when the transmission type of the target network transmission data packet detected by the Trace component is determined to be receive type, and the data packet does not carry target transmission identification information (e.g., Trace ID), this usually means that the data packet originated from outside the system or that the source of the data packet has not yet been identified by the monitoring system. At this time, the Trace component automatically generates a new transmission identification information (e.g., a new Trace ID) for the data packet to ensure that every data packet entering the system can be monitored and traced. The generated transmission identification information will be associated with the data packet, marking it as the starting point of a brand new request or service call.

[0068] Optionally, once new transport identification information is generated, or if transport identification information already exists in the data packet, the Trace component will identify this information as the target transport identification information corresponding to the data packet. The target transport identification information is the core of end-to-end monitoring, used to uniquely identify the complete link of a request or service call, ensuring that every forwarding, processing, and response of a data packet in the system can be traced and associated.

[0069] Optionally, after generating or determining the target transmission identification information, the Trace component temporarily stores the target network transmission data packets and the corresponding transmission identification information in the memory of the distributed system. The purpose of memory storage is to enable fast data access and processing, especially when it needs to be correlated with subsequently sent data packets. The Trace information stored in memory can be quickly retrieved and used to achieve rapid data packet comparison and end-to-end tracing.

[0070] In this step, in response to the target network transmission data packet being of the receive type and the target network transmission data packet not having target transmission identification information, new transmission identification information is generated; the new transmission identification information is identified as the target transmission identification information corresponding to the target network transmission data packet, and then the target transmission identification information is stored in the memory of the distributed system, which can ensure the traceability and rapid processing of target network transmission data packets in the end-to-end monitoring.

[0071] As an optional implementation, the monitoring data processing method further includes: in response to the target network transmission data packet being of the receiving type and containing target transmission identification information, storing the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0072] In this embodiment, when the transmission type of the target network transmission data packet is receive, it means that the target network transmission data packet enters the current service node from an external or another internal service node. If the data packet carries target transmission identification information, this usually indicates that the data packet arrived as part of a service call chain, and that the transmission process of the data packet has been identified throughout the call chain (e.g., through trace information of one or more service nodes).

[0073] Optionally, in order to track and analyze the complete transmission process of the target network transmission data packets of the received type, the target network transmission data packets and their target transmission identification information are stored in the memory of the distributed system.

[0074] In this step of end-to-end monitoring, for received data packets, if the packet carries target transmission identification information, the system temporarily stores these data packets and their identification information in memory. This mechanism ensures rapid access to data packet information and matching with subsequently sent data packets, and is an important component of building a real-time, accurate end-to-end monitoring system. By storing data packets and identification information in memory, the real-time performance and response speed of monitoring can be improved, providing strong support for the operation and optimization of distributed systems.

[0075] As an optional implementation, step S204, determining the target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet, includes: in response to the transmission type of the target network transmission data packet being a sending type, determining the similarity between the target network transmission data packet and multiple network transmission data packets stored in the memory of the distributed system based on a similarity algorithm; determining the source network transmission data packet associated with the target network transmission data packet from the multiple network transmission data packets based on the similarity; and determining the target transmission identification information corresponding to the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet.

[0076] In this embodiment, when the Trace component identifies the transmission type of the target network transmission data packet as "sending," it means that these data packets were sent by a service node within the distributed system. For this type of data packet, a similarity algorithm can be used to analyze the content of the sending type data packet and compare it with multiple network transmission data packets stored in the distributed system's memory to determine which receiving type data packets are most relevant to these data packets. Through similarity algorithm calculation, the Trace component can find the receiving type data packet most relevant to the target sending type data packet from multiple network transmission data packets stored in memory, i.e., the source network transmission data packet. The source data packet is a request initiated from within or outside the system to the target service node before the target data packet is sent, and it may already contain Trace information.

[0077] Optionally, after identifying the source network transmission packet most relevant to the target transmission type packet, the trace information of the source packet will be checked. If the trace information of the source packet exists, the target transmission type packet will be associated with this trace information, thus ensuring the transmission of trace information and the continuity of the link. If the trace information of the source packet does not exist, this usually means that the source packet is making its first request into the monitored link. In this case, a new trace information will be generated to mark and trace the transmission process of the target packet.

[0078] In this step, by responding to the sending type of data packets transmitted over the target network, a similarity algorithm is used to determine the similarity between the target data packet and the source data packet, and then associate or generate target transmission identification information (Trace information). This is crucial for achieving end-to-end tracing in end-to-end monitoring. This process ensures that the transmission link of each data packet can be accurately identified and recorded, providing an important means for the operation and maintenance and performance optimization of distributed systems.

[0079] As an optional implementation method, the monitoring data processing method further includes: determining the transmission link of the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet and the target transmission identification information corresponding to the target network transmission data packet.

[0080] In this embodiment, the source network transmission data packet refers to the first data packet sent in a request or service call chain. For example, when a user initiates a request through the web or an app, the request travels through the network to the system's entry point, and the first data packet generated in this process is considered the source network transmission data packet. After the source data packet is captured and parsed by the Trace component, a unique transmission identification information, such as a Trace ID, is generated. This ID is embedded in the source data packet as the starting point for subsequent transmission chain tracing.

[0081] Optionally, the target network transmission data packet is a data packet that, in the service call chain, starts from the source data packet, passes through a series of service nodes, and finally arrives at the target service node or external system. When the Trace component captures the target data packet, it checks whether it contains transmission identification information (usually a Trace ID). If the target data packet is of the receive type and does not contain target transmission identification information, the Trace component will generate a new target transmission identification information for it; while for the send type data packet, the Trace component will use a similarity algorithm to determine its association with the source data packet and obtain or generate the correct target transmission identification information.

[0082] Optionally, once the source and destination network transmission data packets are assigned transmission identification information, the Trace component's next task is to determine the transmission link of the data packets based on these IDs. Specifically, the Trace component compares the destination transmission identification information with the source transmission identification information. If the destination and source transmission identification information match, it indicates that the destination data packet and the source data packet belong to the same service call link. By matching the transmission identification information between different data packets, the Trace component can reconstruct the complete path of the request, from the source to the destination, spanning multiple service nodes and possible external systems.

[0083] In this step, the transmission link of the data packets is determined by comparing the transmission identification information of the data packets transmitted from the source network and the target network, thus enabling the monitoring and tracking of each node on the service call link.

[0084] The following describes in detail another optional implementation method.

[0085] Related technologies, when monitoring network data packets, rely on embedded data acquisition components, which often collect and parse traffic data for specific application-layer service protocols. External communication components or various small tools cannot be incorporated into the information transmission link, thus failing to achieve true end-to-end link monitoring and a closed loop.

[0086] However, this application proposes a method for processing monitoring data. Through network data capture tools, it monitors and collects data packets in the link in real time. Whether the data packets are received or sent, intelligent algorithms analyze their content similarity to dynamically generate or match trace information, ensuring the traceability and relevance of each data packet throughout the entire link. Specifically targeting external communication components and non-mainstream applications, by adjusting and optimizing the data parsing process, these difficult-to-monitor links are also incorporated into the full-link monitoring system, achieving true end-to-end holistic monitoring. Furthermore, by periodically uploading trace information to a distributed messaging system for persistent storage, and by querying and analyzing the stored data, it provides maintenance personnel with a comprehensive, real-time, and accurate monitoring view, significantly improving the monitoring efficiency and troubleshooting capabilities of the distributed system, laying a solid foundation for optimizing system performance and improving service quality.

[0087] This embodiment provides a trace component based on collected port communication data, thereby enabling end-to-end monitoring. During the port data acquisition phase, existing port data capture tools are used to capture network transmission data. In the data parsing phase, a built-in packet analysis program for various application layer protocols is implemented. After complete data analysis, the data is adjusted accordingly. For example, for received request data, if no trace information is available, it indicates that the request data is external data, and new trace information is recorded and stored in the thread context. If trace information is available, it is recorded in memory and compared with outgoing messages to add new trace information to the outgoing messages. For outgoing request data (generally without trace information), after text conversion, a similarity algorithm from natural language processing is used to determine message similarity, locate its source, and determine trace information. The recorded trace information is periodically and persistently stored in the distributed message system for data acquisition.

[0088] Figure 3 This is an architecture diagram of end-to-end monitoring in a distributed system according to an embodiment of this application, such as... Figure 3As shown, the system includes front-end applications (such as web applications and mobile apps), back-end services (such as web servers, application programming interface (API) gateways, application plugins, and data security service (DSF) services), and external software or components. The front-end application is the user interface, which can be a web application, mobile application, etc., and is the starting point of the data chain. Back-end services are the various service nodes that handle front-end requests, including web servers, API gateways, application plugins, and DSF services. These components exchange data over the network. A dedicated fixed-port trace component is located in the monitoring module between the external software or component and the back-end service. It is responsible for collecting and parsing network port communication data, identifying and associating requests and responses through similarity algorithms, and achieving end-to-end monitoring. The distributed messaging system acts as a data collection and persistent storage layer, receiving data from various service nodes and monitoring components, ensuring reliable data transmission and storage, and providing data support for subsequent monitoring and analysis. The holographic monitoring interface is the final display layer for monitoring data. By querying and analyzing the persistent data in the distributed messaging system, it provides visualized monitoring results, helping operations personnel quickly locate and resolve problems.

[0089] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0090] Example 2

[0091] This application also provides a monitoring data processing apparatus. It should be noted that the monitoring data processing apparatus of this application can be used to execute the monitoring data processing method provided in this application. The monitoring data processing apparatus provided in this application will be described below.

[0092] According to an embodiment of this application, an apparatus for implementing the above-described method for processing monitoring data is also provided. Figure 4 This is a monitoring data processing device according to an embodiment of this application. For example... Figure 4 As shown, the device includes: a data acquisition unit 401, a data parsing unit 402, a first determination unit 403, a second determination unit 404, and a storage unit 405.

[0093] Acquisition unit 401 is used to acquire target network transmission data packets of predefined network ports in a distributed system;

[0094] The parsing unit 402 is used to parse the target network transmission data packet using a network transmission protocol parsing program to obtain the parsing result, wherein the parsing result includes at least the source address and destination address of the target network transmission data packet;

[0095] The first determining unit 403 is used to determine the transmission type of the data packet transmitted by the target network based on the source address and destination address in the parsing result.

[0096] The second determining unit 404 is used to determine the target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet, wherein the target transmission identification information is used to identify the transmission process of the target network transmission data packet.

[0097] Storage unit 405 is used to send the target transmission identification information to the distributed messaging system for persistent storage.

[0098] Optionally, the first determining unit 403 is further configured to: in response to the source address being a component address outside the distributed system and the destination address being a service node address within the distributed system, determine the transmission type of the network transmission data as a receiving type; and in response to the source address being a service node address within the distributed system, determine the transmission type of the network transmission data as a sending type.

[0099] Optionally, the second determining unit 404 is further configured to: generate new transmission identification information in response to the target network transmission data packet having a receiving type and the target network transmission data packet not having target transmission identification information; determine the new transmission identification information as the target transmission identification information corresponding to the target network transmission data packet; the method further includes: storing the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0100] Optionally, the monitoring data processing device 400 is further configured to: in response to the target network transmission data packet having a receive type transmission type and the target network transmission data packet containing target transmission identification information, store the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0101] Optionally, the second determining unit 404 is further configured to: in response to the transmission type of the target network transmission data packet being a sending type, determine the similarity between the target network transmission data packet and multiple network transmission data packets stored in the memory of the distributed system based on a similarity algorithm; determine the source network transmission data packet associated with the target network transmission data packet from the multiple network transmission data packets based on the similarity; and determine the target transmission identification information corresponding to the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet.

[0102] Optionally, the monitoring data processing device 400 is further configured to: determine the transmission link of the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet and the target transmission identification information corresponding to the target network transmission data packet.

[0103] The monitoring data processing device provided in this application, after parsing the target network transmission data packet through a network transmission protocol parsing program and obtaining the parsing result, can use the source address and destination address in the target network transmission data packet included in the parsing result to determine the transmission type of the target network transmission data packet, and then determine the target transmission identification information corresponding to the target network transmission data packet. Based on the target transmission identification information, the transmission process of the target network transmission data packet can be monitored, thereby realizing full-link monitoring of the target network transmission data packet during transmission in a distributed system, and thus solving the technical problem in related technologies that it is impossible to perform full-link monitoring of network transmission data packets.

[0104] It should be noted that the acquisition unit 401, parsing unit 402, first determination unit 403, second determination unit 404, and storage unit 405 correspond to steps S201 to S205 in Embodiment 1. The five modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the modules or units described above can be hardware or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). These modules can also run as part of a device in the computer terminal 10 provided in Embodiment 1.

[0105] Example 3

[0106] Embodiments of this application may provide an electronic device. Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 Only one of the components is shown: processor 502, memory 504, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module, and display.

[0107] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0108] The processor can access information and applications stored in memory via a transmission device to execute the following steps: acquiring target network transmission data packets from predefined network ports in the distributed system; parsing the target network transmission data packets using a network transmission protocol parser to obtain a parsing result, wherein the parsing result includes at least the source address and destination address of the target network transmission data packet; determining the transmission type of the target network transmission data packet based on the source and destination addresses in the parsing result; determining the target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet, wherein the target transmission identification information is used to identify the transmission process of the target network transmission data packet; and sending the target transmission identification information to a distributed message system for persistent storage.

[0109] The processor can also invoke information and applications stored in memory via the transmission device to perform the following steps: in response to a source address being a component address outside the distributed system and a destination address being a service node address within the distributed system, determine that the transmission type of the network transmission data is a receive type; in response to a source address being a service node address within the distributed system, determine that the transmission type of the network transmission data is a send type.

[0110] The processor can also invoke information and application programs stored in the memory through the transmission device to perform the following steps: in response to the target network transmission data packet being of the receiving type and the target network transmission data packet not having target transmission identification information, generating new transmission identification information; determining the new transmission identification information as the target transmission identification information corresponding to the target network transmission data packet; the monitoring data processing device is also used to: store the target network transmission data packet and the target transmission identification information in the memory of the distributed system.

[0111] The processor can also invoke information and applications stored in memory through the transmission device to perform the following steps: in response to the target network transmission data packet being of the receive type and containing target transmission identification information, the target network transmission data packet and the target transmission identification information are stored in the memory of the distributed system.

[0112] The processor can also invoke information and applications stored in memory via the transmission device to perform the following steps: in response to the transmission type of the target network transmission data packet being a send type, determine the similarity between the target network transmission data packet and multiple network transmission data packets stored in the memory of the distributed system based on a similarity algorithm; determine the source network transmission data packet associated with the target network transmission data packet from the multiple network transmission data packets based on the similarity; and determine the target transmission identification information corresponding to the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet.

[0113] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determine the transmission link of the target network transmission data packet based on the transmission identification information corresponding to the source network transmission data packet and the target transmission identification information corresponding to the target network transmission data packet.

[0114] This application provides a method for processing monitoring data. After parsing the target network transmission data packet using a network transmission protocol parsing program and obtaining the parsing result, the transmission type of the target network transmission data packet can be determined using the source address and destination address of the target network transmission data packet included in the parsing result. This allows for the determination of the target transmission identification information corresponding to the target network transmission data packet. Based on the target transmission identification information, the transmission process of the target network transmission data packet can be monitored, thereby achieving full-link monitoring of the target network transmission data packet during transmission in a distributed system. This solves the technical problem in related technologies that cannot perform full-link monitoring of network transmission data packets.

[0115] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PDAs, mobile internet devices (MIDs), personal access devices (PADs), and other terminal devices. Figure 5 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5The different configurations shown.

[0116] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0117] Example 4

[0118] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the monitoring data processing method provided in Embodiment 1.

[0119] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0120] This application also provides a computer program product that, when executed on a data processing device, is suitable for performing method steps for processing monitoring data.

[0121] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0122] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0124] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0126] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0127] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing monitoring data, characterized in that, include: Collect target network transmission data packets on predefined network ports in a distributed system; The target network transmission data packet is parsed using a network transmission protocol parsing program to obtain a parsing result, wherein the parsing result includes at least the source address and destination address of the target network transmission data packet; Based on the source address and destination address in the parsing result, the transmission type of the data packet transmitted over the target network is determined; Based on the transmission type of the target network transmission data packet, the target transmission identification information corresponding to the target network transmission data packet is determined, wherein the target transmission identification information is used to identify the transmission process of the target network transmission data packet; The target transmission identification information is sent to a distributed messaging system for persistent storage.

2. The method according to claim 1, characterized in that, Based on the source address and destination address in the parsing result, the transmission type of the data packet transmitted over the target network is determined, including: In response to the fact that the source address is the address of a component outside the distributed system and the destination address is the address of a service node within the distributed system, the transmission type of the target network data packet is determined to be the receive type; In response to the source address being the address of a service node within the distributed system, the transmission type of the target network data packet is determined to be a sending type.

3. The method according to claim 2, characterized in that, Based on the transmission type of the target network transmission data packet, the target transmission identification information corresponding to the target network transmission data packet is determined, including: In response to the target network transmitting data packet having the transmission type of the receiving type and the target network transmitting data packet not having the target transmission identification information, new transmission identification information is generated; The new transmission identification information is determined as the target transmission identification information corresponding to the target network transmission data packet; The method further includes storing the target network transmission data packet and the target transmission identification information into the memory of the distributed system.

4. The method according to claim 3, characterized in that, The method further includes: In response to the target network transmission data packet having the receiving type as the transmission type and the target network transmission data packet containing the target transmission identification information, the target network transmission data packet and the target transmission identification information are stored in the memory of the distributed system.

5. The method according to claim 2, characterized in that, Based on the transmission type of the target network transmission data packet, the target transmission identification information corresponding to the target network transmission data packet is determined, including: In response to the transmission type of the target network transmission data packet being the sending type, the similarity between the target network transmission data packet and multiple network transmission data packets stored in the memory of the distributed system is determined; Based on the similarity, the source network transmission data packet associated with the target network transmission data packet is determined from the plurality of network transmission data packets; Based on the transmission identification information corresponding to the data packet transmitted from the source network, the target transmission identification information corresponding to the data packet transmitted from the target network is determined.

6. The method according to claim 5, characterized in that, The method further includes: Based on the transmission identification information corresponding to the source network transmission data packet and the target transmission identification information corresponding to the target network transmission data packet, the transmission link of the target network transmission data packet is determined.

7. A device for processing monitoring data, characterized in that, include: The acquisition unit is used to acquire target network transmission data packets on predefined network ports in a distributed system. The parsing unit is used to parse the target network transmission data packet using a network transmission protocol parsing program to obtain a parsing result, wherein the parsing result includes at least the source address and destination address of the target network transmission data packet; The first determining unit is used to determine the transmission type of the target network transmission data packet based on the source address and the destination address in the parsing result; The second determining unit is used to determine the target transmission identification information corresponding to the target network transmission data packet based on the transmission type of the target network transmission data packet, wherein the target transmission identification information is used to identify the transmission process of the target network transmission data packet; The storage unit is used to send the target transmission identification information to the distributed messaging system for persistent storage.

8. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images