Distributed system analysis method and apparatus

Abstract

Embodiments of the present application provide a distributed system analysis method, applied to a distributed system, where the distributed system includes N computer devices and a server device, a kernel space of each of the N computer devices includes an extended Berkley packet filter (eBPF) code, the eBPF code is configured to collect a data packet upon the occurrence of a target event, the method is executed by a first computer device, and the first computer device is any one of the N computer devices, N being an integer greater than 1, the method including: collecting a data packet when the target event occurs; and sending the data packet to the server device, the server device being configured to collect data packets from the N computer devices and analyze dependencies between services associated with the data packets. The technical solutions of this application can increase knowledge about distributed systems and thus improve their performance and reliability.

Description

DISTRIBUTED SYSTEM ANALYSIS METHOD AND APPARATUSTECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of distributed systems, and more specifically, to a distributed system analysis method and an apparatus.BACKGROUND

[0002] Nowadays, the popularity of distributed systems is growing, and there is a strong tendency to design new ones and migrate existing software systems to service-oriented architectures. However, distributed systems are complex and hard to design. In distributed systems, services can be implemented using different programming languages and technologies and can use various communication protocols and even hardware. This is a serious challenge for observability software since it should support all the technologies that can be met in the system. If multiple teams are involved in the development process of a distributed system, the system architecture should be well documented, and this documentation should be shared across all development teams and maintained as relevant. Maintaining the documentation up-to-date is a time-consuming task, while a lack of knowledge about system architecture may lead to bad design solutions, misunderstanding between teams and bugs.

[0003] Therefore, how to increase knowledge about distributed systems and thus improve their performance and reliability is an urgent technical problem to be solved.SUMMARY

[0004] Embodiments of the present application provide a distributed system analysis method and an apparatus. The technical solutions of this application can increase knowledge about distributed systems and thus improve their performance and reliability.

[0005] According to a first aspect, an embodiment of the present application provides adistributed system analysis method, applied to a distributed system, where the distributed system includes N computer devices and a server device, a kernel space of each of the N computer devices includes an extended Berkley packet filter (eBPF) code, the eBPF code is configured to collect a data packet upon the occurrence of a target event, the method is executed by a first computer device, and the first computer device is any one of the N computer devices, N being an integer greater than 1 , the method including: collecting a data packet when the target event occurs; and sending the data packet to the server device, the server device being configured to collect data packets from the N computer devices and analyze dependencies between services associated with the data packets.

[0006] The distributed system analysis method provided by the present application can collect a data packet corresponding to the target event by injecting the eBPF code in the kernel space of a computer device and send the data packet to a server device. The server device can analyze data packets from different computers and obtain dependencies between services associated with the data packets. Analyzing these dependencies can help improve the performance and reliability of the distributed system.

[0007] The eBPF is a technology that can run sandboxed programs in the Linux kernel without changing kernel source code or loading a kernel module.

[0008] In an operating system of a computing device, virtual memory is usually divided into a user space and a kernel space. All applications of the computing device share the kernel space. Since kernel space exists in only one instance (operating system), there is no need to load the eBPF program multiple times for each application.

[0009] In one possible implementation scenario, the dependencies can be a dependency table or a dependency graph.

[0010] In a possible implementation, the first computer device includes a tracing agent, the tracing agent being configured to inject the eBPF code into a kernel space of the first computer device.

[0011] To detect communications between services, tracing agents are deployed on computer devices. When a system of the first computer device has started, a tracing agent injects the eBPF code into the kernel space. This code can contain kernel probes (kprobes) that will be triggered when the target event occurs.

[0012] The tracing agents can use the eBPF to set probes on kernel functions that participate in data packet routing. The kernel functions can for example be sock_sendmsg, sock_recvmsg, tcp_connect, inet_accept, etc.

[0013] The distributed system analysis method provided by the present application can inject the eBPF code into the kernel space of a computer device through a tracing agent, collect data packets corresponding to target events, and send the data packets to a server device. The server device can analyze data packets from different computers and obtain dependencies between services associated with the data packets. Analyzing these dependencies can help improve the performance and reliability of the distributed system.

[0014] In a possible implementation, the target event includes any one or more of the following: a socket is opened or closed, a connection between sockets is established, a data packet is sent or received, an internet protocol virtual server (IP VS) or internet protocol tables (IP Tables) redirects a data packet.

[0015] Each target event has an associated kernel function. The eBPF code is executed when the kernel function associated with the target event is called. The kernel functions can for example be sock_sendmsg, sock_recvmsg, tcp_connect, inet_accept, etc.

[0016] This application observes the lifecycle of a socket by capturing corresponding events (open, close, etc.). Tracing accuracy can be improved by handling the reuse of socket identifiers.

[0017] When requests are routed by middleware, it uses specific kernel modules such as IP VS or IP Tables. The eBPF code that was executed during the request routing makes it possible to find the same data packet collected on a client and a server by matching all four internet protocol (IP) address and port pairs that participated in routing. The four IP address and port pairs refer to a sender's source IP address and port, and destination IP address and port and a receiver's source IP address and port, and destination IP address and port.

[0018] This application uses eBPF to collect arguments passed to kernel functions that participate in IPVS or IP Tables routing to find out how packets are routed when load balancers, containers or orchestrators are used. This unified approach to packet tracing does not rely on any specific API provided by middleware.

[0019] The distributed system analysis method provided by the present application cancollect corresponding data packets when the target event occurs and send the data packets to the server device. If the method provided in this application attaches the eBPF program to the IP VS or IP Tables function that performs the routing, the correct IP address can be obtained. Real communications can be captured even with middleware.

[0020] In a possible implementation, a pointer to the socket is a unique identifier of the socket.

[0021] Since each process has a different file descriptor for the same socket, using socket file descriptors can’t handle cases when multiple processes use the same socket.

[0022] Using a socket pointer as a unique identifier for a socket improves trace accuracy in the following cases: the socket is being used by more than one process; the socket is being passed to another process; and the tracing agent has been restarted.

[0023] The method provided by the present application can use a pointer to a kernel socket structure capable of uniquely identifying a socket, even if the socket is migrated between processes, used concurrently, or the agent is restarted. These structures are allocated in the kernel space, and their position never changes and is always the same for the whole socket lifetime, including the cases mentioned above.

[0024] In a possible implementation, the eBPF code is executed when a kernel function associated with the target event is called.

[0025] According to a second aspect, an embodiment of the present application provides a distributed system analysis method, applied to a distributed system, where the distributed system includes N computer devices and a server device, a kernel space of each of the N computer devices includes an extended Berkley packet filter (eBPF) code, the eBPF code is configured to collect a data packet upon the occurrence of a target event, and the method is executed by the server device, N being an integer greater than 1, the method comprising: receiving multiple data packets from the N computer devices; and generating dependencies between services associated with the multiple data packets.

[0026] The distributed system analysis method provided by the present application can collect a data packet corresponding to the target event by injecting the eBPF code in the kernel space of a computer device and send the data packet to a server device. The server device can analyze data packets from different computers and obtain dependencies between servicesassociated with the data packets. Analyzing these dependencies can help improve the performance and reliability of the distributed system.

[0027] The eBPF is a technology that can run sandboxed programs in the Linux kernel without changing kernel source code or loading a kernel module.

[0028] In an operating system of a computing device, virtual memory is usually divided into a user space and a kernel space. All applications of the computing device share the kernel space. Since kernel space exists in only one instance (operating system), there is no need to load the eBPF program multiple times for each application.

[0029] In one possible implementation scenario, the dependencies can be a dependency table or a dependency graph.

[0030] In a possible implementation, the generating dependencies between services associated with the multiple data packets includes: matching source IP addresses and ports, and destination IP addresses and ports of the multiple data packets; and generating the dependencies between the services associated with the multiple data packets based on the matching.

[0031] The eBPF code that was executed during the request routing makes it possible to find the same data packet collected on a client and a server by matching all four internet protocol (IP) address and port pairs that participated in routing. The four IP address and port pairs refer to a sender’s source IP address and port, and destination IP address and port and a receiver's source IP address and port, and destination IP address and port.

[0032] In a possible implementation, the dependencies include a dependency graph.

[0033] In a possible implementation, a first computer device includes a tracing agent, the tracing agent being configured to inject the eBPF code into a kernel space of the first computer device, the first computer device being any one of the N computer devices.

[0034] In a possible implementation, the target event includes any one or more of the following: a socket is opened or closed, a connection between sockets is established, a data packet is sent or received, an internet protocol virtual server (IP VS) or internet protocol tables (IP Tables) redirects a data packet.

[0035] In a possible implementation, a pointer to the socket is a unique identifier of the socket.

[0036] In a possible implementation, the eBPF code is executed when a kernel functionassociated with the target event is called.

[0037] The beneficial effects of the second aspect of the possible implementation correspond to the beneficial effects of the first aspect of the possible implementation, which will not be repeated.

[0038] According to a third aspect, an embodiment of this application provides a computing apparatus, and the computing apparatus has the function of implementing the method in the first aspect. The function may be implemented by hardware executing corresponding software. The hardware or the software includes one or more modules corresponding to the function.

[0039] According to a fourth aspect, an embodiment of this application provides a computing apparatus, and the computing apparatus has the function of implementing the method in the second aspect. The function may be implemented by hardware executing corresponding software. The hardware or the software includes one or more modules corresponding to the function.

[0040] According to a fifth aspect, an embodiment of this application provides a distributed system including a first computer device and a server device, where the first computer device is enabled to perform the method in the first aspect or any possible design of the first aspect, and the server device is enabled to perform the method in the second aspect or any possible design of the second aspect.

[0041] According to a sixth aspect, an embodiment of this application provides a computer- readable storage medium including instructions. When the instructions are run on a computer, the computer is enabled to perform the method in the first aspect or any possible design of the first aspect.

[0042] According to a seventh aspect, an embodiment of this application provides a computer-readable storage medium including instructions. When the instructions are run on a computer, the computer is enabled to perform the method in the second aspect or any possible design of the second aspect.

[0043] According to an eighth aspect, an embodiment of this application provides a computing device, including a processor, a memory, and a communications interface. The processor is connected to the memory and the communications interface. The memory is configured to store instructions, the processor is configured to execute the instructions, and thecommunications interface is configured to communicate with another network element under control of the processor. When the processor executes the instructions stored in the memory, the processor is enabled to perform the method in the first aspect or any possible design of the first aspect.

[0044] According to a ninth aspect, an embodiment of this application provides a computing device, including a processor, a memory, and a communications interface. The processor is connected to the memory and the communications interface. The memory is configured to store instructions, the processor is configured to execute the instructions, and the communications interface is configured to communicate with another network element under control of the processor. When the processor executes the instructions stored in the memory, the processor is enabled to perform the method in the second aspect or any possible design of the second aspect.

[0045] According to a tenth aspect, an embodiment of this application provides a computing device cluster, including at least one computing device, where the computing device includes a processor and a memory coupled with the processor, where the memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory, so that the computing device cluster executes the method in the first aspect or any possible design of the first aspect.

[0046] According to an eleventh aspect, an embodiment of this application provides a computing device cluster, including at least one computing device, where the computing device includes a processor and a memory coupled with the processor, where the memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory, so that the computing device cluster executes the method in the second aspect or any possible design of the second aspect.

[0047] According to a twelfth aspect, a chip system is provided, where the chip system includes a memory and a processor, the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that a server on which a chip is disposed performs the method in the first aspect or any possible design of the first aspect.

[0048] According to a thirteenth aspect, a chip system is provided, where the chip systemincludes a memory and a processor, the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that a server on which a chip is disposed performs the method in the second aspect or any possible design of the second aspect.

[0049] According to a fourteenth aspect, a computer program product is provided, where when the computer program product is run on a computing device cluster, the computing device cluster is enabled to perform the method in the first aspect or any possible design of the first aspect.

[0050] According to a fifteenth aspect, a computer program product is provided, where when the computer program product is run on a computing device cluster, the computing device cluster is enabled to perform the method in the second aspect or any possible design of the second aspect.DESCRIPTION OF DRAWINGS

[0051] FIG. 1 is a flowchart of a distributed system analysis method according to an embodiment of the present application.

[0052] FIG. 2 is a flowchart of a distributed system analysis method according to an embodiment of the present application.

[0053] FIG. 3 is a schematic diagram of a distributed system according to an embodiment of the present application.

[0054] FIG. 4 is a schematic block diagram of a computing apparatus 1500 according to an embodiment of the present application.

[0055] FIG. 5 is a schematic block diagram of a computing apparatus 1600 according to an embodiment of the present application.

[0056] FIG. 6 is a schematic diagram of an architecture of a computing device according to an embodiment of the present application.

[0057] FIG. 7 is a schematic diagram of an architecture of a computing device cluster according to an embodiment of the present application.

[0058] FIG. 8 is a schematic diagram of a connection between computing devices over anetwork according to an embodiment of the present application.DESCRIPTION OF EMBODIMENTS

[0059] The following describes the technical solutions in the present application with reference to the accompanying drawings.

[0060] Obviously, the described embodiments are part of the embodiments of the present application, and not all of them. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without making creative labor shall fall within the scope of protection of the present application.

[0061] The present application will present aspects, embodiments, or features around systems that include multiple devices, components, modules, etc. It should be understood and appreciated that the individual systems may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc., discussed in connection with the accompanying drawings. In addition, combinations of these options may be used.

[0062] In addition, in the embodiments of the present application, the word "exemplarily" and the phrase "as an example" are used to indicate, for example, illustration or description. Any embodiment or design solution described as "exemplarily" in this application should not be construed as being superior to or more advantageous than other embodiments or design solutions. Rather, the use of the word "example" is intended to present the concept in a specific manner.

[0063] The phrases "in some possible embodiments", "in some possible application scenarios", etc., appearing in various places in this description, do not necessarily refer to the same embodiments, but rather mean "one or more, but not all, embodiments" unless otherwise specifically emphasized. Unless otherwise specifically emphasized, the terms "including", "comprising", "having", and variations thereof all mean "including but not limited to".

[0064] In the present application, "at least one" refers to one or more, and "multiple" refers to two or more. The expression "and / or", describing the association of the associated objects, indicates that three relationships can exist. For example, A and / or B can mean A alone, both Aand B, and B alone, where A and B can be singular or plural. The charactergenerally indicates that the preceding and following associated objects are in an "or" relationship.

[0065] The application scenarios described in the embodiments of the present application are intended to illustrate the technical solutions of the embodiments of the present application more clearly and do not constitute a limitation to the technical solutions provided by the embodiments of the present application. It is known to those of ordinary skill in the art that the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems as the system architecture evolves and new application scenarios emerge.

[0066] In order to better describe the solutions of embodiments in the present application, concepts and terms that may be involved in the present application will be described below.

[0067] (1) Tracing

[0068] Tracing is a process of tracing requests between services in a distributed system.

[0069] (2) Continuous tracing

[0070] Continuous tracing is a tracing that can be performed during application operation without significant influence on application performance.

[0071] (3) Distributed system

[0072] A distributed system is a collection of computer programs that utilize computational resources across multiple, and separated computation nodes to achieve a common, and shared goal.

[0073] (4) Monolithic application

[0074] A monolithic application is a single unified software application which is self- contained and independent from other applications, but typically lacks flexibility.

[0075] (5) System call (or syscall)

[0076] A system call is the programmatic way in which a computer program requests a service from the operating system on which it is executed.

[0077] (6) Kernel space

[0078] Kernel space is a place (usually random-access memory) that holds data important for operating system, e.g. file descriptions, sockets, process / thread ids, etc. Kernel space is shared between all applications, but applications cannot directly access it other than usingspecial function (syscalls).

[0079] (7) Kernel probe (or kprobe)

[0080] The kernel probe (or kprobe) is a specific function that can be triggered when particular function declared in the kernel code is executed.

[0081] (8) Internet protocol virtual server (IPVS)

[0082] An internet protocol (IP) virtual server is a part of Linux kernel which implements transport-layer load balancing, and usually called layer 4 local area network (LAN) switching.

[0083] (9) Internet protocol tables (IP Tables)

[0084] IP Tables is a user-space utility program that allows a system administrator to configure the IP packet filter rules of the Linux kernel firewall, implemented as different Netfilter modules. The filters are organized in different tables, which contain chains of rules for how to treat network traffic packets.

[0085] (10) Middleware

[0086] Middleware is a type of software used by different applications to communicate with each other. It provides functionality to connect applications intelligently and efficiently.

[0087] (11) Extended Berkley packet filter (eBPF)

[0088] The eBPF is a technology that can run sandboxed programs in the Linux kernel without changing kernel source code or loading a kernel module.

[0089] (12) Network socket or socket

[0090] The network socket or socket is a software structure within a network node of a computer network that serves as an endpoint for sending and receiving data across the network.

[0091] Nowadays, the popularity of distributed systems is growing, and there is a strong tendency to design new ones and migrate existing software systems to service-oriented architectures. Such systems have a number of advantages: scalability, fault isolation, language and technology agnosticism, and the ability to be developed by multiple teams simultaneously.

[0092] Systems, designed in a distributed way, consist of many separate applications that communicate with each other through network communication protocols. It results in higher complexity of distributed systems compared to monolithic ones and introduces new challenges.

[0093] Distributed systems are complex and hard to design, which may lead to trade-offs and suboptimal design decisions.

[0094] Distributed systems are heterogeneous. In distributed systems, services can be implemented using different programming languages and technologies and can use various communication protocols and even hardware. This is a serious challenge for observability software since it should support all the technologies that can be met in the system.

[0095] Meanwhile, service-oriented architectures are constantly changing. If multiple teams are involved in the development process, then the system architecture should be well documented, and this documentation should be shared across all development teams and maintained as relevant. Maintaining the documentation up-to-date is a time-consuming task, while a lack of knowledge about system architecture may lead to bad design solutions, misunderstanding between teams and bugs.

[0096] To address these issues, continuous tracers are widely used. Continuous tracers collect information about a system during its lifetime, provide information about interactions between system parts, and describe its architecture. There are two main types of solutions for continuous tracing. The first type uses instrumentation in the application code. The second type relies on packet collect to find out how services interact.

[0097] The most popular implementation among instrumenting tracers is Jaeger. It provides users with specific libraries that allow them to add special code to their services. This code appends headers to hypertext transfer protocol (HTTP) requests in order to track these requests across the system and sends tracing data to a single server, where the tracing data can be interpreted and presented to the users.

[0098] Instrumenting tracers require injection of the special code into the application sources. This requires additional actions from the users’ side. Instrumentation code can contain bugs that can lead to system failures or performance issues.

[0099] Another type of tracer doesn’t require any instrumentation code injection. These tracers use different approaches, which allows them to intercept incoming and outgoing packets. Unlike tracers in the case of instrumentation, these tracers can’t rely on a unique identifier to match requests captured on the sending and receiving sides. Instead, they need to apply special algorithms and approaches based on information about source and destination addresses and ports extracted from collected packets that can show if packets collected on both sides are the same. However, in real-life systems, the direct matching of IP address and port pairs collectedon the server and client is impossible. The vast majority of modern systems use various middleware that hides real communications between services in distributed systems, such as load balancers, proxies, orchestrators (e.g., Kubemetes), and containerization software (e.g., Docker). To overcome this issue, tracers use specific application programming interfaces (APIs) provided by this middleware to obtain information about IP addresses and ports mappings.

[0100] To address the problem of packet matching related to the usage of middleware described above, packet-capturing tracers rely on specific APIs provided by this middleware. This leads to vendor lock. At the same time, most of the solutions in this field rely only on system calls such as close, write, accept, recv, read, and send. Since each process has a different file descriptor for the same socket, using socket file descriptors can’t handle cases when multiple processes use the same socket.

[0101] FIG. 1 is a flowchart of a distributed system analysis method according to an embodiment of the present application. The method is applied to a distributed system. The distributed system includes N computer devices and a server device, where a kernel space of each of the N computer devices includes an eBPF code, and the eBPF code is configured to collect a data packet upon the occurrence of a target event. The method is executed by a first computer device, and the first computer device is any one of the N computer devices, N being an integer greater than 1.

[0102] 110, collecting a data packet when the target event occurs.

[0103] The first computer device can include a tracing agent, and the tracing agent is configured to inject the eBPF code into a kernel space of the first computer device.

[0104] The target event can include any one or more of the following: a socket is opened or closed, a connection between sockets is established, a packet is sent or received, an internet protocol virtual server (IP VS) or internet protocol tables (IP Tables) redirects a packet.

[0105] 120, sending the data packet to the server device.

[0106] The server device is configured to collect data packets from the N computer devices and analyze dependencies between services associated with the data packets. In one possible implementation, the dependencies can be a dependency table or a dependency graph.

[0107] FIG. 2 is a flowchart of a distributed system analysis method according to an embodiment of the present application. The method is applied to a distributed system. Thedistributed system includes N computer devices and a server device, where a kernel space of each of the N computer devices includes an eBPF code, and the eBPF code is configured to collect a data packet upon the occurrence of a target event. The method is executed by the server device, and N is an integer greater than 1.

[0108] 210, receiving multiple data packets from the N computer devices.

[0109] The target event can include any one or more of the following: a socket is opened or closed, a connection between sockets is established, a data packet is sent or received, an internet protocol virtual server (IP VS) or internet protocol tables (IP Tables) redirects a data packet.

[0110] Each of the N computer devices can include a tracing agent that is configured to inject the eBPF code into the kernel space of the computer device. The eBPF code is executed when a kernel function associated with the target event is called.

[0111] 220, generating dependencies between services associated with the multiple data packets.

[0112] The eBPF code that was executed during the request routing makes it possible to find the same data packet collected on a client and a server by matching all four IP address and port pairs that participated in routing. The four IP address and port pairs refer to a sender’s source IP address and port, and destination IP address and port and a receiver's source IP address and port, and destination IP address and port.

[0113] In one possible implementation scenario, the dependencies can be a dependency table or a dependency graph.

[0114] In a possible implementation, the generating dependencies between services associated with the multiple data packets includes: matching source IP addresses and ports, and destination IP addresses and ports of the multiple data packets; and generating the dependencies between the services associated with the multiple data packets based on the matching.

[0115] The server device can analyze data packets from different computers and obtain dependencies between services associated with the data packets. Analyzing these dependencies can help improve the performance and reliability of the distributed system.

[0116] FIG. 3 is a schematic diagram of a distributed system according to an embodiment of the present application.

[0117] The target distributed system can contain multiple hosts, such as host A and host B, where each host can contain any number of virtual machines. Tracing agents are deployed on hosts that need to be observed. Each host can have only one tracing agent deployed.

[0118] In an operating system, a virtual memory is usually divided into a user space and a kernel space. The differences between the user space and the kernel space are privilege levels, resource access rights, memory isolation, execution permission, and execution environment.

[0119] In terms of privilege levels, the user space is a non-privileged mode, while the kernel space is a privileged mode. Applications and user-created processes in the user space are executed in a restricted environment without direct access to low-level hardware and sensitive resources. In contrast, the kernel space operates at a higher privilege level and can access and manage system hardware and sensitive resources.

[0120] In terms of resource access, the user space can only indirectly access underlying resources, such as file systems, network interfaces, and device drivers, through the interfaces and system calls provided by the operating system. It cannot directly access or modify the code and data running in other processes or the kernel space. On the other hand, the kernel space has the capability to directly access and control system resources, enabling it to perform low-level hardware operations and manage system resources.

[0121] In terms of memory isolation, each user space process has its own independent virtual address space, preventing direct access to the memory of other processes. This memory isolation provides security and stability, preventing interference and conflicts between processes. However, the kernel space is shared among all processes and has access to the entire system memory.

[0122] In terms of execution permissions, applications in the user space run with the privileges of regular users and can only execute operations for which they have been authorized. They cannot perform privileged operations that require a higher privilege level, such as direct hardware access or modifying system configurations. In contrast, code running in the kernel space operates at the privilege level of the operating system kernel and can execute privileged operations.

[0123] In terms of execution environment, the user space is where applications and user processes are executed. They focus on specific tasks and functionalities. The user spaceprovides higher-level abstractions and convenience, making application development easier. On the other hand, the kernel space serves as the runtime environment for the operating system kernel, and is responsible for managing system resources, scheduling processes, and providing core system services.

[0124] To detect communications between services, tracing agents are deployed on host machines. The agents use eBPF to set probes on kernel functions that participate in data packet routing. The kernel functions can for example be sock_sendmsg, sock_recvmsg, tcp_connect, inet_accept, etc. Since kernel space exists in only one instance (operating system), there is no need to load the eBPF program multiple times for each application.

[0125] When requests are routed by middleware, it uses specific kernel modules such as IP VS or IP Tables. Setting probes on kernel functions that were executed during the request routing makes it possible to find the same packet collected on a client and a server by matching all four IP address and port pairs that participated in routing. The four IP address and port pairs refer to a sender's source IP address and port, and destination IP address and port and a receiver's source IP address and port, and destination IP address and port.

[0126] The method provided by the present application can use a pointer to a Linux kernel socket structure capable of uniquely identifying a socket, even if the socket is migrated between processes, used concurrently, or the agent is restarted. These structures are allocated in the kernel space, and their position never changes and is always the same for the whole socket lifetime, including the cases mentioned above.

[0127] When the system has started, the tracing agent injects eBPF program into the system kernel. This program contains kprobes that will be triggered when any one or more of the target events occurs: (1) a socket is opened or closed; (2) a connection between sockets is established; (3) a packet is sent or received; and (4) an IPVS or IP Tables redirects the packet.

[0128] This application observes the lifecycle of a socket by capturing corresponding events (open, close, etc.). Tracing accuracy can be improved by handling the reuse of socket identifiers.

[0129] The kprobe is a lightweight kernel debugging technique that can track the execution status of kernel functions. Using kprobe technology can dynamically insert probes into the running kernel, and when the kernel rims to the probe, it can execute the user's predefinedcallback function to collect the required debugging status information without affecting the original execution process of the kernel.

[0130] The collected data is aggregated in batches and sent to the tracing server. The tracing server collects raw data from multiple tracing agents, aggregates it and performs packet matching in order to build a dependency graph between services. The tracing server determines the same packet by matching all four IP address and port pairs participating in routing.

[0131] The dependency graph can be used to find out how services interact, locate bottlenecks and investigate performance issues. The dependency graph can also be replaced with a dependency table or other form, and the present application does not limit the form of the dependencies.

[0132] In some possible implementations, some systems can use the method provided in this application to find dependencies between services and apply graph algorithms to analyze the system architecture and find ways to improve its performance and reliability.

[0133] In some possible implementations, some systems can use the method provided in this application to find dependencies between services and apply artificial intelligence (Al) or machine learning (ML) methods to analyze the system architecture and find ways to improve its performance and reliability.

[0134] This application uses the eBPF to collect arguments passed to kernel functions that participate in IPVS or IP Tables routing to find out how packets are routed when load balancers, containers or orchestrators are used. This unified approach to packet tracing does not rely on any specific API provided by middleware.

[0135] Embodiments of the present application use a pointer to the socket as a unique identifier for a socket. This approach improves trace accuracy in the following cases: the socket is being used by more than one process; the socket is being passed to another process; and the tracing agent has been restarted.

[0136] FIG. 4 is a schematic block diagram of a computing apparatus 1500 according to an embodiment of this application. The computing apparatus 1500 includes an obtaining module 1510 and a sending module 1520.

[0137] The computing apparatus 1500 is applied to a distributed system, where the distributed system includes N computer devices and a server device, a kernel space of each ofthe N computer devices includes an eBPF code, the eBPF code is configured to collect a data packet upon the occurrence of a target event, the apparatus 1500 is set in a first computer device, and the first computer device is any one of the N computer devices, N being an integer greater than 1.

[0138] The obtaining module 1510 is configured to collect a data packet when the target event occurs. The sending module 1520 is configured to send the data packet to the server device, the server device being configured to collect data packets from the N computer devices and analyze dependencies between services associated with the data packets.

[0139] Optionally, the first computer device includes a tracing agent, the tracing agent being configured to inject the eBPF code into a kernel space of the first computer device.

[0140] Optionally, the target event includes any one or more of the following: a socket is opened or closed, a connection between sockets is established, a data packet is sent or received, an IPVS or IP Tables redirects a data packet.

[0141] Optionally, a pointer to the socket is a unique identifier of the socket.

[0142] Optionally, the eBPF code is executed when a kernel function associated with the target event is called.

[0143] FIG. 5 is a schematic block diagram of a computing apparatus 1600 according to an embodiment of this application. The computing apparatus 1600 includes a receiving module 1610 and a processing module 1620.

[0144] The computing apparatus 1600 is applied to a distributed system, where the distributed system includes N computer devices and a server device, a kernel space of each of the N computer devices includes an eBPF code, the eBPF code is configured to collect a data packet upon the occurrence of a target event, and the apparatus 1600 is set in the server device, N being an integer greater than 1.

[0145] The receiving module 1610 is configured to receive multiple data packets from the N computer devices. The processing module 1620 is configured to generate dependencies between services associated with the multiple data packets.

[0146] Optionally, the processing module 1620 is specifically configured to: match source IP addresses and ports, and destination IP addresses and ports of the multiple data packets; and generate the dependencies between the services associated with the multiple data packets basedon the matching.

[0147] Optionally, the dependencies include a dependency graph.

[0148] Optionally, a first computer device includes a tracing agent, the tracing agent being configured to inject the eBPF code into a kernel space of the first computer device, the first computer device being any one of the N computer devices.

[0149] Optionally, the target event includes any one or more of the following: a socket is opened or closed, a connection between sockets is established, a data packet is sent or received, an IPVS or IP Tables redirects a data packet.

[0150] Optionally, a pointer to the socket is a unique identifier of the socket.

[0151] Optionally, the eBPF code is executed when a kernel function associated with the target event is called.

[0152] For example, a "module" can be a software program, a hardware circuit, or a combination of both that implements the above-described functions. Exemplarily, a realization of the obtaining module 1510 is next described as an example of the obtaining module 1510. Similarly, realizations of other modules, such as the sending module 1520, the receiving module 1610, and the processing module 1620 can refer to the realization of the obtaining module 1510.

[0153] As an example of a software functional unit, the obtaining module 1510 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the obtaining module 1510 can include code running on multiple hosts / virtual machines / containers. It is noted that the plurality of hosts / virtual machines / containers are configured to run the code can be distributed in the same region (region) or in different regions. Further, the plurality of hosts / virtual machines / containers are configured to run the code that can be distributed in the same availability zone (AZ) or can be distributed in different AZs, each AZ comprising a single data center or a plurality of geographically proximate data centers. Typically, a region can include multiple AZs.

[0154] Similarly, multiple hosts / virtual machines / containers are configured to run the code that can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, usually a VPC is set up in a region. To communicate between two VPCs in the same region and between VPCs in different regions, a communication gateway should be set up ineach VPC, and the interconnection between VPCs is realized through the communication gateway.

[0155] As an example of a functional unit of hardware, the obtaining module 1510 can include at least one computing device, such as a server and the like. Alternatively, the obtaining module 1510 can be a device utilizing an application-specific integrated circuit (ASIC) implementation, or a programmable logic device (PLD) implementation, and the like. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0156] The plurality of computing devices included in the obtaining module 1510 can be distributed in the same region or can be distributed in different regions. The plurality of computing devices included in the obtaining module 1510 can be distributed in the same AZ or can be distributed in different AZs. Similarly, the plurality of computing devices included in the obtaining module 1510 can be distributed in the same VPC or can be distributed in multiple VPCs. The plurality of computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0157] Thus, the modules of the various examples described in the embodiments of this application are capable of being implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the particular application and design constraints of the technical solution. The skilled professional can use different methods to implement the described functions for each particular application, but such implementations should not be considered outside the scope of this application.

[0158] The computing apparatus provided by the above embodiments is only illustrated by way of example with the division of each of the above-described functional modules when performing the distributed system analysis method. In actual application, the above-described functions can be assigned to be accomplished by different functional modules according to the needs, i.e., the internal structure of the device can be divided into different functional modules in order to accomplish all or part of the above-described functions.

[0159] For example, the obtaining module 1510 can be configured to implement any step in the distributed system analysis method, and the sending module 1520 can be configured toimplement any step in the distributed system analysis method. The steps that the obtaining module 1510, and the sending module 1520 are responsible for realizing may be specified as desired, and the full functions of the computing apparatus are realized by the obtaining module 1510, and the sending module 1520 respectively realizing different steps in the distributed system analysis method.

[0160] In addition, the computing apparatus provided in the above embodiments belongs to the same idea as the distributed system analysis method embodiments, and the specific realization process thereof is detailed in the method embodiments above and will not be repeated herein.

[0161] The distributed system analysis method provided by embodiments of the present application can be performed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a processing unit, a memory, and a memory control unit, the functions and structure of which are then described in detail. The operating system is any one or more computer operating systems that implement business processing through processes, for example, a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a windows operating system. The application layer contains applications such as browsers, address books, word processing software, instant messaging software, and the like. Optionally, the computer system is a handheld device such as a smartphone, or a terminal device such as a personal computer, which is not specifically limited by the present application, as long as it is capable of passing the method provided by the embodiments of the present application. The execution subject of the distributed system analysis method provided by the embodiments of the present application can be a computing device, or, a functional module in the computing device capable of calling a program and executing the program.

[0162] FIG. 6 is a schematic diagram of an architecture of a computing device according to an embodiment of the present application. The computing device 1700 can be a server or a computer or other device with computing capabilities. The computing device 1700 shown in FIG. 6 includes: a processor 1720, a memory 1730, a transceiver 1740, and a bus 1710. Theprocessor 1720, the memory 1730, and the transceiver 1740 are in communication with each other via the bus 1710. The computing device 1700 can be a server or a terminal device. The present application does not limit the number of processors, and memories in the computing device 1700.

[0163] The bus 1710 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in FIG. 6, but it does not mean that there is only one bus or one type of bus. The bus 1710 may include pathways for transferring information between various components of the computing device 1700 (e.g., the memory 1730, the processor 1720, or the transceiver 1740).

[0164] The processor 1720 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0165] The memory 1730 can include volatile memory, such as random access memory (RAM). The processor 1720 can also include non-volatile memory, such as read-only memory (ROM), flash memory, a mechanical hard disk (HDD), or a solid state drive (SSD).

[0166] An executable program code is stored in the memory 1730, and the processor 1720 executes the executable program code to implement the functions of the aforementioned obtaining module 1510, the sending module 1520, the receiving module 1610, or the processing module 1620, respectively, so as to implement the distributed system analysis method described in the above embodiment. That is, the memory 1730 has instructions on it for executing the distributed system analysis method described in the above embodiment.

[0167] The transceiver 1740 uses, for example, but not limited to, a transceiver module such as a network interface card, a transceiver, or the like, to enable communication between the computing device 1700 and other devices or communication networks.

[0168] The structure of the computing device 1700 enumerated above is only an exemplary illustration, and the present application is not limited thereto. The computing device 1700 of embodiments of the present application includes a variety of hardware found in computer systems in the prior art, e.g., the computing device 1700 also includes memory other than the memory 1730, such as, for example, a disk memory, and the like. It should be understood by those skilled in the art that the computing device 1700 can also include other devices necessaryto achieve proper operation. In addition, depending on the specific needs, one of skill in the art should understand that the above-described computing device 1700 may also include other hardware devices for realizing other additional functions. In addition, one of skill in the art should understand that the above-described computing device 1700 may also include only the devices necessary to realize the embodiments of the present application, and need not include all of the devices shown in FIG. 6.

[0169] FIG. 7 is a schematic diagram of an architecture of a computing device cluster according to an embodiment of the present application.

[0170] The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0171] As shown in FIG. 7, the cluster of computing devices described includes at least one computing device 1700. The memory 1730 in one or more of the computing devices 1700 in the cluster of computing devices can hold the same instructions for performing the distributed system analysis method described in the above embodiment.100172] In some possible implementations, some of the instructions for performing the distributed system analysis method described in the above embodiments can also be separately held in the memory 1730 of the one or more computing devices 1700 in the cluster of computing devices. In other words, a combination of the one or more computing devices 1700 can jointly execute instructions for performing the distributed system analysis method described in the above embodiment.

[0173] It is noted that the memories 1730 in the different computing devices 1700 in the cluster of computing devices can store different instructions for performing some of the functions of the computing devices 1700, respectively. That is, the instructions stored in the memories 1730 in the different computing devices 1700 can realize the functions of one or more of the obtaining module 1510 and the sending module 1520, or can realize the functions of one or more of the receiving module 1610 and the processing module 1620.

[0174] In some possible implementations, one or more computing devices in a cluster of computing devices can be connected via a network. Among other things, the network can be awide area network or a local area network, etc. FIG. 8 illustrates one possible implementation. FIG. 8 is a schematic diagram of a connection between computing devices 1700 A and 1700B over a network according to an embodiment of the present application. The two computing devices 1700A and 1700B are connected to each other via a network. Specifically, the connection to said network is made through a communication interface in each computing device. In this class of possible implementations, the memory 1730 in the computing device 1700 A holds instructions for performing the functions of the obtaining module 1510 or the receiving module 1610. At the same time, the memory 1730 in the computing device 1700B holds instructions for performing the functions of the sending module 1520 or the processing module 1620.

[0175] The connection between the clusters of computing devices shown in FIG. 8 can be such that, considering that the distributed system analysis method provided in the present application requires collecting a large number of data packets, it is contemplated that the functions realized by the receiving module 1610 be performed by the computing device 1700B.

[0176] The functions of computing device 1700A illustrated in FIG. 8 can also be accomplished by multiple computing devices 1700. Similarly, the functions of computing device 1700B can be accomplished by multiple computing devices 1700.

[0177] This embodiment also provides a computer program product comprising instructions. Said computer program product can be a software or program product containing instructions that can run on a cluster of computing devices or be stored in any available medium. When run by a cluster of computing devices, it causes the cluster of computing devices to perform the distributed system analysis method provided above, or causes the cluster of computing devices to perform the functions of the computing apparatus 1500 or the computing apparatus 1600 provided above.

[0178] In this embodiment, a computer-readable storage medium is also provided. The computer-readable storage medium can be any usable medium that a computing device can store or a data storage device such as a data center containing one or more usable media. The usable medium can be a magnetic medium (e.g., floppy disk, hard disk, and tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state drive). The computer- readable storage medium includes instructions that, when executed on the computing device,cause the computing device to perform the distributed system analysis method provided above.

[0179] In this embodiment, a computer-readable storage medium is also provided. The computer-readable storage medium can be any usable medium that a computing device can store or a data storage device such as a data center containing one or more usable media. The usable medium can be a magnetic medium, (e.g., floppy disk, hard disk, and tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state drive). The computer- readable storage medium includes instructions that, when executed by the cluster of computing devices, cause the cluster of computing devices to perform the distributed system analysis method provided above.

[0180] In various embodiments of the present application, the magnitude of the serial number of each of the above processes does not imply the order of execution, and the order of execution of each process shall be determined by its function and inherent logic without constituting any limitation of the process of implementation of the embodiments of the present application.

[0181] A person of ordinary skill in the art may be aware that, in combination with the examples described in the embodiments disclosed in this specification, units and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on particular applications and design constraints of the technical solutions. A person skilled in the art may use different methods to implement the described functions for each particular application, but it should not be considered that the implementation goes beyond the scope of this application.

[0182] It may be clearly understood by a person skilled in the art that, for the purpose of convenient and brief description, for a detailed working process of the foregoing system, apparatus, and unit, refer to a corresponding process in the foregoing method embodiment. Details are not described herein again.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the described apparatus embodiment is merely an example. For example, the unit division is merely logical function division and may be other division in actual implementation.For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented through some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.

[0184] The units described as separate parts may be or may not be physically separate, and parts displayed as units may be or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0185] In addition, functional units in the embodiments of this application may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

[0186] When the functions are implemented in a form of a software functional unit and sold or used as an independent product, the functions may be stored in a computer readable storage medium. Based on such an understanding, the technical solutions in this application essentially, or the part contributing to the prior art, or some of the technical solutions may be implemented in a form of a software product. The computer software product is stored in a storage medium, and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or some of the steps of the methods described in the embodiments of this application. The foregoing storage medium includes: any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc.

[0187] The foregoing descriptions are merely specific implementations of this application, but are not intended to limit the protection scope of this application. Any variation or replacement readily figured out by a person skilled in the art within the technical scope disclosed in this application shall fall within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

CLAIMSWhat is claimed is:

1. A distributed system analysis method, applied to a distributed system, wherein the distributed system comprises N computer devices and a server device, a kernel space of each of the N computer devices comprises an extended Berkley packet filter (eBPF) code, the eBPF code is configured to collect a data packet upon the occurrence of a target event, the method is executed by a first computer device, and the first computer device is any one of the N computer devices, N being an integer greater than 1, the method comprising: collecting a data packet when the target event occurs; and sending the data packet to the server device, the server device being configured to collect data packets from the N computer devices and analyze dependencies between services associated with the data packets.

2. The method according to claim 1, wherein the first computer device comprises a tracing agent, the tracing agent being configured to inject the eBPF code into a kernel space of the first computer device.

3. The method according to claim 1 or 2, wherein the target event comprises any one or more of the following: a socket is opened or closed, a connection between sockets is established, a data packet is sent or received, an internet protocol virtual server (IP VS) or internet protocol tables (IP Tables) redirects a data packet.

4. The method according to claim 3, wherein a pointer to the socket is a unique identifier of the socket.

5. The method according to any one of claims 1 to 4, wherein the eBPF code is executed when a kernel function associated with the target event is called.

6. A distributed system analysis method, applied to a distributed system, wherein the distributed system comprises N computer devices and a server device, a kernel space of each of the N computer devices comprises an extended Berkley packet filter (eBPF) code, the eBPF code is configured to collect a data packet upon the occurrence of a target event, and the method is executed by the server device, N being an integer greater than 1, the method comprising:receiving multiple data packets from the N computer devices; and generating dependencies between services associated with the multiple data packets.

7. The method according to claim 6, wherein the generating dependencies between services associated with the multiple data packets comprises: matching source internet protocol (IP) addresses and ports, and destination IP addresses and ports of the multiple data packets; and generating the dependencies between the services associated with the multiple data packets based on the matching.

8. The method according to claim 6 or 7, wherein the dependencies comprise a dependency graph.

9. The method according to any one of claims 6 to 8, wherein a first computer device comprises a tracing agent, the tracing agent being configured to inject the eBPF code into a kernel space of the first computer device, the first computer device being any one of the N computer devices.

10. The method according to any one of claims 6 to 9, wherein the target event comprises any one or more of the following: a socket is opened or closed, a connection between sockets is established, a data packet is sent or received, an internet protocol virtual server (IP VS) or internet protocol tables (IP Tables) redirects a data packet.

11. The method according to claim 10, wherein a pointer to the socket is a unique identifier of the socket.

12. The method according to any one of claims 6 to 11 , wherein the eBPF code is executed when a kernel function associated with the target event is called.

13. A computing apparatus, applied to a distributed system, wherein the distributed system comprises N computer devices and a server device, a kernel space of each of the N computer devices comprises an extended Berkley packet filter (eBPF) code, the eBPF code is configured to collect a data packet upon the occurrence of a target event, the apparatus is set in a first computer device, and the first computer device is any one of the N computer devices, N being an integer greater than 1, the apparatus comprising: an obtaining module configured to collect a data packet when the target event occurs; and a sending module configured to send the data packet to the server device, the server devicebeing configured to collect data packets from the N computer devices and analyze dependencies between services associated with the data packets.

14. The computing apparatus according to claim 13, wherein the first computer device comprises a tracing agent, the tracing agent being configured to inject the eBPF code into a kernel space of the first computer device.

15. The computing apparatus according to claim 13 or 14, wherein the target event comprises any one or more of the following: a socket is opened or closed, a connection between sockets is established, a data packet is sent or received, an internet protocol virtual server (IP VS) or internet protocol tables (IP Tables) redirects a data packet.

16. The computing apparatus according to claim 15, wherein a pointer to the socket is a unique identifier of the socket.

17. The computing apparatus according to any one of claims 13 to 16, wherein the eBPF code is executed when a kernel function associated with the target event is called.

18. A computing apparatus, applied to a distributed system, wherein the distributed system comprises N computer devices and a server device, a kernel space of each of the N computer devices comprises an extended Berkley packet filter (eBPF) code, the eBPF code is configured to collect a data packet upon the occurrence of a target event, and the apparatus is set in the server device, N being an integer greater than 1, the apparatus comprising: a receiving module configured to receive multiple data packets from the N computer devices; and a processing module configured to generate dependencies between services associated with the multiple data packets.

19. The computing apparatus according to claim 18, wherein the processing module is specifically configured to: match source internet protocol (IP) addresses and ports, and destination IP addresses and ports of the multiple data packets; and generate the dependencies between the services associated with the multiple data packets based on the matching.

20. The computing apparatus according to claim 18 or 19, wherein the dependencies comprise a dependency graph.

21. The computing apparatus according to any one of claims 18 to 20, wherein a first computer device comprises a tracing agent, the tracing agent being configured to inject the eBPF code into a kernel space of the first computer device, the first computer device being any one of the N computer devices.

22. The computing apparatus according to any one of claims 18 to 21, wherein the target event comprises any one or more of the following: a socket is opened or closed, a connection between sockets is established, a data packet is sent or received, an internet protocol virtual server (IPVS) or internet protocol tables (IP Tables) redirects a data packet.

23. The computing apparatus according to claim 22, wherein a pointer to the socket is a unique identifier of the socket.

24. The computing apparatus according to any one of claims 18 to 23, wherein the eBPF code is executed when a kernel function associated with the target event is called.

25. A distributed system, comprising a first computer device and a server device, wherein the first computer device is configured to perform the method according to any one of claims 1 to 5, and the server device is configured to perform the method according to any one of claims 6 to 12.

26. A computing device cluster, comprising at least one computing device, wherein the computing device comprises a processor and a memory coupled with the processor, wherein the memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory, so that the computing device cluster executes the method according to any one of claims 1 to 5 or the method according to any one of claims 6 to 12.

27. A computer readable storage medium storing instructions that, when run on a computing device cluster, enable the computing device cluster to perform the method according to any one of claims 1 to 5 or the method according to any one of claims 6 to 12.

28. A computer program product, wherein when the computer program product is run on a computing device cluster, the computing device cluster is enabled to perform the method according to any one of claims 1 to 5 or the method according to any one of claims 6 to 12.

Images