Log data processing method and device, electronic equipment, medium and program product

Abstract

The present disclosure provides a log data processing method and device, electronic equipment, medium and program product, relates to the technical field of computers, in particular to the technical fields of cloud computing, intelligent chips and the like. The log data processing method comprises: in response to detecting a log offloading instruction of a virtualization component, generating second log data according to first log data to be offloaded in the virtualization component and source data of the first log data, wherein the virtualization component represents a component in a host for providing virtualization services; offloading the second log data to a data processor through a log offloading channel matched with the virtualization component, so that the data processor stores the first log data to a storage path corresponding to the source data according to the second log data.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to cloud computing, smart chips, and other related technologies. In particular, it relates to a log data processing method, apparatus, electronic device, medium, and program product. Background Technology

[0002] With the development of technology, data processing units (DPUs) are increasingly being used in cloud computing scenarios. A DPU runs its own operating system and has multiple different working modes. It can be used as a regular network interface card (NIC) or work in conjunction with the host machine's operating system, such as offloading various functions from the host machine to the DPU.

[0003] However, in a virtualization framework where the DPU and the host machine collaborate, the processing of log data can consume excessive host machine resources, especially the log data of the virtualization components within the host machine. Summary of the Invention

[0004] This disclosure provides a log data processing method, apparatus, electronic device, medium, and program product.

[0005] According to one aspect of this disclosure, a log data processing method is provided, comprising: in response to detecting a log unloading instruction of a virtualization component, generating second log data based on first log data to be unloaded in the virtualization component and source data of the first log data, wherein the virtualization component represents a component in the host machine used to provide virtualization services; and unloading the second log data to a data processor through a log unloading channel matching the virtualization component, such that the data processor stores the first log data to a storage path corresponding to the source data based on the second log data.

[0006] According to another aspect of this disclosure, a log data processing method is provided, comprising: in response to receiving second log data obtained from a host machine, storing first log data to a storage path corresponding to the source data of the first log data; wherein the second log data is generated by the host machine in response to detecting a log unloading instruction of a virtualization component, based on the first log data to be unloaded and the source data, the virtualization component representing a component in the host machine used to provide virtualization services; and the second log data is transmitted through a log unloading channel matched between the host machine and the virtualization component.

[0007] According to another aspect of this disclosure, a log data processing apparatus is provided, comprising: a generation module, configured to generate second log data in response to detecting a log unloading instruction of a virtualization component, based on first log data to be unloaded in the virtualization component and source data of the first log data, wherein the virtualization component represents a component in a host machine used to provide virtualization services; and an unloading module, configured to unload the second log data to a data processor through a log unloading channel matching the virtualization component, such that the data processor stores the first log data to a storage path corresponding to the source data based on the second log data.

[0008] According to another aspect of this disclosure, a log data processing apparatus is provided, comprising: a storage module, configured to, in response to receiving second log data obtained from a host machine, store first log data to a storage path corresponding to the source data of the first log data; wherein the second log data is generated by the host machine in response to detecting a log unloading instruction of a virtualization component, based on the first log data to be unloaded and the source data, the virtualization component representing a component in the host machine used to provide virtualization services; and the second log data is transmitted through a log unloading channel matched between the host machine and the component.

[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0011] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 This illustration schematically depicts an exemplary system architecture applicable to log data processing methods and apparatus according to embodiments of the present disclosure;

[0015] Figure 2 A flowchart illustrating a log data processing method according to an embodiment of the present disclosure is shown schematically.

[0016] Figure 3A This schematic diagram illustrates a data flow diagram for generating second log data according to a first embodiment of the present disclosure;

[0017] Figure 3B This illustration schematically shows a data flow diagram for generating second log data according to a second embodiment of the present disclosure;

[0018] Figure 3C This illustration schematically shows a data flow diagram for generating second log data according to a third embodiment of the present disclosure;

[0019] Figure 4A This diagram illustrates an application scenario of unloading second log data according to an embodiment of the present disclosure.

[0020] Figure 4B This diagram illustrates an application scenario of unloading second log data according to another embodiment of the present disclosure.

[0021] Figure 5 A flowchart illustrating the generation of first log data based on first log sub-data and second log sub-data according to an embodiment of the present disclosure is shown.

[0022] Figure 6 The illustration shows a schematic diagram of the process for storing and unloading log data for a virtual machine according to an embodiment of the present disclosure;

[0023] Figure 7 A flowchart illustrating a log data processing method according to another embodiment of the present disclosure is shown schematically;

[0024] Figure 8 A data flow diagram illustrating the storage of first log data according to another embodiment of this disclosure is shown schematically;

[0025] Figure 9 A block diagram of a log data processing apparatus according to an embodiment of the present disclosure is shown schematically;

[0026] Figure 10 A block diagram of a log data processing apparatus according to another embodiment of the present disclosure is shown schematically;

[0027] Figure 11 A block diagram of an electronic device suitable for implementing a log data processing method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0028] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0029] In the embodiments disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0030] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0031] The following will explain the terms used in this disclosure in order to help to understand this disclosure.

[0032] Computer virtualization technology is a technical term in the field of cloud computing. It usually refers to a technology that runs computing components on a virtual platform rather than a real hardware platform.

[0033] Virtualization encompasses two virtualization concepts: the host machine and the guest machine. On top of the host machine, at least one virtual machine can be simulated using technologies such as hardware virtualization. Applications can run either on the host machine or in the virtual machine, and the applications are largely unaware of the differences in the underlying operating system.

[0034] A virtualization emulator is a virtual machine monitor (VMM) on the host machine used to simulate virtual machines. It uses virtualization technology to simulate a central processing unit (CPU) and provides a series of hardware models, allowing the virtual machine's operating system to believe it is interacting directly with the hardware, thus enabling the virtual machine to function properly. For example, a virtualization emulator could be QEMU (QuickEmulator).

[0035] The kernel-based virtual machine (KVM) is an open-source system virtualization module that enables virtualization in collaboration with qemu based on the host Linux operating system.

[0036] Virtualization management tools are used to manage virtual machines. For example, the common virtualization management tool libvirt can manage virtual machines based on KVM.

[0037] The system daemon (systemd) is used to manage system services in the Linux operating system, such as processing log data generated by these services.

[0038] The netconsole is a feature implemented in the Linux operating system kernel that allows kernel debugging information, such as system service information, to be transmitted over a network.

[0039] Liblogging is an open-source logging framework that provides standalone logging functionality.

[0040] The log service process manages log data. In Linux operating systems, a local log service process (such as syslog or rsyslog) typically receives and filters data from various applications on the virtual machine and host machine, processes it for formatting, and then saves it to a local file or forwards the log data through certain communication methods. In existing virtualization architectures where the DPU and host machine coexist, the host-side log service process can process and save log data on the host machine; alternatively, multiple components on the host machine can communicate with the DPU-side log service process via the local log service process to forward log data to the DPU.

[0041] A Data Processing Unit (DPU) is a multi-core processor with high-performance computing capabilities. The DPU can offload applications related to artificial intelligence and storage from other chips in the host machine (such as the CPU and Graphics Processing Unit (GPU)), thereby improving the host machine's data processing performance and reducing the host load.

[0042] A serial port (SP), also known as a serial interface, is an expansion interface that uses serial communication.

[0043] Currently, various cloud computing vendors offload some computing tasks, network virtualization, and storage virtualization to the DPU to reduce virtualization overhead on the host machine, thereby minimizing interference and resource consumption on the host. However, for most architectures, some tasks still exist on the host to ensure the operation of virtualization functions, such as the virtualization management tool libvirt and the virtualization emulation platform qemu. The log data generated by these components can help in troubleshooting.

[0044] However, the log data generated by the aforementioned virtualization components is controlled by the host machine's log service process, and after processing, it is stored locally on the host machine. To view this log data, it is necessary to access the host machine and view it through the log service process, which can lead to instability in the virtual machine's operating environment. Especially when the virtualization component's log data is very large, some erroneous operations may cause the host machine to crash. Furthermore, directly viewing and analyzing logs on the host machine can also lead to excessive consumption of host machine resources, causing unpredictable impacts on the host machine.

[0045] In some log data offloading schemes, the log service process on the host machine receives and processes log data generated by the virtualization components. Then, the host-side log service process offloads the log data to the DPU via communication with the DPU-side log service process. However, in this scheme, if there are multiple virtual machines on the host machine, the DPU cannot distinguish each virtual machine and save it to a separate log file, making log data viewing inconvenient. Furthermore, in this scheme, the host-side log service process has complex log processing, routing, formatting, and storage functions, which require additional resources, leading to excessive host machine resource consumption. As an independent daemon process, the log service process also consumes additional system resources to maintain operation, especially when handling large amounts of log input / output (I / O) or complex filtering rules. In these cases, the host-side log service process consumes significant host machine resources and impacts system stability.

[0046] Therefore, this disclosure provides a log data processing method, including: in response to detecting a log offload instruction from a virtualization component, generating second log data based on first log data to be offloaded from the virtualization component and source data of the first log data, wherein the virtualization component represents a component in the host machine used to provide virtualization services; and offloading the second log data to a data processor through a log offload channel matching the virtualization component, so that the data processor stores the first log data to a storage path corresponding to the source data based on the second log data. Through the above embodiments, by using a log offload channel matching the virtualization component, log data generated by the virtualization component can be directly offloaded to the data processor without relying on the host machine's log process service, reducing resource consumption on the host side and facilitating viewing of the corresponding virtualization component's log data on the DPU side.

[0047] Figure 1 The illustration schematically depicts an exemplary system architecture applicable to log data processing methods and apparatus according to embodiments of the present disclosure.

[0048] like Figure 1As shown, the system architecture 100 of this embodiment includes a host machine M1 and a data processor M2. The host machine M1 includes a virtualization component M110, and the data processor M2 is also known as a DPU with network, storage, and function offloading capabilities. It should be noted that... Figure 1 The number of virtualization components shown is for illustrative purposes only. Host machine M1 can have one or more virtualization components at the same time, each used to provide one or more virtualization services.

[0049] The virtualization component M110 of the host machine M1 can generate second log data based on the first log data to be unloaded and the source data of the first log data in the virtualization component M110, and unload the second log data to the data processor M2 through the log unloading channel matched with the virtualization component M110. If there are one or more virtualization components in the host machine M1, each virtualization component can process its own first log data into second log data through the above process and unload it to the data processor M2, which will not be elaborated further here.

[0050] Figure 2 A flowchart illustrating a log data processing method according to an embodiment of this disclosure is shown schematically. Figure 2 As shown, Embodiment 200 includes operations S210 to S220.

[0051] In operation S210, in response to detecting a log unloading command from the virtualization component, second log data is generated based on the first log data to be unloaded from the virtualization component and the source data of the first log data.

[0052] Virtualization components represent components within the host machine used to provide virtualization services. For example, virtualization components can be components like qemu or libvirt used to emulate or manage virtual machines; alternatively, virtualization components can be foundational components that ensure the host machine can provide virtualization services, such as components that manage system services or the operating system kernel.

[0053] The first log data to be uninstalled can be generated by the virtualization component or received from other components. The first log data generated by the virtualization component is typically used to record the virtualization component's own operational status; receiving the first log data from other components facilitates the virtualization component's management of the virtual machine's log data. For example, the first log data to be uninstalled can be log data generated by QEMU itself, log data received by QEMU from the virtual machine, or log data generated by libvirt itself.

[0054] Source data is used to characterize the source of the first log data. Source data can include at least one level of source, such as component source, process source, virtual machine source, etc.

[0055] In one embodiment, second log data can be generated based on first log data and source data according to predetermined rules. For example, the first log data and source data can be combined into second log data using a predetermined combination method; or, the source data can be incorporated into a data header according to a predetermined data transmission format, and the second log data can be generated based on the data header and the first log data.

[0056] The log unloading command can be triggered when the virtualization component generates or receives the first log data. The log unloading command can be any command format supported by the virtualization component, such as a command format adapted to the host machine's runtime environment. In response to detecting the virtualization component's log unloading command, the log data unloading function can be enabled to process the first log data into second log data and unload it.

[0057] During operation S220, the second log data is offloaded to the data processor through a log offload channel that matches the virtualization component, so that the data processor stores the first log data to the storage path corresponding to the source data based on the second log data.

[0058] Log offloading channels are used for log data transmission and can be interfaces, network channels, etc. A log offloading channel matching a virtualization component can be a dedicated channel between the virtualization component and the DPU for transmitting secondary log data from that virtualization component.

[0059] For example, the log offloading channel can be a newly created interface based on a predetermined transport protocol, or it can reuse an existing interface between the host machine and the data processor (DPU).

[0060] In a virtualization architecture where the DPU and host machine coexist, "offloading" can be understood as transferring functions from the host machine to the DPU for implementation; and / or transferring data from the host machine to the DPU for storage and management via virtualization storage technology. In this embodiment, offloading the second log data to the data processor can be understood as storing the second log data in the DPU, where the DPU performs complex log processing, routing, formatting, and storage functions. For example, the DPU receives the second log data, parses it into first log data and source data, and, through filtering operations, stores the first log data in the storage path corresponding to the source data. This storage path can be within the DPU or external to the DPU, such as storage paths on other hardware. The DPU can determine the external storage path and store the first log data there.

[0061] In one embodiment, after determining the second log data, a log offloading channel can be initialized between the virtualization component and the data processor. After initialization, the second log data is offloaded through the log offloading channel matched to the virtualization component. In another embodiment, the log offloading channel can be initialized between the virtualization component and the data processor when the virtualization component starts. When offloading is required, the second log data is directly offloaded through the log offloading channel matched to the virtualization component.

[0062] In the embodiments of this disclosure, the first log data generated by the virtualization component is directly processed into the second log data and unloaded to the data processor through a log offloading channel that matches the virtualization component. This eliminates the need for the host machine's log process service, thereby reducing the resource consumption on the host machine side and simplifying the host machine system architecture. The processing of log data is uniformly implemented by the DPU, making the system on the host machine more lightweight.

[0063] Furthermore, the generation of second log data based on the first log data and the source data enables the DPU to store the first log data in the storage path corresponding to the source data, facilitating log data management and querying by the virtualization component and simplifying log data viewing operations on the DPU side.

[0064] According to embodiments of this disclosure, generating second log data based on first log data to be unloaded in a virtualization component and source data of the first log data includes: generating a storage path identifier based on the source data of the first log data; and generating second log data in a predetermined format based on the storage path identifier and the first log data; wherein the predetermined format includes data formats supported by the log service process of the data processor.

[0065] The storage path identifier is used to characterize the storage path where the DPU stores the first log data. For example, the storage path identifier can be a partial character extracted from the source data; or, the storage path identifier can be an IP address determined from the source data.

[0066] The source data may include information such as the IP (Internet Protocol) address and identifier of the source of the first log data. In addition, the source data may also include information such as the importance of the first log data, the process name, and the process ID; in this case, a storage path identifier can be generated based on at least one piece of information from the source data.

[0067] There is a unique mapping relationship between the source data and the storage path. Therefore, the storage path identifier can be generated based on the source data of the first log data.

[0068] Generating second log data in a predetermined format based on the storage path identifier and the first log data can be achieved by using the storage path identifier as a data header, which together with the first log data constitutes the second log data. The data header of a preset length can be limited according to actual needs, so that the preset length of the data header in the second log data can reflect the source data.

[0069] For example, the storage path identifier can be the process name of the current virtualization component or a custom name "pr1", and the second log data generated based on the storage path identifier and the first log data "XXX-time-log" can be "pr1-XXX-time-log".

[0070] It should be noted that the embodiments disclosed herein remove the log service process on the host side to reduce the load on the host machine. However, the DPU side can receive the second log data offloaded from the virtualization components on the host machine through the log service process. Therefore, when generating the second log data, the data format supported by the log service process is used to ensure that the DPU side can receive and process the second log data.

[0071] For example, the logging service process on the DPU side could be rsyslog. The predefined format for the second log data generated on the host side could be the syslog data format, that is, the data format supported by the logging service process rsyslog.

[0072] In the embodiments of this disclosure, a storage path identifier is generated based on the source data of the first log data; and second log data in a predetermined format is generated based on the storage path identifier and the first log data, so that when the log data is unloaded from the host side, the data source of the first log data can be included. While realizing the log data unloading function, it supports storing the first log data according to the source data, which facilitates log data management.

[0073] Figure 3A A schematic diagram illustrating the data flow for generating second log data according to a first embodiment of this disclosure is shown. Figure 3A As shown, in the first embodiment 300A, the storage path identifier 302 can be determined based on the source data 301 of the first log data 303, and then the second log data 304 is generated based on the storage path identifier 302 and the first log data 303. Afterwards, the second log data 304 can be offloaded to the DPU through a log offload channel that matches the virtualization component.

[0074] In one specific embodiment, generating a storage path identifier based on the source data of the first log data includes: generating a storage path identifier based on first source sub-data, wherein the first source sub-data is used to characterize the virtualization component that generates the first log data; or, generating a storage path identifier based on the first source sub-data and second source sub-data, wherein the second source sub-data is used to characterize the process in the virtualization component that generates the first log data.

[0075] The source data may include first source sub-data and / or second source sub-data. The first source sub-data may be the identifier and / or IP address of various virtualization components based on different operating systems, such as the identifiers of components like qemu, libvirt, and Xen. The second source sub-data may be the identifier of the process that generated the first log data; for example, it may be the process name and / or process ID. Within a virtualization component, multiple processes (i.e., applications) can be virtualized; therefore, the second source sub-data can be used to distinguish the process generating the first log data within the virtualization component.

[0076] For example, a storage path identifier can be generated solely from the first source sub-data, and second log data can be generated based on the storage path identifier and the first log data to achieve DPU log data storage at the virtualization component granularity. Alternatively, a storage path identifier can be generated from both the first and second source sub-data, and second log data can be generated based on the storage path identifier and the first log data to achieve DPU log data storage at the intra-virtualization component process granularity. The operation of generating the storage path identifier has been described above and will not be repeated here.

[0077] In embodiments of this disclosure, a storage path identifier is generated based on the first source sub-data, or the first source sub-data plus the second source sub-data, so that the unloaded second log data can be unloaded at the virtualization component granularity or process granularity, which facilitates log data management and querying.

[0078] Figure 3B A schematic diagram illustrating the data flow for generating second log data according to a second embodiment of this disclosure is shown. Figure 3B As shown, in the second embodiment 300B, the source data of the first log data 303 may include: first source sub-data 3011 and second source sub-data 3012. A storage path identifier 302 can be determined based on the first source sub-data 3011 and the second source sub-data 3012. Then, second log data 304 is generated based on the storage path identifier 302 and the first log data 303. Afterwards, the second log data 304 can be offloaded to the DPU through a log offload channel that matches the virtualization component.

[0079] In another specific embodiment, the virtualization component includes: a first component for simulating or managing a virtual machine; the method further includes: when the first log data is generated by a virtual machine simulated by the first component: generating a storage path identifier based on third source sub-data and fourth source sub-data, the third source sub-data being used to characterize the first component and the fourth source sub-data being used to characterize the virtual machine; or, generating a storage path identifier based on third source sub-data, fourth source sub-data and fifth source sub-data, wherein the fifth source sub-data characterizes the process in the virtual machine used to generate the first log data.

[0080] For a first component used to simulate or manage virtual machines, the first log data can be generated by the first component itself or by the virtual machine simulated by the first component. On a single host machine, the first component can simulate or manage multiple virtual machines, and each virtual machine can run multiple processes. When the first log data is generated by a virtual machine simulated by the first component, the first component can obtain the first log data to be unloaded from the virtual machine, and characterize the first component through third-source sub-data, the virtual machine through fourth-source sub-data, and the process within the virtual machine that generated the first log data through fifth-source sub-data.

[0081] For example, similar to the first source sub-data, the third source sub-data could be the IP address of the first component, the component identifier, etc., such as the component identifier being qemu. The fourth source sub-data could be the identifier of a virtual machine, such as virtual machine number 2.

[0082] It should be noted that the first source sub-data and the third source sub-data can be the same or different. When the first source sub-data and the third source sub-data are different, a new identifier can be added to indicate that the first component is a component that manages or simulates a virtual machine, and that the first log data is generated by the virtual machine. Alternatively, when the first source sub-data and the third source sub-data are the same, the presence or absence of fourth sub-data can be used to determine whether the first log data is generated by the first component or obtained from the virtual machine.

[0083] In one embodiment, a storage path identifier can be generated based on third-source sub-data and fourth-source sub-data, or it can be generated based on third-source sub-data, fourth-source sub-data, and fifth-source sub-data. Then, second log data is generated based on the storage path identifier and the first log data. The operation for generating the storage path identifier is as described above and will not be repeated here.

[0084] In the embodiments of this disclosure, a storage path identifier is generated based on the third source sub-data + fourth source sub-data, or the third source sub-data + fourth source sub-data + fifth source sub-data, so that the unloaded second log data can be unloaded at the two-level virtual machine granularity or at the three-level virtual machine intra-process granularity. Log identifiers are implemented on the host side of the log data unloading, thereby facilitating log data management and querying.

[0085] Figure 3C A schematic diagram illustrating the data flow for generating second log data according to a third embodiment of this disclosure is shown. Figure 3C As shown, in the third embodiment 300C, the source data of the first log data 303 may include: third source sub-data 3013, fourth source sub-data 3014, and fifth source sub-data 3015. Based on the third source sub-data 3013, fourth source sub-data 3014, and fifth source sub-data 3015, a storage path identifier 302 can be determined. Then, based on the storage path identifier 302 and the first log data 303, second log data 304 is generated. Afterward, the second log data 304 can be offloaded to the DPU through a log offload channel that matches the virtualization component.

[0086] For different virtualization components, the second log data can be offloaded to the DPU through a log offload channel that matches the virtualization component. However, for various virtualization components, there may be multiple specific log offload channels and offload methods. To facilitate understanding of the process of offloading the second log data to the DPU, the following explanation will use the first, second, and third components of the virtualization component as examples.

[0087] According to an embodiment of this disclosure, for the first component, operation S220 includes: unloading second log data to the data processor through a second log unloading interface between the first component and the data processor, wherein the second log unloading interface is created using a log library framework embedded in the first component.

[0088] The first component in user space can be a user-space application, such as libvirt or qemu.

[0089] A logging library framework can be a lightweight framework solely for logging data, embedded in the compilation process of the first component. For example, a logging library framework could be the open-source, lightweight framework liblogging that logs data. A logging library framework embedded in the first component's compilation process can be considered part of the first component, not a separate process; therefore, the host machine does not need to expend additional resources to keep the process alive.

[0090] The second log offloading interface is a new interface added to the first component. It is used to generate second log data from the first component's global log data and source data, and then offload the second log data to the DPU. For example, the global first log data includes the first log data generated by the first component and the first log data obtained from the virtual machine.

[0091] The second log offloading interface can offload second log data through a predefined transport protocol. For example, it can offload second log data using a connectionless transport layer protocol to reduce the impact of network fluctuations or congestion on the host machine and virtual machine during log data offloading. For instance, it can use the User Datagram Protocol (UDP) to establish a network connection and transmit data with the log service process rsyslog on the DPU side.

[0092] For example, when compiling the first component, a second log offloading interface can be created using a logging library framework embedded in the first component. This second log offloading interface can be a socket-based interface. Upon starting the first component, it can call the newly added second log offloading interface for initialization, providing its source data, such as an identifier. The logging library framework embedded in the first application creates a UDP socket and related socket information based on the source data from the first component, and returns the initialized second log offloading interface to the first component. Subsequent log data can then be directly offloaded through this second log offloading interface. For example, the above initialization process can be implemented using the liblogging logging library framework.

[0093] Furthermore, for the first component embedded with the logging library framework, the first component utilizes the logging library framework to add a driver to generate second log data based on the first log data and the source data. For example, taking the DPU-side logging service process as rsyslog, the added driver can generate a syslog-formatted storage path identifier based on the source data, and use the storage path identifier as a data header to combine with the first log data to form the second log data. Subsequently, the second log data is unloaded to the DPU through the second log unloading interface.

[0094] In embodiments of this disclosure, by embedding a log library framework supporting simple log data recording into the first component, and using the second log unloading interface created with this log library framework, a unified interface can be provided for the entire first component. This allows all first log data to be unloaded within the first component to generate second log data within the first component, and the logs to be unloaded through the second log unloading interface. In this way, the simple log data recording function consumes fewer resources than existing log service processes, reducing resource consumption on the host system's central processing unit (CPU) and memory. Furthermore, the unified second log unloading interface for the entire first component also reduces the redundant resource consumption of creating and initializing interfaces, thereby reducing the consumption of host system resources.

[0095] According to an embodiment of this disclosure, the virtualization component includes a second component, which is used to support system services of the host machine; for the second component, operation S220 includes: calling the system service manager of the second component, and unloading the second log data to the data processor through a first log unloading interface created by the system service manager, wherein the first log unloading interface is created by the system service manager according to the interface configuration information in the first configuration file.

[0096] For example, for the Linux operating system, the second component is used to support system services of the Linux operating system on the host machine. The second component can be a kernel component, and the system service manager of the second component can be the operating system's native system service manager, such as systemd.

[0097] The system service manager can provide logging services. Therefore, after removing the separate logging service process on the host side, the system service manager can be modified to create a new first log offloading interface for the second log data to be offloaded to the DPU.

[0098] The first log offloading interface is created by the system service manager based on the interface configuration information in the first configuration file. In one embodiment, the first log offloading interface can be a socket-based interface based on the UDP protocol. The first configuration file records the interface configuration information for message services supported by the system service manager. When adding a new first log offloading interface, interface configuration information can be added to the first configuration file so that the system service manager can create the first log offloading interface based on the interface configuration information in the first configuration file.

[0099] The interface configuration information can include various parameters such as input parameters, output parameters, and data format for the first log offloading interface. Taking rsyslog as the log service process on the DPU side as an example, the interface configuration information can include the data format, such as syslog format, so that the second log data in syslog format can be offloaded to the DPU through the second log offloading interface. In the first configuration file, the log service interface of each system service can be set as the first log offloading interface, and the corresponding service identifier can be set, so as to realize the offloading of log data of multiple system services through a unified first log offloading interface.

[0100] For example, when the second component starts, a fixed-format message string is generated based on the interface configuration information passed in at startup and sent to the system service manager, such as systemd, to initialize the logging service. After parsing the message string, the system service manager initializes the first log offloading interface with the DPU-side logging service process rsyslog. Afterwards, the second log data can be offloaded to the DPU through the first log offloading interface.

[0101] In the embodiments disclosed herein, by utilizing the system service manager of the second component to create the first log offload interface, the complexity and workload of modifying the source code of each system service individually can be reduced. Furthermore, offloading the second log data through the second log offload interface does not require the host machine's log service process, thus achieving the technical effect of reducing host machine resource consumption.

[0102] According to an embodiment of this disclosure, the virtualization component includes: a third component, which supports the kernel services of the host operating system; for the third component, operation S220 includes: offloading second log data to the data processor through a network channel between the third component and the data processor; wherein the network channel is constructed by the third component based on network parameters at the time of operating system startup.

[0103] For example, in the case of the Linux operating system, the third component can be a kernel component that supports Linux kernel services. In this case, the first log data is also called the kernel log.

[0104] The network channel between the third component and the DPU can be created using the netconsole. Network parameters can be added to the boot parameters of the operating system's kernel services, allowing the network channel between the third component and the data processor to be created based on these parameters during operating system startup.

[0105] Network parameters can include the network address and port used to start netconsole. This network address and port can be the network log and port of the DPU's logging service process, invoking netconsole to establish a network channel between the third component and the DPU's logging service process. Afterward, all kernel service log data will be offloaded to the DPU via the network link.

[0106] In the embodiments of this disclosure, the network channel established between the third component and the data processor via netconsole enables the continued unloading of log data to the DPU even when the network protocol stack is unavailable or the interruption mechanism is abnormal, thereby ensuring the stability of the log unloading function.

[0107] Figure 4A The diagram illustrates an application scenario of unloading second log data according to an embodiment of this disclosure. Figure 4A As shown in Embodiment 400A, the host machine M1 includes a first component C1, a second component C2, and a third component C3, wherein the first component C1 and the second component C2 run in the user-mode environment, and the third component C3 runs in the kernel-mode environment.

[0108] The first component C1 can unload the second log data 1 to the log service process P1 of the data processor M2 (also known as the DPU) through the second log unloading interface. The second component C2 can unload the second log data 2 to the log service process P1 of the data processor M2 through the first log unloading interface created by the system service manager P2. The third component C2 unloads the second log data 3 to the log service process P1 of the data processor M2 through the network channel between itself and the log service process P1 in the DPU.

[0109] In one specific embodiment, for a first component capable of simulating a virtual machine, the first log data in the first component may be obtained from the virtual machine.

[0110] According to embodiments of this disclosure, the method further includes: receiving first log data from a virtual machine when the first log data is generated using a virtual machine simulated by the first component, to generate second log data based on the first log data and source data; and unloading the second log data to a data processor via a second log offload interface.

[0111] The first component and the simulated virtual machine transmit the first log data via a serial port. This first log data is also called the serial port log.

[0112] For the first log data generated by the virtual machine, the virtualization component receives the first log data from the virtual machine via a serial port and reuses the log offloading method of the first component to offload the log data. For example, based on the first log data and the source data, second log data is generated; and the second log data is offloaded to the data processor through the second log offloading interface. The log data offloading is as described above and will not be repeated here. For example, after each time the first log data is obtained from the virtual machine, the log offloading method of the first component can be reused to offload the log data.

[0113] It is understood that, in this embodiment, the source data may include third source sub-data and fourth source sub-data; or, it may include third source sub-data, fourth source sub-data and fifth source sub-data.

[0114] For example, each time the first component receives first log data from the virtual machine, it writes the received first log data to a specified character device. In this embodiment, a new character device can be added. The write operation to this character device follows the definition of the log library framework. After processing the first log data in the character device into second log data, the second log data is unloaded through the second log unloading interface.

[0115] In the embodiments of this disclosure, the first log data generated by the virtual machine is unloaded by reusing the second log unloading interface of the first component, which can support the unloading of serial port logs without modification, thus reducing the workload of modification.

[0116] Figure 4B The diagram illustrates an application scenario of unloading second log data according to another embodiment of this disclosure. Figure 4B As shown, in embodiment 400B, the host machine M1 includes a first component 1 C11, a first component 2 C12, and a virtual machine V1 simulated by the first component 2 C12. The first component 1 C11 can be a user-space application, such as libvirt, which can generate second log data 1 based on the first log data it generates and the source data, and unload the second log data 1 to the log service process P1 of the data processor M2 through the second log unloading interface.

[0117] The first component 2, C12, can be QEMU. When virtual machine V1 starts, the first component 2, C12, simulates a serial port device for virtual machine V1. This serial port device can be of the type of socket, etc. Each time virtual machine V1 writes the first log data 4 to the serial port device, the write operation triggers a Port I / O (PIO) event. After judgment by the kernel-built-in virtual machine kvm, a virtual machine exit event occurs, and then the first log data 4 is transmitted to the first component 2, C12, via the simulated serial port. The log data offloading method of the first component 2, C12, is reused to generate second log data 4 based on the transmitted first log data 4 and the source data. This second log data 4 is then offloaded to the log service process P1 of the data processor M2 through the second log offloading interface.

[0118] However, when receiving the first log data via the serial port, the serial port writes one byte at a time, and the first component also processes the serial port data in units of one byte. This single-byte writing method is extremely inefficient. When receiving too much first log data via the serial port, the virtual machine's CPU performance is severely impacted, with significant jitter in the virtual CPU (VCPU) latency, affecting the operation of other tasks on that VCPU.

[0119] Therefore, this disclosure further optimizes the transmission process for the first log data generated by the virtual machine.

[0120] Figure 5 A flowchart illustrating the generation of first log data based on first log sub-data and second log sub-data according to an embodiment of the present disclosure is shown.

[0121] like Figure 5 As shown, embodiment 500 includes operations S510 to S520.

[0122] When operating S510, the first log sub-data of the target data length is obtained from the virtual machine.

[0123] In operation S520, the first log data is determined based on the first log sub-data and at least one second log sub-data, wherein the second log sub-data with a target data length is stored in the cache of the first component, and the second log sub-data is obtained from the virtual machine before the first log sub-data.

[0124] When the first component starts the simulated virtual machine, it pre-allocates a fixed-size cache for each virtual machine to cache the log sub-data acquired from that virtual machine. In this embodiment, log sub-data of the target data length can be acquired via the serial port between the virtual machine and the first component, and the acquired log sub-data is stored in the cache. For distinction, the currently acquired log sub-data is referred to as the first log sub-data; the log sub-data acquired from the same virtual machine and stored in the cache before the first log sub-data is referred to as the second log sub-data.

[0125] In this embodiment, instead of unloading the log every time a first log sub-data is received, the first log data is determined based on the first log sub-data and at least one second log sub-data, and then the first log data is regarded as the first log data to be unloaded.

[0126] The target data length can be a single byte, but the first log data determined based on the first log sub-data and at least one second log sub-data is longer than a single byte.

[0127] In one specific embodiment, the first log data may have a predetermined data length, thereby generating first log data of the predetermined data length based on the first log sub-data and at least one second log sub-data, and performing operations S210 and S220 to complete the unloading of the log data.

[0128] In embodiments of this disclosure, by setting a cache in the first component to store at least one second log sub-data of a target data length, and determining the first log data based on the first log sub-data and at least one second log sub-data, the existing method of receiving log data from the virtual machine and unloading it each time is optimized to merging the log data received from the virtual machine and performing an unloading once, thereby reducing the number of times log data is unloaded under the same data volume, reducing the latency jitter of the virtual machine VCPU, and improving the performance of the virtual machine.

[0129] According to embodiments of this disclosure, determining the first log data to be unloaded based on the first log sub-data and at least one second log sub-data includes: if the data volume of the first log sub-data and at least one second log sub-data meets a predetermined condition, determining the first log sub-data and at least one second log sub-data as the first log data; if the data volume of the first log sub-data and at least one second log sub-data does not meet the predetermined condition, storing the first log sub-data as newly added second log sub-data in the cache, and obtaining new first log sub-data from the virtual machine until the predetermined condition is met.

[0130] The predetermined condition may be one of the following: the sum of the data volume of the first log sub-data and at least one second log sub-data reaches the data volume threshold, reaches the upper limit of the cache allocated to the virtual machine in the first component, is within the predetermined data volume range, or reaches the log line break condition.

[0131] If the total amount of the first log sub-data and at least one second log sub-data meets a predetermined condition, the first log sub-data and at least one second log sub-data can be combined into a first log data according to the reception time. This first log data is the first log data to be unloaded. After determining the first log sub-data and at least one second log sub-data as the first log data, the cache can be cleared. If the total amount of the first log sub-data and at least one second log sub-data does not meet the predetermined condition, the first log sub-data is stored in the cache as a newly added second log sub-data, and new first log sub-data is obtained from the virtual machine until the predetermined condition is met before the first log data is determined. After determining the first log data, operations S210 and S220 can be executed. The specific operations are as described above and will not be repeated here.

[0132] In the embodiments of this disclosure, by judging the amount of data of the first log sub-data and at least one second log sub-data, and determining the first log sub-data and at least one second log sub-data as the first log data when the amount of data is reached, the subsequent log data unloading is performed, thereby optimizing the unloading operation of the first log data generated by the virtual machine and improving the performance of the virtual machine.

[0133] According to embodiments of this disclosure, the method further includes: determining at least one second log data as the first log data when it is determined that the first log sub-data represents a log line break.

[0134] Log data generated by virtualization components and virtual machines is in line format. Therefore, the first byte of log data can be a carriage return and / or a newline character. Different systems can use carriage returns and / or newlines to indicate line breaks in the log. For example, some systems use a carriage return plus a newline character to indicate a line break, while others only use a newline character to indicate a line break.

[0135] By examining the content of the first log sub-data, it can be determined whether the first log sub-data represents a newline character. For example, if the first log data is determined to be a newline character, then the first log sub-data represents a newline character; otherwise, it represents a newline character.

[0136] If the first log sub-data indicates a newline character, at least one second log data can be directly combined into the first log data according to the reception time to execute the subsequent log data unloading function. Furthermore, after combining at least one second log data into the first log data according to the reception time, the buffer can be cleared and new first log sub-data can continue to be received, thus skipping the newline character for log data unloading.

[0137] In embodiments of this disclosure, by determining whether the first log sub-data indicates a log line break, and if so, identifying at least one second log data as the first log data, the stored first log data conforms to the habit of reading line by line while reducing the number of log data unloading operations, facilitating subsequent log data queries. Furthermore, the line break determination in the first component can filter out special characters such as newline characters and carriage returns from the first log data, shielding different systems from different handling methods of log line breaks and preventing garbled characters in the log data stored by the DPU due to incompatible system line break characters.

[0138] Figure 6 The illustration shows a schematic diagram of the process for storing and unloading log data for a virtual machine according to an embodiment of the present disclosure.

[0139] like Figure 6 As shown, embodiment 600 includes operations S601 to S606.

[0140] When operating S601, the first component starts.

[0141] When operating the S602, the first log data is received via the serial port.

[0142] In operation S603, determine whether the first log sub-data is a newline character. If yes, execute operation S605; otherwise, execute operation S604.

[0143] In operation S604, determine if the buffer is full. If yes, execute operation S605; otherwise, return to execute operation S602, store the first log sub-data into the buffer, and continue receiving the first log sub-data via the serial port.

[0144] In operation S605, first log data is generated. If operation S603 determines that it is true, operation S605 is executed, and at least one second log sub-data in the cache is used as the first log data; if operation S604 determines that it is true, the first log data is generated based on the first log sub-data and the second log sub-data in the cache.

[0145] In operation S606, the first log data is unloaded and the cache is cleared. For example, second log data is generated based on the first log data and the source data. The second log data is unloaded to the DPU through the second log unloading interface, and the cache is requested. After executing operation S606, operation S602 can be returned to continue receiving new first log sub-data.

[0146] In addition, for the first component that does not have the log offloading function enabled, at least one second log sub-data can still be stored in the cache to reduce the number of times the log sub-data generated by the virtual machine is written to the host machine, thereby reducing the VCPU latency jitter of the virtual machine.

[0147] In summary, the embodiments disclosed herein achieve log data offloading of the host-side virtualization component under the DPU architecture without requiring code modifications, avoiding environmental risks and performance fluctuations to the virtual machine caused by logging into the host machine to view log data in maintenance scenarios. Furthermore, by removing the log service process from the host machine and utilizing a log offloading channel matching the virtualization component for log data offloading, the host machine's dependence on the log service process is reduced, making the host machine more lightweight. Additionally, for the first component, optimizing serial port log processing through caching reduces the performance impact of writing serial port logs on the virtual machine, improving the user experience of cloud service functions.

[0148] Figure 7 A flowchart illustrating a log data processing method according to another embodiment of this disclosure is shown schematically. Figure 7 As shown, another embodiment 700 of the log data processing method includes operation S710.

[0149] In operation S710, in response to receiving second log data obtained from the host machine, the first log data is stored in the storage path corresponding to the source data of the first log data according to the second log data.

[0150] The second log data is generated by the host machine in response to the detection of a log offload command from a virtualization component, based on the first log data to be offloaded and the source data. The virtualization component represents the component in the host machine used to provide virtualization services. The second log data is transmitted through a log offload channel that matches the host machine and the virtualization component. For details on this operation, please refer to the description above; it will not be repeated here.

[0151] For example, the host machine can parse the received second log data to obtain the first log data and the source data corresponding to the first log data. Then, it determines the storage path corresponding to the source data and stores the first log data in the storage path corresponding to the source data.

[0152] The storage path can be a storage path within the DPU's internal storage space or an external storage path, such as a storage path on other hardware. For external storage paths, the DPU is responsible for determining the storage path and forwarding or storing the first log data to the external storage path.

[0153] In the embodiments of this disclosure, since the second log data received from the host machine includes the first log data and the source data of the first log data, the first log data can be stored in a targeted manner according to the source data, which facilitates the subsequent log data management and query of virtualization components in the DPU and simplifies the log data viewing operation on the DPU side.

[0154] According to embodiments of this disclosure, storing first log data in a storage path corresponding to the source data of the first log data, based on second log data, includes: invoking a log service process to parse the second log data into first log data and a storage path identifier; determining the source data and the storage path corresponding to the source data based on a second configuration file and the storage path identifier; and storing the first log data in the storage path corresponding to the source data.

[0155] A log service process can be a process capable of performing complex log processing, routing, formatting, and storage functions; for example, it could be rsyslog.

[0156] The second log data has a predefined format. For example, for the log service process rsyslog, the second log data can be in syslog data format. The log service process can parse the predefined formatted second log data to obtain the first log data and a storage path identifier as the data header. The parsing process can be the reverse process of generating the second log data based on the storage path identifier and the first log data; the specific operations correspond to the generation process and will not be elaborated here.

[0157] The second configuration file may include a unique mapping between storage path identifiers and the source data of the first log data, and / or the storage path corresponding to the source data. Therefore, the DPU's log service process can filter and select storage path identifiers based on the second configuration file to determine the storage path corresponding to the source data. Subsequently, the first log data is stored in that storage path.

[0158] In the embodiments of this disclosure, the second log data is parsed into the first log data and the storage path identifier. Based on the second configuration file and the storage path identifier, the source data and the storage path corresponding to the source data are determined. The first log data is stored in the storage path corresponding to the source data. While realizing the log data unloading function, the DPU can support storing the first log data according to the source data, which facilitates log data management.

[0159] Figure 8 A data flow diagram illustrating the storage of first log data according to another embodiment of this disclosure is shown schematically. Figure 8 As shown, in embodiment 800, the second log data 801 can be parsed to obtain the storage path identifier 802 and the first log data 803. The DPU can call the log service process to determine the source data 805 and the storage path 806 corresponding to the source data 805 based on the second configuration file 804 and the storage path identifier 802. Therefore, the first log data 803 can be stored in the storage path 806.

[0160] According to embodiments of this disclosure, the virtualization component includes a first component; determining source data and a storage path corresponding to the source data based on a second configuration file and a storage path identifier includes: determining at least one source sub-data based on the storage path identifier, wherein the source sub-data includes at least one of the following: a first source sub-data, a second source sub-data, a third source sub-data, a fourth source sub-data, and a fifth source sub-data; and determining the storage path based on the second configuration file and at least one source sub-data.

[0161] The first component is used to simulate or manage virtual machines; for example, it could be qemu or libvirt.

[0162] Similar to the process of generating storage path identifiers, when the first component on the host side generates the first log data itself, the first source sub-data can be determined based on the storage path identifier; or, the first source sub-data and the second source sub-data can be determined based on the storage path identifier. When the first component on the host side is generated by a simulated virtual machine, the third source sub-data and the fourth source sub-data can be determined based on the storage path identifier; or, the third source sub-data, the fourth source sub-data, and the fifth source sub-data can be determined based on the storage path identifier.

[0163] In other embodiments, the storage path identifier may also be generated based on at least one of the following: a first source sub-data, a second source sub-data, a third source sub-data, a fourth source sub-data, and a fifth source sub-data; thus, the source sub-data determined by the DPU may include at least one of the above.

[0164] The second configuration file may also include a unique mapping between each source sub-data and the storage path, so that after determining at least one source sub-data, the storage path can be determined based on the second configuration file and at least one source sub-data.

[0165] In the embodiments of this disclosure, by parsing at least one source sub-data and a second configuration file to determine the storage path, it is possible to realize log data storage at multiple granularities such as virtualization components, processes within virtualization components, virtual machines, and processes within virtual machines, so that specific log data at multiple granularities can be directly obtained on the DPU side according to the storage path.

[0166] Figure 9 A block diagram of a log data processing apparatus according to an embodiment of the present disclosure is shown schematically. Figure 9 As shown, the log data processing device 900 includes a generation module 910 and an unloading module 920.

[0167] The generation module 910 is used to generate second log data in response to the detection of a log unloading instruction of a virtualization component, based on the first log data to be unloaded in the virtualization component and the source data of the first log data, wherein the virtualization component represents a component in the host machine used to provide virtualization services.

[0168] The unloading module 920 is used to unload the second log data to the data processor through a log unloading channel that matches the virtualization component, so that the data processor stores the first log data to the storage path corresponding to the source data based on the second log data.

[0169] According to embodiments of this disclosure, the generation module 910 includes:

[0170] The first generation submodule is used to generate storage path identifiers based on the source data of the first log data.

[0171] The second generation submodule is used to generate second log data in a predetermined format based on the storage path identifier and the first log data; wherein the predetermined format includes data formats supported by the log service process of the data processor.

[0172] According to embodiments of this disclosure, the first generation submodule includes either a first generation unit or a second generation unit.

[0173] The first generation unit is used to generate a storage path identifier based on the first source sub-data, wherein the first source sub-data is used to characterize the virtualization component that generates the first log data.

[0174] The second generation unit is used to generate a storage path identifier based on the first source sub-data and the second source sub-data, wherein the second source sub-data is used to characterize the process in the virtualization component used to generate the first log data.

[0175] According to embodiments of this disclosure, the virtualization component includes: a first component for simulating or managing virtual machines; the first generation submodule further includes: a third generation unit or a fourth generation unit. When the first log data is generated by a virtual machine simulated by the first component,

[0176] The third generation unit is used to generate a storage path identifier based on the third source sub-data and the fourth source sub-data. The third source sub-data is used to characterize the first component, and the fourth source sub-data is used to characterize the virtual machine.

[0177] The fourth generation unit is used to generate a storage path identifier based on the third source sub-data, the fourth source sub-data, and the fifth source sub-data, wherein the fifth source sub-data represents the process in the virtual machine used to generate the first log data.

[0178] According to embodiments of this disclosure, the virtualization component includes a second component for supporting system services of the host machine; the offloading module 920 includes:

[0179] The first unloading submodule is used to call the system service manager of the second component. Through the first log unloading interface created by the system service manager, the second log data is unloaded to the data processor. The first log unloading interface is created by the system service manager according to the interface configuration information in the first configuration file.

[0180] According to embodiments of this disclosure, the virtualization component includes: a third component, the third component being used to support the kernel services of the host operating system; the offloading module 920 includes:

[0181] The second unloading submodule is used to unload the second log data to the data processor through the network channel between the third component and the data processor; wherein the network channel is constructed by the third component based on the network parameters when the operating system starts.

[0182] According to embodiments of this disclosure, the unloading module 920 includes:

[0183] The third unloading submodule is used to unload the second log data to the data processor through the second log unloading interface between the first component and the data processor. The second log unloading interface is created using the log library framework embedded in the first component.

[0184] According to embodiments of this disclosure, the log data offloading device 900 further includes: for the first component,

[0185] The first acquisition submodule is used to receive first log data from the virtual machine when the first log data is generated by the virtual machine simulated by the first component, so as to generate second log data based on the first log data and the source data.

[0186] The fourth unloading submodule is used to unload the second log data to the data processor through the second log unloading interface.

[0187] According to embodiments of this disclosure, the log data offloading device 900 further includes: for the first component, when the first log data is generated using a virtual machine simulated by the first component,

[0188] The second acquisition submodule is used to obtain the first log subdata of the target data length from the virtual machine.

[0189] A determination submodule is configured to determine the first log data based on the first log sub data and at least one second log sub data, wherein the second log sub data with a target data length is stored in the cache of the first component, and the second log sub data is obtained from the virtual machine before the first log sub data.

[0190] According to embodiments of this disclosure, the determining submodule includes: a first determining unit and a second determining unit.

[0191] The first determining unit is configured to determine the first log data and at least one second log data as first log data when the data volume of the first log sub-data and at least one second log sub-data meets a predetermined condition;

[0192] The second determining unit is configured to, when the amount of data of the first log sub-data and at least one second log sub-data does not meet a predetermined condition, store the first log sub-data as a newly added second log sub-data in the cache, and obtain new first log sub-data from the virtual machine until the predetermined condition is met.

[0193] According to an embodiment of this disclosure, the determining submodule further includes: a third determining unit, configured to determine at least one second log data as the first log data when it is determined that the first log sub data represents a log line break.

[0194] Figure 10 A block diagram of a log data processing apparatus according to another embodiment of the present disclosure is shown schematically. Figure 10 As shown, the log data processing device 1000 includes a storage module 1010, used to, in response to receiving second log data obtained from the host machine, store the first log data to a storage path corresponding to the source data of the first log data; wherein, the second log data is generated by the host machine in response to detecting a log unloading instruction from a virtualization component, based on the first log data to be unloaded and the source data, and the virtualization component represents a component in the host machine used to provide virtualization services; the second log data is transmitted through a log unloading channel that matches the host machine and the virtualization component.

[0195] According to embodiments of this disclosure, the storage module 1010 includes a parsing submodule, a path determination submodule, and a storage submodule. The log service process is invoked to parse the parsing submodule into first log data and a storage path identifier.

[0196] The path determination submodule is used to determine the source data and the storage path corresponding to the source data based on the second configuration file and the storage path identifier.

[0197] The storage submodule is used to store the first log data to the storage path corresponding to the source data.

[0198] According to embodiments of this disclosure, the virtualization component includes a first component; the path determination submodule includes a source determination unit and a path determination unit.

[0199] The source determination unit is configured to determine at least one source sub-data based on the storage path identifier, wherein the source sub-data includes at least one of the following: first source sub-data, second source sub-data, third source sub-data, fourth source sub-data, and fifth source sub-data.

[0200] The path determination unit is used to determine the storage path based on the second configuration file and at least one source sub-data.

[0201] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0202] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0203] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0204] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0205] Figure 11A block diagram of an electronic device suitable for implementing a log data processing method according to embodiments of the present disclosure is illustrated schematically. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0206] like Figure 11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded into a random access memory (RAM) 1103 from a storage unit 1108. The RAM 1103 may also store various programs and data required for the operation of the electronic device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0207] Multiple components in electronic device 1100 are connected to input / output (I / O) interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0208] The computing unit 1101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as log data processing methods. For example, in some embodiments, the log data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the log data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform a log data processing method by any other suitable means (e.g., by means of firmware).

[0209] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0210] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0211] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0212] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0213] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0214] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0215] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0216] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A log data processing method, comprising: In response to the detection of a log unloading instruction for a virtualization component, second log data is generated based on the first log data to be unloaded in the virtualization component and the source data of the first log data. The virtualization component represents a component in the host machine used to provide virtualization services. The first log data includes at least one of data for recording the virtualization component's own operating status and data to facilitate the virtualization component's management of virtual machines. The virtualization component includes at least one of the following: a first component for simulating or managing virtual machines, a second component for supporting system services of the host machine, and a third component for supporting kernel services of the host machine's operating system. The second log data is offloaded to the data processor through a log offload channel that matches the virtualization component, so that the data processor stores the first log data to a storage path corresponding to the source data based on the second log data, so as to query and manage the first log data in the data processor; The process of offloading the second log data to the data processor via a log offloading channel that matches the virtualization component includes at least one of the following methods: The second log data is unloaded to the data processor through the second log unloading interface between the first component and the data processor, wherein the second log unloading interface is created using a log library framework embedded in the first component; The system service manager of the second component is invoked, and the second log data is unloaded to the data processor through the first log unloading interface created by the system service manager. The first log unloading interface is created by the system service manager according to the interface configuration information in the first configuration file. The second log data is offloaded to the data processor via a network channel between the third component and the data processor; wherein the network channel is constructed by the third component based on network parameters at the time of operating system startup.

2. The method according to claim 1, wherein, The step of generating second log data based on the first log data to be unloaded in the virtualization component and the source data of the first log data includes: Generate a storage path identifier based on the source data of the first log data; Based on the storage path identifier and the first log data, second log data in a predetermined format is generated; wherein, the predetermined format includes data formats supported by the log service process of the data processor.

3. The method according to claim 2, wherein, The step of generating a storage path identifier based on the source data of the first log data includes: The storage path identifier is generated based on the first source sub-data, wherein the first source sub-data is used to characterize the virtualization component that generated the first log data; or... The storage path identifier is generated based on the first source sub-data and the second source sub-data, wherein the second source sub-data is used to characterize the process in the virtualization component that generates the first log data.

4. The method according to claim 3, wherein, The method further includes: when the first log data is generated by a virtual machine simulated by the first component, The storage path identifier is generated based on the third-source sub-data and the fourth-source sub-data, wherein the third-source sub-data is used to characterize the first component, and the fourth-source sub-data is used to characterize the virtual machine; or... The storage path identifier is generated based on the third source sub-data, the fourth source sub-data, and the fifth source sub-data, wherein the fifth source sub-data represents the process in the virtual machine used to generate the first log data.

5. The method according to claim 1, further comprising: Regarding the first component, If the first log data is generated by a virtual machine simulated using the first component, the first log data is received from the virtual machine to generate the second log data based on the first log data and the source data. as well as The second log data is unloaded to the data processor via the second log unloading interface.

6. The method according to claim 1, further comprising: Regarding the first component, if the first log data is generated using a virtual machine simulated by the first component, Obtain the first log sub-data of the target data length from the virtual machine; The first log data is determined based on the first log sub-data and at least one second log sub-data, wherein the second log sub-data with a target data length is stored in the cache of the first component, and the second log sub-data is obtained from the virtual machine before the first log sub-data.

7. The method according to claim 6, wherein, The step of determining the first log data to be uninstalled based on the first log sub-data and at least one second log sub-data includes: If the data volume of the first log sub-data and at least one second log sub-data meets a predetermined condition, the first log sub-data and at least one second log sub-data are determined as the first log data. If the amount of data of the first log sub-data and at least one second log sub-data does not meet the predetermined condition, the first log sub-data is stored in the cache as a newly added second log sub-data, and new first log sub-data is obtained from the virtual machine until the predetermined condition is met.

8. The method according to claim 6 or 7, further comprising: If it is determined that the first log sub-data represents a log line break, at least one second log data is identified as the first log data.

9. A log data processing method, comprising: In response to receiving second log data obtained from the host machine, the first log data is stored in a storage path corresponding to the source data of the first log data, so as to query and manage the first log data in the data processor. The first log data includes at least one of data for recording the operation status of the virtualization component and data for facilitating the virtualization component to manage virtual machines. Wherein, the second log data is generated by the host machine in response to the detection of a log unloading instruction of a virtualization component, based on the first log data to be unloaded and the source data. The virtualization component represents a component in the host machine used to provide virtualization services. The virtualization component includes at least one of the following: a first component for simulating or managing virtual machines, a second component for supporting system services of the host machine, and a third component for supporting kernel services of the operating system of the host machine. The second log data is transmitted via a log offloading channel that matches the host machine and the virtualization component using at least one of the following methods: The second log data is unloaded to the data processor through the second log unloading interface between the first component and the data processor, wherein the second log unloading interface is created using a log library framework embedded in the first component; The system service manager of the second component is invoked, and the second log data is unloaded to the data processor through the first log unloading interface created by the system service manager. The first log unloading interface is created by the system service manager according to the interface configuration information in the first configuration file. The second log data is offloaded to the data processor via a network channel between the third component and the data processor; wherein the network channel is constructed by the third component based on network parameters at the time of operating system startup.

10. The method according to claim 9, wherein, The step of storing the first log data to the storage path corresponding to the source data of the first log data based on the second log data includes: invoking the log service process. The second log data is parsed into the first log data and the storage path identifier; Based on the second configuration file and the storage path identifier, the source data and the storage path corresponding to the source data are determined; and The first log data is stored in the storage path corresponding to the source data.

11. The method according to claim 10, wherein, The virtualization component includes a first component; determining the source data and the storage path corresponding to the source data based on the second configuration file and the storage path identifier includes: Based on the storage path identifier, at least one source sub-data is determined, wherein the source sub-data includes at least one of the following: a first source sub-data, a second source sub-data, a third source sub-data, a fourth source sub-data, and a fifth source sub-data; and The storage path is determined based on the second configuration file and at least one of the source sub-data.

12. A log data processing apparatus, comprising: A generation module is configured to, in response to the detection of a log unloading instruction from a virtualization component, generate second log data based on first log data to be unloaded from the virtualization component and source data of the first log data. The virtualization component represents a component in the host machine used to provide virtualization services. The first log data includes at least one of data recording the operating status of the virtualization component and data facilitating the management of virtual machines by the virtualization component. The virtualization component includes at least one of the following: a first component for simulating or managing virtual machines, a second component for supporting system services of the host machine, and a third component for supporting kernel services of the host machine's operating system. An unloading module is used to unload the second log data to a data processor through a log unloading channel that matches the virtualization component, so that the data processor stores the first log data to a storage path corresponding to the source data based on the second log data, so as to query and manage the first log data in the data processor; The unloading module also includes at least one of the following: The first unloading submodule is used to call the system service manager of the second component and unload the second log data to the data processor through the first log unloading interface created by the system service manager. The first log unloading interface is created by the system service manager according to the interface configuration information in the first configuration file. The second unloading submodule is used to unload the second log data to the data processor through the network channel between the third component and the data processor; wherein the network channel is constructed by the third component based on the network parameters when the operating system starts. The third unloading submodule is used to unload the second log data to the data processor through the second log unloading interface between the first component and the data processor, wherein the second log unloading interface is created using a log library framework embedded in the first component.

13. The apparatus according to claim 12, wherein, The generation module includes: The first generation submodule is used to generate a storage path identifier based on the source data of the first log data; The second generation submodule is used to generate the second log data in a predetermined format based on the storage path identifier and the first log data; wherein the predetermined format includes data formats supported by the log service process of the data processor.

14. The apparatus according to claim 13, wherein, The first generation submodule includes: A first generation unit is configured to generate the storage path identifier based on first source sub-data, wherein the first source sub-data is used to characterize the virtualization component that generates the first log data; or... The second generation unit is configured to generate the storage path identifier based on the first source sub-data and the second source sub-data, wherein the second source sub-data is used to characterize the process in the virtualization component that generates the first log data.

15. The apparatus according to claim 14, wherein, The first generation submodule further includes: when the first log data is generated by a virtual machine simulated by the first component, The third generation unit is configured to generate the storage path identifier based on third source sub-data and fourth source sub-data, wherein the third source sub-data is used to characterize the first component and the fourth source sub-data is used to characterize the virtual machine; or... The fourth generation unit is used to generate the storage path identifier based on the third source sub-data, the fourth source sub-data, and the fifth source sub-data, wherein the fifth source sub-data represents the process in the virtual machine used to generate the first log data.

16. The apparatus of claim 12, further comprising: Regarding the first component, The first acquisition submodule is configured to receive the first log data from the virtual machine when the first log data is generated by a virtual machine simulated by the first component, so as to generate the second log data based on the first log data and the source data. as well as The fourth unloading submodule is used to unload the second log data to the data processor through the second log unloading interface.

17. The apparatus of claim 12, further comprising: Regarding the first component, if the first log data is generated using a virtual machine simulated by the first component, The second acquisition submodule is used to acquire first log sub-data of a target data length from the virtual machine; A determination submodule is configured to determine the first log data based on the first log subdata and at least one second log subdata, wherein the second log subdata with a target data length is stored in the cache of the first component, and the second log subdata is obtained from the virtual machine before the first log subdata.

18. The apparatus according to claim 17, wherein, The determining submodule includes: The first determining unit is configured to determine the first log data and at least one second log data as the first log data when the data volume of the first log sub-data and at least one second log sub-data meets a predetermined condition. The second determining unit is configured to, when the amount of the first log sub-data and at least one second log sub-data does not meet the predetermined condition, store the first log sub-data as a newly added second log sub-data in the cache, and obtain new first log sub-data from the virtual machine until the predetermined condition is met.

19. The apparatus according to claim 17 or 18, wherein the determining submodule further comprises: The third determining unit is used to determine at least one of the second log data as the first log data when it is determined that the first log sub-data represents a log line break.

20. A log data processing apparatus, comprising: A storage module is configured to, in response to receiving second log data obtained from a host machine, store first log data to a storage path corresponding to the source data of the first log data, so as to query and manage the first log data in a data processor. The first log data includes at least one of data for recording the operating status of the virtualization component and data for facilitating the virtualization component to manage virtual machines. Wherein, the second log data is generated by the host machine in response to the detection of a log unloading instruction of a virtualization component, based on the first log data to be unloaded and the source data, and the virtualization component represents a component in the host machine used to provide virtualization services; the virtualization component includes at least one of the following: a first component for simulating or managing virtual machines, a second component for supporting system services of the host machine, and a third component for supporting kernel services of the operating system of the host machine; The second log data is transmitted via a log offloading channel that matches the host machine and the virtualization component using at least one of the following methods: The second log data is unloaded to the data processor through the second log unloading interface between the first component and the data processor, wherein the second log unloading interface is created using a log library framework embedded in the first component; The system service manager of the second component is invoked, and the second log data is unloaded to the data processor through the first log unloading interface created by the system service manager. The first log unloading interface is created by the system service manager according to the interface configuration information in the first configuration file. The second log data is offloaded to the data processor via a network channel between the third component and the data processor; wherein the network channel is constructed by the third component based on network parameters at the time of operating system startup.

21. The apparatus according to claim 20, wherein, The storage module includes: a log service call process. The parsing submodule is used to parse the second log data into the first log data and the storage path identifier; The path determination submodule is used to determine the source data and the storage path corresponding to the source data based on the second configuration file and the storage path identifier; and The storage submodule is used to store the first log data to the storage path corresponding to the source data.

22. The apparatus according to claim 21, wherein, The virtualization component includes a first component; the path determination submodule includes: A source determination unit is configured to determine at least one source sub-data based on the storage path identifier, wherein the source sub-data includes at least one of the following: a first source sub-data, a second source sub-data, a third source sub-data, a fourth source sub-data, and a fifth source sub-data; and A path determination unit is configured to determine the storage path based on the second configuration file and at least one of the source sub-data.

23. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 11.

25. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images