Log data processing method and device, electronic equipment, medium and program product

Abstract

The invention provides a log data processing method and device, electronic equipment, a medium and a program product, and relates to the technical field of computers, in particular to the technical fields of cloud computing, intelligent chips and the like. The log data processing method comprises the steps that in response to a detected log unloading instruction of a virtualization component, second log data is generated according to first log data to be unloaded in the virtualization component and source data of the first log data, and the virtualization component represents a component used for providing virtualization service in a host machine; and unloading the second log data to the data processor through a log unloading channel matched with the virtualization component, so that the data processor stores the first log data to a storage path corresponding to the source data according to the second log data.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to cloud computing, smart chips and other technical fields, and more particularly to a log data processing method, device, electronic device, medium and program product. Background Art

[0002] With the development of technology, data processing unit (DPU) is gradually applied in cloud computing scenarios. DPU runs an independent operating system and has multiple working modes. It can be used as a common network card or work with the host operating system, for example, to offload multiple functions of the host to DPU.

[0003] However, in a virtualization framework where the DPU and the host collaborate, the processing of log data will occupy too many host resources, especially the log data of the virtualization components in the host. Summary of the invention

[0004] The present disclosure provides a log data processing method, device, electronic device, medium and program product.

[0005] According to one aspect of the present disclosure, a log data processing method is provided, comprising: in response to detecting a log unloading instruction of a virtualization component, generating second log data according to first log data to be unloaded in the virtualization component and source data of the first log data, wherein the virtualization component represents a component in a host machine for providing virtualization services; unloading the second log data to a data processor through a log unloading channel matching the virtualization component, so that the data processor stores the first log data to a storage path corresponding to the source data according to the second log data.

[0006] According to another aspect of the present disclosure, a log data processing method is provided, including: in response to receiving second log data obtained from a host machine, storing first log data in a storage path corresponding to source data of the first log data according to the second log data; wherein the second log data is generated by the host machine in response to detecting a log uninstallation instruction of a virtualization component, based on the first log data to be uninstalled and the source data, and the virtualization component represents a component in the host machine for providing virtualization services; the second log data is transmitted through a log uninstallation channel that matches the host machine and the virtualization component.

[0007] According to another aspect of the present disclosure, a log data processing device is provided, including: a generation module for generating second log data in response to detecting a log unloading instruction of a virtualization component according to first log data to be unloaded in the virtualization component and source data of the first log data, wherein the virtualization component represents a component in a host machine for providing virtualization services; an unloading module for unloading the second log data to a data processor through a log unloading channel matching the virtualization component, so that the data processor stores the first log data to a storage path corresponding to the source data according to the second log data.

[0008] According to another aspect of the present disclosure, a log data processing device is provided, including: a storage module, for storing first log data in a storage path corresponding to source data of the first log data in response to receiving second log data obtained from a host machine, according to the second log data; wherein the second log data is generated by the host machine in response to detecting a log uninstall instruction of a virtualization component, according to the first log data to be uninstalled and the source data, and the virtualization component represents a component in the host machine for providing virtualization services; the second log data is transmitted through a log uninstall channel that matches the host machine and the component.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0011] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described above when executed by a processor.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 An exemplary system architecture that can be applied to the log data processing method and device according to an embodiment of the present disclosure is schematically shown;

[0015] Figure 2 A flow chart schematically shows a log data processing method according to an embodiment of the present disclosure;

[0016] Figure 3A Schematically shows a data flow diagram for generating second log data according to the first embodiment of the present disclosure;

[0017] Figure 3B Schematically shows a data flow diagram for generating second log data according to the second embodiment of the present disclosure;

[0018] Figure 3C Schematically shows a data flow diagram for generating second log data according to the third embodiment of the present disclosure;

[0019] Figure 4A A schematic diagram of an application scenario of uninstalling second log data according to an embodiment of the present disclosure is shown;

[0020] Figure 4B A schematic diagram of an application scenario of uninstalling second log data according to another embodiment of the present disclosure is shown;

[0021] Figure 5 Schematically shows a flow chart of generating first log data according to first log sub-data and second log sub-data according to an embodiment of the present disclosure;

[0022] Figure 6 A schematic diagram of a process for storing and unloading log data for a virtual machine according to an embodiment of the present disclosure is shown;

[0023] Figure 7 A flow chart schematically shows a log data processing method according to another embodiment of the present disclosure;

[0024] Figure 8 Schematically shows a data flow diagram for storing first log data according to another embodiment of the present disclosure;

[0025] Fig. 9 A block diagram of a log data processing device according to an embodiment of the present disclosure is schematically shown;

[0026] Fig.10 A block diagram schematically shows a log data processing device according to another embodiment of the present disclosure;

[0027] Fig.11 A block diagram of an electronic device suitable for implementing a log data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] In the embodiments of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved all comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0030] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0031] The terms used in the present disclosure will be explained below to facilitate understanding of the present disclosure.

[0032] Virtualization (Computer virtualization technology): is a technical term in the field of cloud computing technology, usually referring to a technology that runs computing components on a virtual platform rather than a real hardware platform.

[0033] Virtualization includes two virtualization concepts: host and guest. On top of the host, at least one virtual machine can be simulated through hardware virtualization and other technologies. Applications can run on both the host and the virtual machine, and the application can hardly feel the difference in the underlying operating system.

[0034] The virtualization simulation base is a virtual machine monitor (VMM) on the host machine, which is used to simulate virtual machines. The virtualization simulation base can simulate the central processing unit (CPU) through virtualization technology and provide a series of hardware models so that the operating system of the virtual machine thinks that it is dealing directly with the hardware, thereby realizing the normal operation of the virtual machine. For example, the virtualization simulation base can be qemu (QuickEmulator).

[0035] The kernel-based virtual machine (KVM) is an open source system virtualization module. KVM can achieve virtualization based on the host's Linux operating system and collaborate with qemu.

[0036] Virtualization management tools are used to manage virtual machines. For example, the common virtualization management tool libvirt can manage virtual machines based on KVM.

[0037] The system daemon (systemd) is used to manage the system services of the Linux operating system, such as processing the log data generated by the system services of the Linux operating system.

[0038] The network console (netconsole) is a feature implemented in the kernel of the Linux operating system that allows kernel debugging information, such as system service information, to be transmitted over the network.

[0039] The logging library framework (Liblogging) is an open source logging library framework with independent logging functions.

[0040] The log service process is used to manage log data. For the Linux operating system, the data of each application on the virtual machine and the host is generally received and filtered through a local log service process (such as syslog or rsyslog), and the data is saved in a local file after format processing or forwarded through certain communication means. In the existing virtualization architecture where the DPU and the host coexist, the log service process on the host side can process the log data and save it on the host side; or, multiple components on the host side can communicate with the log service process on the DPU side with the help of the local log service process and forward the log data to the DPU.

[0041] The data processing unit (DPU) is a multi-core processor with high-performance computing capabilities. The DPU can offload AI, storage and other related applications running on other chips in the host (such as the CPU and graphics processing unit (GPU)), thereby improving the data processing performance of the host and reducing the host load.

[0042] Serial Port (SP), also known as serial port, is an expansion interface that uses serial communication.

[0043] At present, various cloud computing vendors have offloaded some computing tasks, network virtualization, and storage virtualization to the DPU to reduce the interference and resource occupation of the host in order to reduce the virtualization loss on the host. However, for most architectures, there are still some tasks on the host to ensure the operation of virtualization functions, such as the virtualization management tool libvirt and the virtualization simulation base qemu. The log data generated by these components can assist in problem location.

[0044] However, the log data generated by the above virtualization components is controlled by the log service process of the host machine, and is saved locally on the host machine after being processed by the log service process. In order to view the above log data, it is necessary to enter the host machine and view it through the log service process, which will cause instability in the virtual machine operating environment. Especially when the log data of the virtualization component is very large, the host machine may be rammed due to some erroneous operations. In addition, directly viewing and analyzing logs on the host machine will also lead to excessive occupation of host machine resources, causing unpredictable effects on the host machine.

[0045] In some log data unloading schemes, the log data generated by the virtualization component can be received and processed by the log service process on the host side, and then the log service process on the host side unloads the log data to the DPU through communication with the log service process on the DPU side. However, in this scheme, if there are multiple virtual machines on the host, the DPU side cannot distinguish each virtual machine and save it in a separate log file, and it is still inconvenient to view the log data. In addition, in this scheme, the log service process on the host side has complex log processing, routing, formatting and storage functions, which require additional resources to support, resulting in excessive occupation of host resources. As an independent daemon process, the log service process also consumes additional system resources to maintain operation, especially when processing a large amount of log input / output (I / O) or complex filtering rules, the log service process on the host side will occupy a large amount of host resources and affect system stability.

[0046] Therefore, the present disclosure provides a log data processing method, including: in response to detecting a log unloading instruction of a virtualization component, generating second log data according to the first log data to be unloaded in the virtualization component and the source data of the first log data, wherein the virtualization component represents a component in the host machine for providing virtualization services; unloading the second log data to a data processor through a log unloading channel matching the virtualization component, so that the data processor stores the first log data to a storage path corresponding to the source data according to the second log data. Through the above embodiment, through the log unloading channel matching the virtualization component, the log data generated by the virtualization component can be directly unloaded to the data processor without the help of the log process service of the host machine, thereby reducing the resource usage on the host machine side and facilitating viewing of the log data of the corresponding virtualization component on the DPU side.

[0047] Figure 1 An exemplary system architecture that can be applied to the log data processing method and device according to an embodiment of the present disclosure is schematically shown.

[0048] like Figure 1As shown, the system architecture 100 of the embodiment of the present disclosure includes a host machine M1 and a data processor M2, wherein the host machine M1 includes a virtualization component M110, wherein the data processor M2 is also a DPU with network, storage, and function offloading functions. It should be noted that Figure 1 The number of virtualization components is for illustrative purposes only. The host machine M1 may have one or more virtualization components at the same time, which are used to provide one or more virtualization services.

[0049] The virtualization component M110 of the host machine M1 can generate second log data according to the first log data to be uninstalled in the virtualization component M110 and the source data of the first log data, and uninstall the second log data to the data processor M2 through the log uninstallation channel matching the virtualization component M110. If there are one or more virtualization components in the host machine M1, each virtualization component can process its own first log data into second log data through the above process and uninstall it to the data processor M2, which will not be repeated here.

[0050] Figure 2 The flowchart of the log data processing method according to the embodiment of the present disclosure is schematically shown. Figure 2 As shown, embodiment 200 includes operations S210 to S220.

[0051] In operation S210 , in response to detecting a log uninstallation instruction of a virtualization component, second log data is generated according to first log data to be uninstalled in the virtualization component and source data of the first log data.

[0052] Virtualization components represent components in the host machine that are used to provide virtualization services. For example, virtualization components can be components such as qemu and libvirt that are used to simulate or manage virtual machines; or virtualization components can also be basic components that ensure that the host machine can provide virtualization services, such as components that manage system services and components that manage the operating system kernel.

[0053] The first log data to be uninstalled may be generated by the virtualization component or received from other components. The first log data generated by the virtualization component is usually used to record the operation of the virtualization component itself; receiving the first log data from other components facilitates the virtualization component to manage the log data of the virtual machine. For example, the first log data to be uninstalled may be the log data generated by qemu itself, or the log data received by qemu from the virtual machine; or the log data generated by libvirt itself.

[0054] The source data is used to characterize the source of the first log data, and the source data may include at least one level of source, such as component source, process source, virtual machine source, etc.

[0055] In one embodiment, the second log data may be generated according to the first log data and the source data according to a predetermined rule. For example, the first log data and the source data may be combined into the second log data in a predetermined combination manner; or, the source data may be integrated into the data header according to a predetermined data transmission format, and the second log data may be generated according to the data header and the first log data.

[0056] The log unloading instruction may be triggered when the virtualization component generates the first log data or receives the first log data. The log unloading instruction may be an instruction form supported by the virtualization component, such as an instruction form adapted to the host machine operating environment. In response to detecting the log unloading instruction of the virtualization component, the log data unloading function may be enabled so as to process the first log data into the second log data and unload it.

[0057] In operation S220, the second log data is unloaded to the data processor through a log unloading channel matching the virtualization component, so that the data processor stores the first log data in a storage path corresponding to the source data according to the second log data.

[0058] The log offloading channel is used for log data transmission, and may be an interface, a network channel, etc. The log offloading channel matching the virtualization component may be a channel between the virtualization component and the DPU, which is dedicated to transmitting the second log data of the virtualization component.

[0059] For example, the log offloading channel may be a newly created interface based on a predetermined transmission protocol, or may reuse an existing interface between the host machine and the data processor DPU.

[0060] In the virtualization structure where the DPU and the host coexist, "offloading" can be understood as transferring the functions on the host to the DPU, which is implemented by the DPU; and / or, transferring the data on the host and storing it in the DPU, which stores and manages the data through virtualization storage technology. In this embodiment, unloading the second log data to the data processor can be understood as storing the second log data in the DPU, and the DPU implements complex log processing, routing, formatting and storage functions. For example, the DPU receives the second log data, parses the second log data into the first log data and the source data, and stores the first log data in the storage path corresponding to the source data through screening and filtering operations. The storage path can be a storage path within the DPU, or a storage path outside the DPU, such as a storage path under other hardware. The DPU can determine the external storage path and store the first log data in the external storage path.

[0061] In one embodiment, after determining the second log data, a log unloading channel can be initialized between the virtualization component and the data processor, and after the initialization operation is completed, the second log data is unloaded through the log unloading channel that matches the virtualization component. In another embodiment, when the virtualization component is started, the log unloading channel can be initialized between the virtualization component and the data processor. When unloading is required, the second log data is unloaded directly through the log unloading channel that matches the virtualization component.

[0062] In the embodiments of the present disclosure, through a log offloading channel that matches the virtualization component, the first log data generated by the virtualization component is directly processed into second log data and unloaded to the data processor without the need to rely on the log process service of the host machine, thereby achieving the technical effect of reducing the resource usage on the host machine side and simplifying the host machine system architecture, so that the processing of log data is uniformly implemented by the DPU, and the system on the host machine is more lightweight.

[0063] In addition, the second log data is generated based on the first log data and the source data, so that the DPU can store the first log data to the storage path corresponding to the source data based on the second log data, which facilitates the log data management and query of the virtualization component and simplifies the log data viewing operation on the DPU side.

[0064] According to an embodiment of the present disclosure, generating second log data based on first log data to be uninstalled in a virtualization component and source data of the first log data includes: generating a storage path identifier based on the source data of the first log data; generating second log data in a predetermined format based on the storage path identifier and the first log data; wherein the predetermined format includes a data format supported by a log service process of a data processor.

[0065] The storage path identifier is used to characterize the storage path where the DPU stores the first log data. For example, the storage path identifier may be a portion of characters intercepted from the source data; or, the storage path identifier may also be an IP address determined from the source data.

[0066] The source data may include information such as the IP (Internet Protocol) address and identifier of the source of the first log data. In addition, the source data may also include information such as the importance, process name, and process number of the first log data; in this case, a storage path identifier may be generated based on at least one information in the source data.

[0067] There is a unique mapping relationship between the source data and the storage path, and thus, a storage path identifier can be generated according to the source data of the first log data.

[0068] Generating second log data of a predetermined format based on a storage path identifier and first log data can be: using the storage path identifier as a data header to constitute the second log data together with the first log data, wherein the data header of a preset length can be limited according to actual needs, so that the data header of the preset length in front of the constituted second log data can reflect the source data.

[0069] For example, the storage path identifier may be the process name or custom name “pr1” of the current virtualization component, and the second log data generated according to the storage path identifier and the first log data “XXX-time-log” may be “pr1-XXX-time-log”.

[0070] It should be noted that the embodiment of the present disclosure removes the log service process on the host side to reduce the load pressure of the host, but the DPU side can receive the second log data uninstalled from the virtualization component of the host through the log service process. Therefore, when generating the second log data, a data format supported by the log service process is generated to ensure that the DPU side can receive and process the second log data.

[0071] For example, the log service process on the DPU side may be rsyslog. The predetermined format of the second log data generated on the host side may be a syslog data format, that is, a data format supported by the log service process rsyslog.

[0072] In an embodiment of the present disclosure, a storage path identifier is generated based on source data of first log data; second log data in a predetermined format is generated based on the storage path identifier and the first log data, so that when the log data is unloaded from the host side, the data source of the first log data can be included. While realizing the log data unloading function, it supports storing the first log data according to the source data, which facilitates log data management.

[0073] Figure 3A The data flow diagram for generating the second log data according to the first embodiment of the present disclosure is schematically shown. Figure 3A As shown, in the first embodiment 300A, the storage path identifier 302 can be determined according to the source data 301 of the first log data 303, and the second log data 304 is generated according to the storage path identifier 302 and the first log data 303. Afterwards, the second log data 304 can be offloaded to the DPU through the log offloading channel matching the virtualization component.

[0074] In a specific embodiment, a storage path identifier is generated based on source data of the first log data, including: generating a storage path identifier based on first source sub-data, wherein the first source sub-data is used to characterize a virtualization component that generates the first log data; or generating a storage path identifier based on the first source sub-data and second source sub-data, wherein the second source sub-data is used to characterize a process in the virtualization component that is used to generate the first log data.

[0075] The source data may include first source sub-data and / or second source sub-data. The first source sub-data may be the identifiers and / or IP addresses of various virtualization components based on various operating systems, such as the identifiers of components such as qemu, libvirt, and Xen. The second source sub-data may be the identifier of the process that generates the first log data, for example, it may be the process name and / or process number. In the virtualization component, multiple processes (i.e., applications) may be virtualized, and thus, the process that generates the first log data in the virtualization component may be distinguished by the second source sub-data.

[0076] For example, a storage path identifier can be generated only based on the first source sub-data, and the second log data can be generated based on the storage path identifier and the first log data, so as to realize the log data storage of the DPU based on the granularity of the virtualization component. Alternatively, a storage path identifier can be generated based on the first source sub-data and the second source sub-data, and the second log data can be generated based on the storage path identifier and the first log data, so as to realize the log data storage of the DPU based on the granularity of the process within the virtualization component. The operation of generating the storage path identifier is as described above and will not be repeated here.

[0077] In an embodiment of the present disclosure, a storage path identifier is generated based on the first source sub-data, or the first source sub-data + the second source sub-data, so that the unloaded second log data can achieve targeted unloading at the virtualization component granularity or the process granularity, which is convenient for log data management and query.

[0078] Figure 3B The data flow diagram for generating the second log data according to the second embodiment of the present disclosure is schematically shown. Figure 3B As shown, in the second embodiment 300B, the source data of the first log data 303 may include: first source sub-data 3011 and second source sub-data 3012, according to which the storage path identifier 302 may be determined, and then the second log data 304 may be generated according to the storage path identifier 302 and the first log data 303. Afterwards, the second log data 304 may be offloaded to the DPU through a log offloading channel that matches the virtualization component.

[0079] In another specific embodiment, the virtualization component includes: a first component, the first component is used to simulate or manage a virtual machine; the method also includes: when the first log data is generated by a virtual machine simulated by the first component: generating a storage path identifier based on third source sub-data and fourth source sub-data, the third source sub-data is used to characterize the first component, and the fourth source sub-data is used to characterize the virtual machine; or, generating a storage path identifier based on the third source sub-data, the fourth source sub-data and the fifth source sub-data, wherein the fifth source sub-data characterizes the process in the virtual machine used to generate the first log data.

[0080] For the first component used to simulate or manage a virtual machine, the first log data may be generated by the first component or by the virtual machine simulated by the first component. In a host machine, multiple virtual machines can be simulated or managed by the first component, and multiple processes can be run in each virtual machine. In the case where the first log data is generated by the virtual machine simulated by the first component, the first component can obtain the first log data to be uninstalled from the virtual machine, and characterize the first component through the third source sub-data, characterize the virtual machine through the fourth source sub-data, and characterize the process in the virtual machine that generates the first log data through the fifth source sub-data.

[0081] For example, similar to the first source sub-data, the third source sub-data may be the IP address of the first component, the component identifier, etc., such as the component identifier may be qemu. The fourth source sub-data may be the identifier of the virtual machine, such as virtual machine No. 2.

[0082] It should be noted that the first source sub-data and the third source sub-data may be the same or different. When the first source sub-data and the third source sub-data are different, a newly added identifier may be used to indicate that the first component is a component for managing or simulating a virtual machine, and that the first log data is generated by the virtual machine. Alternatively, when the first source sub-data and the third source sub-data are the same, whether the fourth sub-data is included may be used to determine whether the first log data is generated by the first component or obtained from the virtual machine.

[0083] In one embodiment, the storage path identifier can be generated according to the third source sub-data and the fourth source sub-data, or the storage path identifier can be generated according to the third source sub-data, the fourth source sub-data and the fifth source sub-data. Afterwards, the second log data is generated according to the storage path identifier and the first log data. The operation of generating the storage path identifier is as described above and will not be repeated here.

[0084] In an embodiment of the present disclosure, a storage path identifier is generated based on the third source sub-data + the fourth source sub-data, or the third source sub-data + the fourth source sub-data + the fifth source sub-data, so that the unloaded second log data can achieve targeted unloading at the granularity of two-level virtual machines, or targeted unloading at the granularity of processes within three-level virtual machines, and log identification is implemented on the host side where the log data is unloaded, thereby facilitating log data management and query.

[0085] Figure 3C The data flow diagram for generating the second log data according to the third embodiment of the present disclosure is schematically shown. Figure 3C As shown, in the third embodiment 300C, the source data of the first log data 303 may include: third source sub-data 3013, fourth source sub-data 3014 and fifth source sub-data 3015. According to the third source sub-data 3013, fourth source sub-data 3014 and fifth source sub-data 3015, the storage path identifier 302 may be determined, and then the second log data 304 may be generated according to the storage path identifier 302 and the first log data 303. Afterwards, the second log data 304 may be offloaded to the DPU through a log offloading channel that matches the virtualization component.

[0086] For different virtualization components, the second log data can be unloaded to the DPU through the log unloading channel that matches the virtualization component. However, for multiple virtualization components, there may be multiple specific log unloading channels and unloading methods. In order to facilitate understanding of the process of unloading the second log data to the DPU, the first component, the second component, and the third component in the virtualization component will be used as examples for explanation below.

[0087] According to an embodiment of the present disclosure, for the first component, operation S220 includes: unloading the second log data to the data processor through a second log unloading interface between the first component and the data processor, wherein the second log unloading interface is created using a log library framework embedded in the first component.

[0088] The first component in the user state may be an application in the user state, such as libvirt and qemu.

[0089] The log library framework can be a lightweight framework used only for recording log data, and can be embedded in the compilation process of the first component. For example, the log library framework can be an open source, lightweight framework liblogging that can record log data. The log library framework embedded in the compilation process of the first component can be regarded as a part of the first component, not as an independent process, so the host does not need to consume additional resources to keep the process alive.

[0090] The second log offloading interface is a new interface added in the first component, which is used to generate second log data from the first log data of the first component in the global scope and the source data, and offload the second log data to the DPU. For example, the first log data in the global scope is also the first log data generated by the first component and the first log data obtained from the virtual machine.

[0091] The second log unloading interface can unload the second log data through a predetermined transmission protocol. For example, the second log data is unloaded through a connectionless transport layer protocol to reduce the impact of network fluctuations or blockages on the host machine and the virtual machine when unloading log data. For example, the User Datagram Protocol (UDP) can be used to connect to the network and transmit data with the log service process rsyslog on the DPU side.

[0092] For example, when compiling the first component, a second log unloading interface can be created using the log library framework embedded in the first component, and the second log unloading interface can be an interface in the form of a socket. When starting the first component, the first component can call the newly added second log unloading interface for initialization, and provide the source data of the first component, such as an identifier. The log library framework embedded in the first application creates a UDP socket and socket-related information based on the source data of the first component, and returns the initialized second log unloading interface to the first component. The second log data can then be directly unloaded through the second log unloading interface. For example, the above initialization process can be implemented through the log library framework liblogging.

[0093] In addition, for the first component embedded with the log library framework, the first component uses the log library framework to add a driver to generate second log data based on the first log data and the source data. For example, taking the DPU side log service process as rsyslog as an example, the newly added driver can generate a storage path identifier in the syslog format based on the source data, and use the storage path identifier as a data header and the first log data to form the second log data. Subsequently, the second log data is unloaded to the DPU through the second log unloading interface.

[0094] In an embodiment of the present disclosure, a log library framework that supports simple log data recording is embedded in the first component, and a second log unloading interface created by the log library framework can provide a unified interface for the entire first component, so that the first log data to be unloaded in the first component can generate second log data in the first component, and log unloading is performed through the second log unloading interface. In this way, the resources consumed by the simple log data recording function are less than the resources consumed by the existing log service process, which can reduce the resource consumption of the central processing unit (CPU) in the host system and the consumption of memory resources. In addition, through the unified second log unloading interface of the entire first component, the repeated resource consumption of creating interfaces and initializing interfaces can also be reduced, thereby reducing the consumption of host resources.

[0095] According to an embodiment of the present disclosure, the virtualization component includes a second component, which is used to support the system service of the host machine; for the second component, operation S220 includes: calling the system service manager of the second component, and unloading the second log data to the data processor through the first log unloading interface created by the system service manager, wherein the first log unloading interface is created by the system service manager according to the interface configuration information in the first configuration file.

[0096] For example, for the Linux operating system, the second component is used to support the system service of the Linux operating system on the host machine. The second component may be a kernel component, and the system service manager of the second component may be a native system service manager of the operating system, such as systemd.

[0097] The system service manager can provide log services. Therefore, after removing the separate log service process on the host side, for the second component, the system service manager can be modified so that the system service manager can create a new first log offloading interface for offloading the second log data to the DPU.

[0098] The first log offloading interface is created by the system service manager according to the interface configuration information in the first configuration file. In one embodiment, the first log offloading interface can be an interface in the form of a socket based on the UDP protocol. The first configuration file records the interface configuration information of the system service manager supporting the message service. When adding a new first log offloading interface, the interface configuration information can be added to the first configuration file so that the system service manager can create the first log offloading interface according to the interface configuration information in the first configuration file.

[0099] The interface configuration information may include various parameters such as input parameters, output parameters, and data formats of the first log unloading interface. Taking the log service process on the DPU side as rsyslog as an example, the interface configuration information may include a data format, such as a syslog format, so as to unload the second log data in the syslog format to the DPU through the second log unloading interface. In the first configuration file, the log service interface of each system service can be set to the first log unloading interface, and the identifier of the corresponding service can be set, so as to realize the log data unloading of multiple system services through a unified first log unloading interface.

[0100] For example, when starting the second component, a fixed-format message string is generated according to the interface configuration information passed in at startup and sent to the system service manager, such as systemd, to initialize the log service. After the system service manager parses the message string, it initializes the first log unloading interface between the log service process rsyslog on the DPU side. Afterwards, the second log data can be unloaded to the DPU through the first log unloading interface.

[0101] In the embodiment of the present disclosure, by using the system service manager of the second component to create the first log unloading interface, the complexity and workload of modifying the source code of each system service separately can be reduced. In addition, unloading the second log data through the second log unloading interface does not require the log service process of the host machine, achieving the technical effect of reducing the resource occupation of the host machine.

[0102] According to an embodiment of the present disclosure, the virtualization component includes: a third component, which is used to support the kernel service of the operating system of the host machine; for the third component, operation S220 includes: unloading the second log data to the data processor through a network channel between the third component and the data processor; wherein the network channel is constructed by the third component according to the network parameters when the operating system is started.

[0103] For example, for an operating system Linux, the third component may be a kernel component that supports a kernel service of the operating system Linux. In this case, the first log data is also referred to as a kernel log.

[0104] The network channel between the third component and the DPU can be created using a network console (netconsole). Network parameters can be added to the startup parameters of the kernel service of the operating system so that when the operating system is started, the network channel between the third component and the data processor is created according to the network parameters.

[0105] The network parameters may include a network address and port for starting the netconsole, which may be the network log and port of the log service process of the DPU, so as to call the netconsole to establish a network channel between the third component and the log service process of the DPU. After that, all log data of the kernel service will be unloaded to the DPU through the network link.

[0106] In an embodiment of the present disclosure, the network channel established between the third component and the data processor through the netconsole can continue to unload log data to the DPU when the network protocol stack is unavailable or the interrupt mechanism is abnormal, so as to ensure the stability of the log unloading function.

[0107] Figure 4A The following schematically shows an application scenario diagram of uninstalling the second log data according to an embodiment of the present disclosure. Figure 4A As shown, in embodiment 400A, the host machine M1 includes a first component C1, a second component C2, and a third component C3, wherein the first component C1 and the second component C2 run in a user state environment, and the third component C3 runs in a kernel state environment.

[0108] The first component C1 can unload the second log data 1 to the log service process P1 of the data processor M2 through the second log unloading interface, and the data processor M2 is also the DPU. The second component C2 can unload the second log data 2 to the log service process P1 of the data processor M2 through the first log unloading interface created by the system service manager P2. The third component C2 unloads the second log data 3 to the log service process P1 of the data processor M2 through the network channel between the third component C2 and the log service process P1 in the DPU.

[0109] In a specific embodiment, for a first component capable of simulating a virtual machine, first log data in the first component may be obtained from the virtual machine.

[0110] According to an embodiment of the present disclosure, the method also includes: for the first component, when the first log data is generated by a virtual machine simulated by the first component, receiving the first log data from the virtual machine to generate second log data based on the first log data and source data; and unloading the second log data to the data processor through a second log offloading interface.

[0111] The first log data is transmitted between the first component and the simulated virtual machine through the serial port. In this case, the first log data is also called a serial port log.

[0112] For the first log data generated by the virtual machine, the virtualization component receives the first log data from the virtual machine through the serial port, and reuses the log unloading method of the first component to unload the log data. For example, second log data is generated based on the first log data and the source data; and the second log data is unloaded to the data processor through the second log unloading interface. The unloading of log data is as described above and will not be repeated here. For example, each time after obtaining the first log data from the virtual machine, the log unloading method of the first component can be reused to unload the log data.

[0113] It can be understood that, in this embodiment, the source data may include the third source sub-data and the fourth source sub-data; or, include the third source sub-data, the fourth source sub-data and the fifth source sub-data.

[0114] For example, each time the first component receives the first log data from the virtual machine, it writes the received first log data to the specified character device. In this embodiment, a character device can be added, and the write operation of the character device follows the definition of the log library framework. After the first log data in the character device is processed into the second log data, the second log data is unloaded through the second log unloading interface.

[0115] In the embodiment of the present disclosure, for the first log data generated by the virtual machine, the log data is unloaded by reusing the second log unloading interface of the first component, and the unloading of the serial port log can be supported without modification, thereby reducing the modification workload.

[0116] Figure 4B The following schematically shows an application scenario diagram of uninstalling the second log data according to another embodiment of the present disclosure. Figure 4B As shown, in embodiment 400B, the host machine M1 includes a first component 1 C11, a first component 2 C12, and a virtual machine V1 simulated by the first component 2 C12. The first component 1 C11 can be a user-mode application, such as libvirt, which can generate second log data 1 according to the first log data and source data generated by itself, and unload the second log data 1 to the log service process P1 of the data processor M2 through the second log unloading interface.

[0117] The first component 2 C12 can be qemu. When the virtual machine V1 is started, the first component 2 C12 simulates a serial port device for the virtual machine V1. The type of the serial port device can be a socket, etc. After each time the virtual machine V1 writes the first log data 4 to the serial port device, the write operation on the serial port device will trigger a port input and output (Port I / O, PIO) event. After judgment in the virtual machine kvm built into the kernel, a virtual machine exit (VM exit) event occurs, and then the first log data 4 is transmitted to the first component 2 C12 through the simulated serial port. The log data unloading method of the first component 2 C12 is reused, and the second log data 4 is generated according to the transmitted first log data 4 and the source data, and the second log data 4 is unloaded to the log service process P1 of the data processor M2 through the second log unloading interface.

[0118] However, when receiving the first log data through the serial port, the serial port writes one byte each time, and the first component also processes the serial port data in units of one byte. The single-byte writing method is too inefficient. When receiving too much first log data through the serial port, the CPU performance of the virtual machine will be seriously affected, and the virtual CPU (Virtual CPU, VCPU) delay of the virtual machine will have very obvious jitter, affecting the operation of other tasks on the VCPU.

[0119] To this end, the present disclosure further optimizes the transmission process for the first log data generated by the virtual machine.

[0120] Figure 5 The flowchart of generating the first log data according to the first log sub-data and the second log sub-data according to an embodiment of the present disclosure is schematically shown.

[0121] like Figure 5 As shown, embodiment 500 includes operations S510 to S520.

[0122] In operation S510, first log sub-data of a target data length is obtained from a virtual machine.

[0123] In operation S520, first log data is determined based on first log sub-data and at least one second log sub-data, wherein the second log sub-data having a target data length is stored in a cache of the first component and the second log sub-data is obtained from the virtual machine before the first log sub-data.

[0124] When the first component starts the simulated virtual machine, it will pre-allocate a fixed-size cache for each started virtual machine to cache the log sub-data obtained from the virtual machine. In this embodiment, the log sub-data of the target data length can be obtained through the serial port between the virtual machine and the first component, and the obtained log sub-data is stored in the cache. For the purpose of distinction, the currently obtained log sub-data is referred to as the first log sub-data; the log sub-data obtained from the same virtual machine and stored in the cache before the first log sub-data is referred to as the second log sub-data.

[0125] In this embodiment, log unloading is not performed every time a first log sub-data is received, but first log data is determined according to the first log sub-data and at least one second log sub-data, and then the first log data is regarded as the first log data to be unloaded.

[0126] The target data length may be a single byte, and the first log data determined according to the first log sub-data and the at least one second log sub-data is greater than the single byte.

[0127] In a specific embodiment, the first log data may have a predetermined data length, so that the first log data of the predetermined data length is generated according to the first log sub-data and at least one second log sub-data, and operations S210 and S220 are performed to complete the unloading of the log data.

[0128] In an embodiment of the present disclosure, a cache is set in the first component to store at least one second log sub-data of a target data length, and the first log data is determined based on the first log sub-data and at least one second log sub-data. The existing method of receiving log data from the virtual machine each time and unloading it is optimized to merging the log data received from the virtual machine and unloading it once, thereby reducing the number of times the log data is unloaded under the same data volume, reducing the delay jitter of the virtual machine VCPU, and improving the performance of the virtual machine.

[0129] According to an embodiment of the present disclosure, first log data to be uninstalled is determined based on first log sub-data and at least one second log sub-data, including: if the data volume of the first log sub-data and at least one second log sub-data meets a predetermined condition, the first log sub-data and at least one second log sub-data are determined as first log data; if the data volume of the first log sub-data and at least one second log sub-data does not meet the predetermined condition, the first log sub-data is stored in the cache as newly added second log sub-data, and new first log sub-data is obtained from the virtual machine until the predetermined condition is met.

[0130] The predetermined condition may be one of the following: the sum of the data volume of the first log sub-data and at least one second log sub-data reaches a data volume threshold, reaches the cache upper limit allocated to the virtual machine in the first component, is within a predetermined data volume range, or reaches a log line break condition.

[0131] In the case where the data volume of the first log sub-data and at least one second log sub-data meets a predetermined condition, the first log sub-data and at least one second log sub-data can be combined into the first log data according to the receiving time, and the first log data is the first log data to be uninstalled. After the first log sub-data and at least one second log sub-data are determined as the first log data, the cache can also be cleared. In the case where the data volume of the first log sub-data and at least one second log sub-data does not meet the predetermined condition, the first log sub-data is stored in the cache as the newly added second log sub-data, and new first log sub-data is obtained from the virtual machine until the predetermined condition is met. The first log data is determined. After determining the first log data, operations S210 and S220 can be performed, and the specific operations are as above and will not be repeated here.

[0132] In an embodiment of the present disclosure, by judging the data amount of the first log sub-data and at least one second log sub-data, and determining the first log sub-data and at least one second log sub-data as the first log data when the data amount is reached, so as to perform subsequent log data unloading, the unloading operation of the first log data generated by the virtual machine is optimized and the performance of the virtual machine is improved.

[0133] According to an embodiment of the present disclosure, the method further includes: in a case where it is determined that the first log sub-data represents a log line break, determining at least one second log data as the first log data.

[0134] The log data generated by the virtualization component and the virtual machine is a line-based log, so the first log sub-data of a single byte can be a carriage return character and / or a line feed character. Different systems can use carriage return characters and / or line feed characters to perform log line feeds. For example, some systems use carriage return characters + line feed characters to represent line feeds, while some systems only use line feed characters to represent line feeds.

[0135] Whether the first log sub-data represents a log line break can be determined by judging the content of the first log sub-data. For example, if the first log data is determined to be a line break character, the first log sub-data is determined to represent a log line break; otherwise, the first log sub-data is determined to represent a log line break.

[0136] In the case where it is determined that the first log sub-data represents a log line break, at least one second log data can be directly combined into the first log data according to the receiving time to perform a subsequent log data unloading function. In addition, after at least one second log data is combined into the first log data according to the receiving time, the cache can be cleared and new first log sub-data can continue to be received to realize skipping the line break to unload the log data.

[0137] In the embodiment of the present disclosure, by determining whether the first log sub-data represents a log line break, and in the case of representing a log line break, determining at least one second log data as the first log data, the stored first log data conforms to the habit of reading by line while reducing the number of log data unloading times, which is convenient for subsequent log data queries. In addition, the judgment of log line breaks in the first component can also filter out special symbols such as line breaks and carriage returns from the first log data, shielding different processing methods for log line breaks in different systems, and avoiding garbled characters in the log data stored by the DPU due to incompatibility of system line breaks.

[0138] Figure 6 The flowchart of log data storage and unloading for a virtual machine according to an embodiment of the present disclosure is schematically shown.

[0139] like Figure 6 As shown, embodiment 600 includes operations S601 to S606.

[0140] In operation S601, a first component is started.

[0141] In operation S602, first log sub-data is received through a serial port.

[0142] In operation S603, it is determined whether the first log sub-data is a line break character. If it is determined to be yes, operation S605 is performed; otherwise, operation S604 is performed.

[0143] In operation S604, it is determined whether the cache is full. If it is, operation S605 is executed; if it is not, the process returns to operation S602, stores the first log sub-data in the cache, and continues to receive the first log sub-data through the serial port.

[0144] In operation S605, first log data is generated. If operation S603 is judged as yes, operation S605 is performed to use at least one second log sub-data in the cache as the first log data; if operation S604 is judged as yes, the first log data is generated according to the first log sub-data and the second log sub-data in the cache.

[0145] In operation S606, the first log data is unloaded and the cache is cleared. For example, the second log data is generated according to the first log data and the source data, and the second log data is unloaded to the DPU through the second log unloading interface, and the cache is requested. After executing operation S606, the operation can return to operation S602 to continue to receive new first log sub-data.

[0146] In addition, for the first component that does not enable the log offloading function, at least one second log sub-data can still be stored in the cache to reduce the number of times the log sub-data generated by the virtual machine is written to the host machine, thereby reducing the VCPU delay jitter of the virtual machine.

[0147] In summary, the embodiments of the present disclosure achieve the log data unloading of the virtualization component on the host side under the DPU architecture without modifying the code, thereby avoiding the environmental risks and performance jitters caused to the virtual machine by logging into the host to view the log data in the operation and maintenance scenario. In addition, by removing the log service process on the host machine and unloading the log data using the log unloading channel that matches the virtualization component, the host side's dependence on the log service process is reduced, making the host machine more lightweight. In addition, for the first component, by optimizing the processing of serial port logs through caching, the performance impact of writing serial port logs on the virtual machine can be reduced, thereby improving the user experience of the cloud service function.

[0148] Figure 7 The following is a flow chart schematically showing a method for processing log data according to another embodiment of the present disclosure. Figure 7 As shown, the log data processing method of another embodiment 700 includes operation S710.

[0149] In operation S710 , in response to receiving second log data acquired from a host machine, first log data is stored in a storage path corresponding to source data of the first log data according to the second log data.

[0150] The second log data is generated by the host machine in response to detecting the log uninstallation instruction of the virtualization component, based on the first log data to be uninstalled and the source data, and the virtualization component represents the component in the host machine used to provide virtualization services; the second log data is transmitted through the log uninstallation channel that matches the host machine and the virtualization component. This part of the operation is described above, and will not be repeated here.

[0151] For example, the host machine may parse the received second log data to obtain the first log data and source data of the first log data, then determine a storage path corresponding to the source data, and store the first log data in the storage path corresponding to the source data.

[0152] The storage path may be a storage path of a storage space within the DPU, or a storage path outside the DPU, such as a storage path under other hardware. For an external storage path, the DPU is responsible for determining the storage path, and forwarding or storing the first log data to the external storage path.

[0153] In an embodiment of the present disclosure, since the second log data received from the host machine includes the first log data and the source data of the first log data, targeted storage of the first log data can be achieved based on the source data, which facilitates subsequent log data management and query of virtualized components on the DPU and simplifies the log data viewing operation on the DPU side.

[0154] According to an embodiment of the present disclosure, according to the second log data, the first log data is stored in a storage path corresponding to the source data of the first log data, including: calling a log service process to parse the second log data into the first log data and a storage path identifier; determining the source data and the storage path corresponding to the source data according to the second configuration file and the storage path identifier; and storing the first log data in the storage path corresponding to the source data.

[0155] The log service process may be a process capable of implementing complex log processing, routing, formatting, and storage functions, for example, may be rsyslog.

[0156] The second log data has a predetermined format. For example, for the log service process rsyslog, the second log data can be a syslog data format. The log service process can parse the second log data in a predetermined format, and parse to obtain the first log data and a storage path identifier as a data header. The parsing process can be the reverse process of generating the second log data according to the storage path identifier and the first log data, and the specific operation corresponds to the generation process, which will not be repeated here.

[0157] The second configuration file may include a unique mapping of the storage path identifier and the source data of the first log data, and / or a storage path corresponding to the source data, whereby the log service process of the DPU may filter and screen the storage path identifier according to the second configuration file to determine the storage path corresponding to the source data. Subsequently, the first log data is stored in the storage path.

[0158] In an embodiment of the present disclosure, the second log data is parsed into the first log data and a storage path identifier, and the source data and the storage path corresponding to the source data are determined according to the second configuration file and the storage path identifier; and the first log data is stored in the storage path corresponding to the source data. While realizing the log data unloading function, the DPU can support the storage of the first log data according to the source data, which facilitates log data management.

[0159] Figure 8 A data flow diagram for storing first log data according to another embodiment of the present disclosure is schematically shown. Figure 8 As shown, in embodiment 800, the second log data 801 can be parsed to obtain a storage path identifier 802 and the first log data 803. The DPU can call the log service process to determine the source data 805 and the storage path 806 corresponding to the source data 805 according to the second configuration file 804 and the storage path identifier 802. Thus, the first log data 803 can be stored in the storage path 806.

[0160] According to an embodiment of the present disclosure, a virtualization component includes a first component; determining source data and a storage path corresponding to the source data based on a second configuration file and a storage path identifier, including: determining at least one source sub-data based on the storage path identifier, wherein the source sub-data includes at least one of the following: first source sub-data, second source sub-data, third source sub-data, fourth source sub-data, and fifth source sub-data; and determining the storage path based on the second configuration file and the at least one source sub-data.

[0161] The first component is used to simulate or manage virtual machines, for example, it may be qemu or libvirt.

[0162] Similar to the process of generating the storage path identifier, when the first component on the host side generates the first log data itself, the first source sub-data can be determined according to the storage path identifier; or, the first source sub-data and the second source sub-data can be determined according to the storage path identifier. When the first component on the host side is generated by a simulated virtual machine, the third source sub-data and the fourth source sub-data can be determined according to the storage path identifier; or, the third source sub-data, the fourth source sub-data and the fifth source sub-data can be determined according to the storage path identifier.

[0163] In other embodiments, the storage path identifier may also be generated based on at least one of the following: first source sub-data, second source sub-data, third source sub-data, fourth source sub-data and fifth source sub-data; thus, the source sub-data determined by the DPU may include at least one of the above.

[0164] The second configuration file may also include a unique mapping between each source sub-data and a storage path, so that after determining at least one source sub-data, the storage path is determined according to the second configuration file and the at least one source sub-data.

[0165] In an embodiment of the present disclosure, by determining the storage path through parsing at least one source sub-data and a second configuration file, it is possible to implement log data storage at multiple granularities such as virtualization components, processes within virtualization components, virtual machines, and processes within virtual machines, so that specific log data at multiple granularities can be directly obtained on the DPU side based on the storage path.

[0166] Fig. 9 The block diagram of the log data processing device according to the embodiment of the present disclosure is schematically shown. Fig. 9 As shown, the log data processing device 900 includes a generating module 910 and an unloading module 920 .

[0167] The generation module 910 is used to generate second log data in response to detecting a log uninstallation instruction of a virtualization component according to first log data to be uninstalled in the virtualization component and source data of the first log data, wherein the virtualization component represents a component in a host machine for providing virtualization services.

[0168] The unloading module 920 is used to unload the second log data to the data processor through a log unloading channel matching the virtualization component, so that the data processor stores the first log data to a storage path corresponding to the source data according to the second log data.

[0169] According to an embodiment of the present disclosure, the generation module 910 includes:

[0170] The first generating submodule is used to generate a storage path identifier according to source data of the first log data.

[0171] The second generating submodule is used to generate second log data in a predetermined format according to the storage path identifier and the first log data; wherein the predetermined format includes a data format supported by the log service process of the data processor.

[0172] According to an embodiment of the present disclosure, the first generating submodule includes: a first generating unit or a second generating unit.

[0173] The first generating unit is used to generate a storage path identifier according to first source sub-data, wherein the first source sub-data is used to characterize a virtualization component that generates first log data.

[0174] The second generating unit is used to generate a storage path identifier according to the first source sub-data and the second source sub-data, wherein the second source sub-data is used to characterize a process in the virtualization component for generating the first log data.

[0175] According to an embodiment of the present disclosure, the virtualization component includes: a first component, the first component is used to simulate or manage a virtual machine; the first generation submodule also includes: a third generation unit or a fourth generation unit. In the case where the first log data is generated by the virtual machine simulated by the first component,

[0176] The third generating unit is used to generate a storage path identifier according to third source sub-data and fourth source sub-data, the third source sub-data is used to characterize the first component, and the fourth source sub-data is used to characterize the virtual machine.

[0177] The fourth generating unit is used to generate a storage path identifier according to the third source sub-data, the fourth source sub-data and the fifth source sub-data, wherein the fifth source sub-data represents a process in the virtual machine for generating the first log data.

[0178] According to an embodiment of the present disclosure, the virtualization component includes a second component, and the second component is used to support the system service of the host machine; the uninstall module 920 includes:

[0179] The first unloading submodule is used to call the system service manager of the second component to unload the second log data to the data processor through the first log unloading interface created by the system service manager, wherein the first log unloading interface is created by the system service manager according to the interface configuration information in the first configuration file.

[0180] According to an embodiment of the present disclosure, the virtualization component includes: a third component, the third component is used to support the kernel service of the operating system of the host machine; the uninstall module 920 includes:

[0181] The second unloading submodule is used to unload the second log data to the data processor through the network channel between the third component and the data processor; wherein the network channel is constructed by the third component according to the network parameters when the operating system is started.

[0182] According to an embodiment of the present disclosure, the uninstall module 920 includes:

[0183] The third unloading submodule is used to unload the second log data to the data processor through the second log unloading interface between the first component and the data processor, wherein the second log unloading interface is created using the log library framework embedded in the first component.

[0184] According to an embodiment of the present disclosure, the log data unloading device 900 further includes: for the first component,

[0185] The first acquisition submodule is used to receive the first log data from the virtual machine when the first log data is generated by the virtual machine simulated by the first component, so as to generate the second log data according to the first log data and the source data.

[0186] The fourth unloading submodule is used to unload the second log data to the data processor through the second log unloading interface.

[0187] According to an embodiment of the present disclosure, the log data unloading device 900 further includes: for the first component, when the first log data is generated by a virtual machine simulated by the first component,

[0188] The second acquisition submodule is used to acquire first log sub-data of target data length from the virtual machine.

[0189] A determination submodule is used to determine the first log data based on the first log sub-data and at least one second log sub-data, wherein the second log sub-data with a target data length is stored in the cache of the first component, and the second log sub-data is obtained from the virtual machine before the first log sub-data.

[0190] According to an embodiment of the present disclosure, the determining submodule includes: a first determining unit and a second determining unit.

[0191] a first determining unit, configured to determine the first log sub-data and the at least one second log sub-data as the first log data if the data amounts of the first log sub-data and the at least one second log sub-data meet a predetermined condition;

[0192] The second determination unit is used to store the first log sub-data as newly added second log sub-data in the cache when the data amount of the first log sub-data and at least one second log sub-data does not meet the predetermined condition, and obtain new first log sub-data from the virtual machine until the predetermined condition is met.

[0193] According to an embodiment of the present disclosure, the determination submodule further includes: a third determination unit, configured to determine at least one second log data as the first log data when it is determined that the first log sub-data represents a log line break.

[0194] Fig.10 A block diagram of a log data processing device according to another embodiment of the present disclosure is schematically shown. Fig.10 As shown, the log data processing device 1000 includes a storage module 1010, which is used to store the first log data in a storage path corresponding to the source data of the first log data in response to receiving the second log data obtained from the host machine, according to the second log data; wherein the second log data is generated by the host machine in response to detecting the log uninstallation instruction of the virtualization component, according to the first log data to be uninstalled and the source data, and the virtualization component represents the component in the host machine for providing virtualization services; the second log data is transmitted through the log uninstallation channel that matches the host machine and the virtualization component.

[0195] According to an embodiment of the present disclosure, the storage module 1010 includes: a parsing submodule, a path determination submodule and a storage submodule. The log service process is called, and the parsing submodule is used to parse the second log data into the first log data and the storage path identifier.

[0196] The path determination submodule is used to determine the source data and the storage path corresponding to the source data according to the second configuration file and the storage path identifier.

[0197] The storage submodule is used to store the first log data in a storage path corresponding to the source data.

[0198] According to an embodiment of the present disclosure, the virtualization component includes a first component; the path determination submodule includes: a source determination unit and a path determination unit.

[0199] The source determination unit is used to determine at least one source sub-data according to the storage path identifier, wherein the source sub-data includes at least one of the following: first source sub-data, second source sub-data, third source sub-data, fourth source sub-data and fifth source sub-data.

[0200] The path determination unit is used to determine a storage path according to the second configuration file and at least one source sub-data.

[0201] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0202] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0203] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0204] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.

[0205] Fig.11A block diagram of an electronic device suitable for implementing a log data processing method according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0206] like Fig.11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 to a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the electronic device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0207] Multiple components in the electronic device 1100 are connected to an input / output (I / O) interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0208] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as the log data processing method. For example, in some embodiments, the log data processing method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the log data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the log data processing method in any other appropriate manner (eg, by means of firmware).

[0209] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0210] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0211] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0212] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0213] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0214] A computer system may include clients and servers. The client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server in combination with a blockchain.

[0215] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0216] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A log data processing method, comprising: In response to detecting a log uninstallation instruction of a virtualization component, generating second log data according to first log data to be uninstalled in the virtualization component and source data of the first log data, wherein the virtualization component represents a component in a host machine for providing a virtualization service; The second log data is unloaded to a data processor through a log unloading channel matching the virtualization component, so that the data processor stores the first log data in a storage path corresponding to the source data according to the second log data.

2. The method according to claim 1, wherein: The generating the second log data according to the first log data to be uninstalled in the virtualization component and the source data of the first log data comprises: Generate a storage path identifier according to source data of the first log data; The second log data in a predetermined format is generated according to the storage path identifier and the first log data; wherein the predetermined format includes a data format supported by a log service process of the data processor.

3. The method according to claim 2, wherein: The step of generating a storage path identifier according to the source data of the first log data includes: generating the storage path identifier according to first source sub-data, wherein the first source sub-data is used to characterize a virtualization component that generates the first log data; or The storage path identifier is generated according to the first source sub-data and the second source sub-data, wherein the second source sub-data is used to characterize a process in the virtualization component for generating the first log data.

4. The method according to claim 3, wherein: The virtualization component includes: a first component, the first component is used to simulate or manage a virtual machine; the method further includes: in the case where the first log data is generated by a virtual machine simulated by the first component, generating the storage path identifier according to third source sub-data and fourth source sub-data, wherein the third source sub-data is used to characterize the first component and the fourth source sub-data is used to characterize the virtual machine; or The storage path identifier is generated according to the third source sub-data, the fourth source sub-data and the fifth source sub-data, wherein the fifth source sub-data represents a process in the virtual machine for generating the first log data.

5. The method according to any one of claims 1 to 4, wherein: The virtualization component includes a second component, and the second component is used to support the system service of the host machine; The step of unloading the first log data to the data processor through a log unloading channel matching the virtualization component comprises: Call the system service manager of the second component, and unload the two log data to the data processor through the first log unloading interface created by the system service manager, wherein the first log unloading interface is created by the system service manager according to the interface configuration information in the first configuration file.

6. The method according to any one of claims 1 to 4, wherein: The virtualization component includes: a third component, the third component is used to support the kernel service of the operating system of the host machine; the unloading of the second log data to the data processor through the log unloading channel matching the virtualization component includes: The second log data is unloaded to the data processor via a network channel between the third component and the data processor; wherein the network channel is constructed by the third component according to network parameters when the operating system is started.

7. The method according to any one of claims 1 to 4, wherein: The step of unloading the second log data to a data processor through a log unloading channel matching the virtualization component includes: The second log data is unloaded to the data processor via a second log unloading interface between the first component and the data processor, wherein the second log unloading interface is created using a log library framework embedded in the first component.

8. The method according to claim 7, further comprising: For the first component, In a case where the first log data is generated by a virtual machine simulated by the first component, receiving the first log data from the virtual machine to generate the second log data according to the first log data and the source data; as well as The second log data is unloaded to the data processor through the second log unloading interface.

9. The method according to any one of claims 1 to 8, further comprising: For the first component, when the first log data is generated by a virtual machine simulated by the first component, Acquire first log sub-data of target data length from the virtual machine; The first log data is determined based on the first log sub-data and at least one second log sub-data, wherein the second log sub-data having a target data length is stored in a cache of the first component, and the second log sub-data is obtained from the virtual machine before the first log sub-data.

10. The method according to claim 9, wherein: The determining the first log data to be uninstalled according to the first log sub-data and at least one second log sub-data includes: In a case where the data amounts of the first log sub-data and at least one of the second log sub-data meet a predetermined condition, determining the first log sub-data and at least one of the second log sub-data as the first log data; When the data amount of the first log sub-data and at least one of the second log sub-data does not meet the predetermined condition, the first log sub-data is stored in the cache as newly added second log sub-data, and new first log sub-data is obtained from the virtual machine until the predetermined condition is met.

11. The method according to claim 9 or 10, further comprising: When it is determined that the first log sub-data represents a log line break, at least one of the second log data is determined as the first log data.

12. A log data processing method, comprising: In response to receiving second log data obtained from the host machine, storing the first log data in a storage path corresponding to source data of the first log data according to the second log data; The second log data is generated by the host machine in response to detecting a log uninstallation instruction of a virtualization component, based on the first log data to be uninstalled and the source data, and the virtualization component represents a component in the host machine for providing virtualization services; the second log data is transmitted through a log uninstallation channel that matches the host machine and the virtualization component.

13. The method according to claim 12, wherein: According to the second log data, storing the first log data in a storage path corresponding to the source data of the first log data includes: calling a log service process, parsing the second log data into the first log data and a storage path identifier; Determining the source data and the storage path corresponding to the source data according to the second configuration file and the storage path identifier; and The first log data is stored in the storage path corresponding to the source data.

14. The method according to claim 12, wherein: The virtualization component includes a first component; the determining the source data and the storage path corresponding to the source data according to the second configuration file and the storage path identifier includes: Determine at least one source sub-data according to the storage path identifier, wherein the source sub-data includes at least one of the following: first source sub-data, second source sub-data, third source sub-data, fourth source sub-data, and fifth source sub-data; and The storage path is determined according to the second configuration file and at least one of the source sub-data.

15. A log data processing device, comprising: A generating module, configured to generate second log data in response to detecting a log uninstallation instruction of a virtualization component according to first log data to be uninstalled in the virtualization component and source data of the first log data, wherein the virtualization component represents a component in a host machine for providing a virtualization service; An unloading module is used to unload the second log data to a data processor through a log unloading channel matching the virtualization component, so that the data processor stores the first log data to a storage path corresponding to the source data according to the second log data.

16. The device according to claim 15, wherein: The generation module comprises: A first generating submodule, used for generating a storage path identifier according to source data of the first log data; The second generating submodule is used to generate the second log data in a predetermined format according to the storage path identifier and the first log data; wherein the predetermined format includes a data format supported by the log service process of the data processor.

17. The device according to claim 16, wherein: The first generation submodule comprises: A first generating unit is configured to generate the storage path identifier according to first source sub-data, wherein the first source sub-data is used to characterize a virtualization component that generates the first log data; or The second generating unit is configured to generate the storage path identifier according to the first source sub-data and the second source sub-data, wherein the second source sub-data is used to characterize a process in the virtualization component for generating the first log data.

18. The device according to claim 17, wherein: The virtualization component includes: a first component, the first component is used to simulate or manage a virtual machine; the first generation submodule also includes: in the case where the first log data is generated by a virtual machine simulated by the first component, A third generating unit is configured to generate the storage path identifier according to third source sub-data and fourth source sub-data, wherein the third source sub-data is used to characterize the first component and the fourth source sub-data is used to characterize the virtual machine; or The fourth generating unit is used to generate the storage path identifier according to the third source sub-data, the fourth source sub-data and the fifth source sub-data, wherein the fifth source sub-data represents the process in the virtual machine for generating the first log data.

19. The device according to any one of claims 15 to 18, wherein: The virtualization component includes a second component, and the second component is used to support the system service of the host machine; the uninstall module includes: The first unloading submodule is used to call the system service manager of the second component to unload the second log data to the data processor through the first log unloading interface created by the system service manager, wherein the first log unloading interface is created by the system service manager according to the interface configuration information in the first configuration file.

20. The device according to any one of claims 15 to 18, wherein: The virtualization component includes: a third component, the third component is used to support the kernel service of the operating system of the host machine; the uninstall module includes: The second unloading submodule is used to unload the second log data to the data processor through the network channel between the third component and the data processor; wherein the network channel is constructed by the third component according to the network parameters when the operating system is started.

21. The device according to any one of claims 15 to 18, wherein: The uninstall module comprises: The third unloading submodule is used to unload the second log data to the data processor through a second log unloading interface between the first component and the data processor, wherein the second log unloading interface is created using a log library framework embedded in the first component.

22. The apparatus according to claim 21, further comprising: For the first component, a first acquisition submodule, configured to, when the first log data is generated by a virtual machine simulated by the first component, receive the first log data from the virtual machine, so as to generate the second log data according to the first log data and the source data; as well as A fourth unloading submodule is used to unload the second log data to the data processor through the second log unloading interface.

23. The device according to any one of claims 15 to 22, further comprising: For the first component, when the first log data is generated by a virtual machine simulated by the first component, A second acquisition submodule, used for acquiring first log sub-data of target data length from the virtual machine; A determination submodule is used to determine the first log data based on the first log sub-data and at least one second log sub-data, wherein the second log sub-data having a target data length is stored in the cache of the first component, and the second log sub-data is obtained from the virtual machine before the first log sub-data.

24. The device according to claim 23, wherein: The determination submodule comprises: a first determining unit, configured to determine the first log sub-data and at least one of the second log sub-data as the first log data if the data amounts of the first log sub-data and at least one of the second log sub-data meet a predetermined condition; A second determination unit is configured to store the first log sub-data as newly added second log sub-data in the cache and obtain new first log sub-data from the virtual machine when the data volume of the first log sub-data and at least one of the second log sub-data does not meet the predetermined condition, until the predetermined condition is met.

25. The apparatus according to claim 23 or 24, wherein the determining submodule further comprises: The third determining unit is configured to determine at least one of the second log data as the first log data when it is determined that the first log sub-data represents a log line break.

26. A log data processing device, comprising: A storage module, configured to, in response to receiving second log data acquired from a host machine, store the first log data in a storage path corresponding to source data of the first log data according to the second log data; The second log data is generated by the host machine in response to detecting a log uninstallation instruction of a virtualization component, based on the first log data to be uninstalled and the source data, and the virtualization component represents a component in the host machine for providing virtualization services; the second log data is transmitted through a log uninstallation channel that matches the host machine and the virtualization component.

27. The device according to claim 26, wherein: The storage module includes: calling a log service process, A parsing submodule, used for parsing the second log data into the first log data and a storage path identifier; a path determination submodule, configured to determine the source data and the storage path corresponding to the source data according to a second configuration file and the storage path identifier; and The storage submodule is used to store the first log data in the storage path corresponding to the source data.

28. The device according to claim 26, wherein The virtualization component includes a first component; the path determination submodule includes: a source determination unit, configured to determine at least one source sub-data according to the storage path identifier, wherein the source sub-data includes at least one of the following: first source sub-data, second source sub-data, third source sub-data, fourth source sub-data, and fifth source sub-data; and A path determination unit is used to determine the storage path according to the second configuration file and at least one of the source sub-data.

29. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of claims 1 to 14.

30. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14.

31. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images