Log collection method, device and system, electronic equipment and storage medium

Abstract

The invention provides a log collection method, device and system, electronic equipment and a storage medium, and relates to the technical field of containers, and the method comprises the steps: obtaining a job application object corresponding to a target job sent by a job server and a parameter object corresponding to the job application object; creating a job execution container group corresponding to the job application object based on the job application object, setting a log configuration file path of the job execution container group based on a first local directory of the parameter object, and setting a log configuration file path of the job execution container group based on a server address and a server path of the parameter object, and mounting the first local directory of the job execution container group to a network storage space. Therefore, the log recording modes of the multiple job execution container groups can be adjusted by maintaining the log configuration file of the network storage space, the convenience of configuring the log recording modes of the job execution container groups is improved, and the efficiency of collecting the log information of the job execution container groups is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of container technology, and in particular to a log collection method, device, system, electronic device, and storage medium. Background Art

[0002] With the continuous development of computer technology, more and more computing needs are met through container orchestration architecture. The log collection system orchestrates user needs into container groups for execution. Each container group generates log information during operation, and different users and different businesses have different requirements for the way of recording log information.

[0003] In the related art, a container group defines the way log information is recorded through a log configuration file. Technical personnel usually need to configure the corresponding log configuration file in the container image according to the log information recording requirements when making a container image. When the container group is deployed according to the container image, the log configuration file in the container image is stored in the local directory of the container group, so that the container group records log information based on the log configuration file during operation.

[0004] The above log information collection method for container groups requires re-making container images containing different log configuration files when adjusting the log recording method, which makes it difficult to adjust the log configuration file and has low efficiency in recording log information. Summary of the invention

[0005] The embodiments of the present application provide a log collection method, device, system, electronic device and storage medium to solve the problems in the related art that the log configuration file of the container group is difficult to adjust and the efficiency of recording log information is low.

[0006] In a first aspect, an embodiment of the present application provides a log collection method, which is applied to a container orchestration platform, and the method includes:

[0007] Obtaining a job application object corresponding to a target job sent by a job server and a parameter object corresponding to the job application object; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space;

[0008] Creating a job execution container group corresponding to the job application object based on the job application object, and setting a log configuration file path of the job execution container group based on the first local directory;

[0009] Based on the server address and the server path, the first local directory of the job execution container group is mounted to the network storage space, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file.

[0010] Optionally, after creating a job execution container group corresponding to the job application object based on the job application object, the method includes:

[0011] Generate a log path corresponding to the job execution container group based on the first local directory;

[0012] A log record file corresponding to the job execution container group is generated in the log path; wherein the log record file is used to record log information written by the job execution container group in the network storage space.

[0013] Optionally, generating a log path corresponding to the job execution container group based on the first local directory includes:

[0014] Obtaining a job application identifier corresponding to the job application object;

[0015] The job application identifier and the first local directory are concatenated to obtain the log path.

[0016] Optionally, the concatenating the job application identifier and the first local directory to obtain the log path includes:

[0017] Obtaining a container group identifier corresponding to the job execution container group;

[0018] A log subdirectory is determined according to the job application identifier, a log file name is determined according to the container group identifier, and the first local directory, the log subdirectory and the log file name are concatenated to obtain the log path.

[0019] Optionally, generating a log record file corresponding to the job execution container group in the log path includes:

[0020] An entry point script of the container image corresponding to the job execution container group is executed, the entry point script generates the log record file based on the log configuration file and the log path, and stores the log configuration file based on the log path.

[0021] In a second aspect, an embodiment of the present application provides a log collection method, which is applied to a job server, and the method includes:

[0022] In response to a target job, a job application object corresponding to the target job is generated, and a parameter object corresponding to the job application object is configured; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space;

[0023] The job application object and the parameter object are sent to a container orchestration platform, so that the container orchestration platform creates a job execution container group corresponding to the job application object based on the job application object, sets a log configuration file path of the job execution container group based on the first local directory, and mounts the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file.

[0024] Optionally, the method further comprises:

[0025] Generate a container group identifier corresponding to the job execution container group;

[0026] The container group identifier is sent to the container orchestration platform, so that the container orchestration platform determines the log subdirectory according to the job application identifier, determines the log file name according to the container group identifier, and concatenates the first local directory, the log subdirectory and the log file name to obtain the log path.

[0027] Optionally, the method further comprises:

[0028] The job application identifier is obtained from the container orchestration platform, and the job application identifier and the container group identifier are sent to a user client, so that the user client queries the log information in the network storage space based on the server address, the server path, the job application identifier, and the container group identifier to obtain a log query result.

[0029] In a third aspect, an embodiment of the present application provides a log collection device, which is applied to a container orchestration platform, and the device includes:

[0030] An acquisition module, used to acquire a job application object corresponding to a target job sent by a job server and a parameter object corresponding to the job application object; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space;

[0031] A first path module, configured to create a job execution container group corresponding to the job application object based on the job application object, and to set a log configuration file path of the job execution container group based on the first local directory;

[0032] A mounting module is used to mount the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file.

[0033] Optionally, the device further comprises:

[0034] A second path module, used to generate a log path corresponding to the job execution container group based on the first local directory;

[0035] A log record file module is used to generate a log record file corresponding to the job execution container group in the log path; wherein the log record file is used to record the log information written by the job execution container group in the network storage space.

[0036] Optionally, the second path module includes:

[0037] A job application identification submodule, used to obtain the job application identification corresponding to the job application object;

[0038] The log path determination submodule is used to concatenate the job application identifier and the first local directory to obtain the log path.

[0039] Optionally, the log path determination submodule includes:

[0040] A container group identifier obtaining unit, used to obtain a container group identifier corresponding to the job execution container group;

[0041] The log path concatenation unit is used to determine the log subdirectory according to the job application identifier, determine the log file name according to the container group identifier, and concatenate the first local directory, the log subdirectory and the log file name to obtain the log path.

[0042] Optionally, the log file module includes:

[0043] The entry point script execution submodule is used to execute the entry point script of the container image corresponding to the job execution container group, and the entry point script generates the log record file based on the log configuration file and the log path, and stores the log configuration file based on the log path.

[0044] In a fourth aspect, an embodiment of the present application provides a log collection device, which is applied to a job server, and the device includes:

[0045] A generation module, configured to generate a job application object corresponding to the target job in response to the target job, and configure a parameter object corresponding to the job application object; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space;

[0046] A sending module is used to send the job application object and the parameter object to a container orchestration platform, so that the container orchestration platform creates a job execution container group corresponding to the job application object based on the job application object, sets a log configuration file path of the job execution container group based on the first local directory, and mounts the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file.

[0047] Optionally, the device further comprises:

[0048] A container group identifier generating module, used to generate a container group identifier corresponding to the job execution container group;

[0049] A container group identifier sending module is used to send the container group identifier to the container orchestration platform, so that the container orchestration platform determines the log subdirectory according to the job application identifier, determines the log file name according to the container group identifier, and concatenates the first local directory, the log subdirectory and the log file name to obtain the log path.

[0050] Optionally, the device further comprises:

[0051] A query module is used to obtain the job application identifier from the container orchestration platform, and send the job application identifier and the container group identifier to a user client, so that the user client queries the log information in the network storage space based on the server address, the server path, the job application identifier and the container group identifier to obtain a log query result.

[0052] In a fifth aspect, an embodiment of the present application further provides a log collection system, including a job server and a container orchestration platform;

[0053] The job server is used to generate a job application object corresponding to the target job in response to the target job, configure a parameter object corresponding to the job application object, and send the job application object and the parameter object to the container orchestration platform; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space;

[0054] The container orchestration platform is used to obtain a job application object corresponding to a target job sent by a job server and a parameter object corresponding to the job application object, create a job execution container group corresponding to the job application object based on the job application object, set a log configuration file path of the job execution container group based on the first local directory, and mount the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file; wherein the parameter object includes the first local directory, the server address of the network storage space storing the log configuration file, and the server path of the network storage space.

[0055] In a sixth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a log collection method as in the first aspect or the second aspect.

[0056] In the seventh aspect, an embodiment of the present application further provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the log collection method of the first aspect or the second aspect.

[0057] In an embodiment of the present application, the method includes: obtaining a job application object corresponding to a target job sent by a job server and a parameter object corresponding to the job application object; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space; creating a job execution container group corresponding to the job application object based on the job application object, and setting the log configuration file path of the job execution container group based on the first local directory; mounting the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file. The log configuration file path corresponding to each job execution container group can be set based on the first local directory, and the first local directory can be mounted to the network storage space, so that the job execution container group can directly obtain the log configuration file from the network storage space based on the log configuration file path and record log information based on the log configuration file, so that the log recording method of multiple job execution container groups can be adjusted by maintaining the log configuration file of the network storage space, which helps to improve the convenience of configuring the log recording method of the job execution container group and improves the efficiency of collecting log information of the job execution container group.

[0058] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0060] Figure 1 This is a flow chart of the steps of a log collection method provided by an embodiment of the present application;

[0061] Figure 2 is a flowchart of another log collection method provided by an embodiment of the present application;

[0062] Figure 3 It is a log recording architecture provided by an embodiment of the present application;

[0063] Figure 4 This is a flow chart of steps of another log collection method provided by an embodiment of the present application;

[0064] Figure 5 It is a log collection system provided by an embodiment of the present application;

[0065] Figure 6 It is an interactive schematic diagram provided in an embodiment of the present application;

[0066] Figure 7 It is a structural diagram of a log collection device provided in an embodiment of the present application;

[0067] Figure 8 is a structural diagram of another log collection device provided in an embodiment of the present application;

[0068] Fig. 9 It is a logic block diagram of an electronic device according to an embodiment of the present application;

[0069] Fig.10 It is a logic block diagram of an electronic device of another embodiment provided by an embodiment of the present application. DETAILED DESCRIPTION

[0070] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0071] Figure 1 This is a flow chart of the steps of a log collection method provided by an embodiment of the present application, which is applied to a container orchestration platform, such as Figure 1 As shown, the method includes:

[0072] Step 101, obtain the job application object corresponding to the target job sent by the job server and the parameter object corresponding to the job application object; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space.

[0073] A job server is a server that can receive, manage, schedule and execute computing tasks (jobs, etc.). In distributed computing and big data environments, job servers can usually be used to coordinate and manage the execution of large-scale tasks to ensure the effective use of computing resources and the correct execution of tasks. For example, in the Apache Spark system, the job server can be a Spark Job Server, and users can submit jobs to the job server through the REST API. The job server can generate corresponding job application objects based on the job requests submitted by the user, and schedule the job to be executed in the Spark cluster through the job application object. It should be noted that different distributed computing engines can use different job servers. For example, Apache Hadoop can use YARN as a job server, which is not specifically limited in the embodiments of the present application.

[0074] When the distributed computing engine is Apache Spark, the job application object may be SparkApplication; when the distributed computing engine is Apache Hadoop, the job application object may be MapReduce Job or YARN Application, which is not specifically limited in the embodiments of the present application.

[0075] A container orchestration platform (such as Kubernetes, Docker Swarm, etc.) is a management platform for managing and scheduling containerized applications. The container orchestration platform can automatically deploy and manage corresponding container groups (Pods) according to the job application objects to actually execute the corresponding jobs. Among them, the container group is the basic deployment unit of the container orchestration platform. A container group is a collection of one or more containers. Containers in the same container group can share storage and network resources.

[0076] The parameter object corresponding to the job application object refers to the configuration information used to control the job execution behavior. For example, in Apache Spark, the parameter object corresponding to the job application object may be Spark Conf, and in Apache Hadoop, the parameter object corresponding to the job application object may be JobConf.

[0077] In an embodiment of the present application, the job server can generate a corresponding job application object and a parameter object in response to a user's job request. The parameter object can include a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space. Among them, the log configuration file can define information such as the logging method, logging format, and logging location of the job execution container group deployed based on the job application object to indicate how to record log information during the operation of the job execution container group. The server address of the network storage space can indicate the storage server address of the network storage space where the above-mentioned first local directory needs to be mounted, and the server path of the network storage space can indicate the path of the above-mentioned network storage space on the above-mentioned storage server. Specifically, when generating a parameter object, the job server can configure the preset first local directory, the server address of the network storage space, and the server path of the network storage space in the parameter object, so that after the container orchestration platform receives the parameter object, it can parse it to obtain the first local directory, the server address, and the server path.

[0078] The above-mentioned storage server may include but is not limited to a Network File System (NFS), a Common Internet File System (CIFS), Ceph, etc., and is not specifically limited in the embodiments of the present application.

[0079] For example, when the distributed computing engine is Spark, the job application object can be generated by the job server by specifying each parameter in the above parameter object.

[0080] io.k8s.sparkoperator.v1beta2ji interface built.

[0081] It should be noted that when the job server constructs a job application object, it can set the log configuration file path corresponding to each type of job execution container group in the parameter object corresponding to the job application object, so that the job execution container group can correctly obtain the log configuration file and record log information based on the log configuration file.

[0082] For example, when the distributed computing engine is Spark, the following parameters can be added to Spark Conf to configure the log configuration file path corresponding to the job execution container group of driver and executor types:

[0083] spark.executor.extraJavaOptions=-Dlog4j.configuration=file: / opt / log4j / log4j.properties;

[0084] spark.driver.extraJavaOptions=-Dlog4j.configuration=file: / opt / log4j / log4j.properties.

[0085] Step 102: Create a job execution container group corresponding to the job application object based on the job application object, and set a log configuration file path of the job execution container group based on the first local directory.

[0086] In an embodiment of the present application, a job execution container group corresponding to a job application object may be created by a container orchestration platform based on the job application object, wherein the job execution container group may include but is not limited to a driver container group (driver pod), an executor container group (executor pod), etc., which are not specifically limited in the embodiment of the present application. The log configuration file path of the job execution container group may indicate the path for the job execution container group to obtain a log configuration file (such as log4j.properties or log4j2.xml, etc.), and the log configuration file may define information such as the log recording mode, log recording format, and log recording location to indicate how to record log information during the operation of the job execution container group.

[0087] In an embodiment of the present application, the container orchestration platform may, after creating and deploying a job execution container group corresponding to the job application object, modify the log configuration file path corresponding to the job execution container group to a path in the first local directory; or the log configuration file path in the container image (Docker Image) required to deploy the job execution container group may be set in the first local directory to create a job execution container group whose log configuration file path is located in the first local directory; or the container image whose log configuration file path has been set to the first local directory may be directly obtained from the image repository, and a job execution container group whose log configuration file path is located in the first local directory may be directly created based on the container image, which is not specifically limited in the embodiment of the present application.

[0088] In an embodiment of the present application, the method for determining the log configuration file path may depend on the actual storage location of the log configuration file in the network storage space. If the log configuration file is directly stored in the network storage space, the log configuration file path may be obtained by concatenating the first local directory and the file name of the log configuration file. If the log configuration file is stored in a subdirectory of the network storage space, the log configuration file path may also be obtained by concatenating the first local directory, the subdirectory under the first local directory, and the file name of the log configuration file.

[0089] Specifically, the log configuration file path of the log configuration file can be defined in the entry point script of the container image. For example, when the distributed computing engine is Spark, the log configuration file path can be set in the entrypoint.sh file through the following statement, and then the container image is made based on DockerFile, and the container image is pushed to the container image warehouse. The container orchestration platform can generate a job execution container group based on these container images, and realize the setting of the log configuration file path of the job execution container group based on the first local directory.

[0090] log4jfile=" / opt / log4j / log4j.properties".

[0091] Among them, log4jfile is the log configuration file path setting parameter, / opt / log4j represents the first local directory, and / opt / log4j / log4j.properties represents the log configuration file path.

[0092] Furthermore, in an embodiment of the present application, different types of job execution container groups may correspond to different first local directories, server addresses, and server paths, that is, different types of job execution container groups may be mounted to different network storage spaces. Specifically, when the job server constructs a job application object, it may set the first local directories, server addresses, and server paths corresponding to each type of job execution container group in the parameter object corresponding to the job application object. The first local directories and network storage spaces corresponding to different types of job execution container groups may be the same or different, and the embodiment of the present application does not specifically limit this.

[0093] For example, if the network storage space to be mounted is NFS, the following parameters can be added to Spark Conf, Spark-default.conf or volume config to indicate that the first local directory and network storage space corresponding to the driver type job application object need to be mounted:

[0094] spark.kubernetes.driver.volumes.nfs.oasis.mount.readOnly=false;

[0095] / / Set the NFS network storage space mounted by the driver type job execution container group to be readable and writable;

[0096] spark.kubernetes.driver.volumes.nfs.oasis.mount.path= / opt / log4j;

[0097] / / Set the first local directory of the driver type job execution container group to mount the network storage space to / opt / log4j;

[0098] spark.kubernetes.driver.volumes.nfs.oasis.options.server=${nfs_server_addr ess};

[0099] / / Set the server address of the network storage space that needs to be mounted by the driver type job execution container group;

[0100] spark.kubernetes.driver.volumes.nfs.oasis.options.path= / nfs / log;

[0101] / / Set the server path of the network storage space on the NFS server that needs to be mounted by the driver type job execution container group.

[0102] You can add the following parameters in Spark Conf to indicate the first local directory and network storage space corresponding to the executor type job application object that needs to be mounted:

[0103] spark.kubernetes.executor.volumes.nfs.oasis.mount.readOnly=false;

[0104] / / Set the NFS network storage space mounted by the executor type job execution container group to be readable and writable;

[0105] spark.kubernetes.executor.volumes.nfs.oasis.mount.path= / opt / log4j;

[0106] / / Set the first local directory of the executor type job execution container group to mount the network storage space to / opt / log4j;

[0107] spark.kubernetes.executor.volumes.nfs.oasis.options.server=${nfs_server_ad dress};

[0108] / / Set the server address of the network storage space that needs to be mounted by the executor type job execution container group;

[0109] spark.kubernetes.executor.volumes.nfs.oasis.options.path= / nfs / log;

[0110] / / Set the server path of the network storage space on the NFS server that needs to be mounted by the executor type job execution container group.

[0111] Step 103, based on the server address and the server path, mount the first local directory of the job execution container group to the network storage space, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file.

[0112] In an embodiment of the present application, the first local directory of each job execution container group can be mounted to a network storage space based on the server address and server path, and each execution container group can be started to run, so that each job execution container group can read the log configuration file from the network storage space through the log configuration file path in the local path format, and record log information based on the log recording method, recording format and other parameters in the log configuration file.

[0113] The container orchestration platform can mount the first local directory of the job execution container group to the network storage space based on the server address and server path. During the operation of the job execution container group, the access operation to its first local directory is actually access to the network storage space. Mounting refers to associating a storage space with a specific path so that users or programs can access the storage space through the path. Mounting can mount an external storage volume (such as network storage or cloud storage) to a local directory of the container, so that the application in the container can access and use these external storage volumes.

[0114] Since the first local directories of all execution container groups of the target job will be mounted to the network storage space, all execution container groups of the target job can obtain the log configuration file from the same network storage space. By maintaining the log configuration file of the network storage space, the logging method of all execution container groups can be updated. When updating the logging method, there is no need to remake and modify the container image.

[0115] A log collection method provided by an embodiment of the present application includes: obtaining a job application object corresponding to a target job sent by a job server and a parameter object corresponding to the job application object; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space; creating a job execution container group corresponding to the job application object based on the job application object, and setting the log configuration file path of the job execution container group based on the first local directory; mounting the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file. The log configuration file path corresponding to each job execution container group can be set based on the first local directory, and the first local directory can be mounted to the network storage space, so that the job execution container group can directly obtain the log configuration file from the network storage space based on the log configuration file path and record log information based on the log configuration file, so that the log recording mode of multiple job execution container groups can be adjusted by maintaining the log configuration file of the network storage space, which helps to improve the convenience of configuring the log recording mode of the job execution container group and improves the efficiency of collecting log information of the job execution container group.

[0116] Reference Figure 2 , Figure 2 is a flowchart of another log collection method provided by an embodiment of the present application, such as Figure 2 As shown, the method includes:

[0117] Step 201, obtain the job application object corresponding to the target job sent by the job server and the parameter object corresponding to the job application object; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space.

[0118] This step can refer to step 101 and will not be described in detail in this embodiment of the present application.

[0119] Step 202: Create a job execution container group corresponding to the job application object based on the job application object, and set a log configuration file path of the job execution container group based on the first local directory.

[0120] This step can be referred to as step 102, and will not be described in detail in this embodiment of the present application.

[0121] Step 203: Generate a log path corresponding to the job execution container group based on the first local directory.

[0122] In an embodiment of the present application, in order to improve the maintainability and ease of use of log information, the log path corresponding to the job execution container group can be determined according to the first local directory, so that the job execution container group can store the log information to the network storage space according to the log path, so as to uniformly manage the log information generated by each job execution container group through the network storage space.

[0123] In an embodiment of the present application, the container orchestration platform can determine the log path corresponding to each job execution container group based on the first local directory, and generate a log record file corresponding to the job execution container group in the log path, so as to record the log information written by the job execution container group in the above network storage space through the first local directory through the log record file. Among them, the log path can only include the first local directory, such as / opt / log4j / , and the log path can also include the first local directory and the log file name, such as / opt / / log4j / log / pod1.log.

[0124] It should be noted that in order to enable the job execution container group to correctly write the log record file to the log path, the log record path parameter can be configured as the first local directory in the container image that generates the job execution container group. Through the above operation, the container in the job execution container group can write the log information to the first local directory during operation. Since the first local directory is mounted with the network storage space, the log information of the job execution container group is actually written to the above network storage space.

[0125] Specifically, the log path can be obtained by concatenating the first local directory and the log file name, where the log file name can correspond to the job execution container group. For example, if the first local directory is

[0126] / opt / log4j. For job execution container group 1, you can set the log file name A, and for job execution container group 2, you can set the log file name B. Then the log path corresponding to job execution container group 1 can be / opt / log4j / log file name A, and the log path corresponding to job execution container group 2 can be

[0127] / opt / log4j / log file name B.

[0128] Optionally, step 203 may include:

[0129] Sub-step 2031, obtaining the job application identifier corresponding to the job application object.

[0130] After the job server builds the job application object, it submits the job application object to the container orchestration platform. When the operation controller (such as Spark Operator) listens to the job application object, it builds the submission command of the target job (such as Spark Submit) according to the parameter object corresponding to the job application object, and assigns a job application identifier (such as

[0131] The application identifier and submission command are handed over to the container orchestration platform for further processing.

[0132] In an embodiment of the present application, after the container orchestration platform obtains the job application identifier corresponding to the target job, the job application identifier can be used to configure the log path corresponding to the job execution container group of the target job.

[0133] Sub-step 2032: concatenate the job application identifier and the first local directory to obtain the log path.

[0134] In an embodiment of the present application, the job application identifier and the first local directory can be concatenated to obtain a log path. Specifically, in one embodiment, the job application identifier can be used as the log file name, and the first local directory and the log file name can be concatenated to obtain a log path. For example, if the first local directory is / opt / log4j and the job application identifier is "application_16329876", then the concatenation can obtain a log path of / opt / log4j / application_16329876. In addition, the extension of the log file can also be concatenated together to improve the accuracy of log file search. Continuing with the above example, if the log file extension is .log, then the concatenation can obtain a log path of

[0135] / opt / log4j / application_16329876.log.

[0136] In another implementation, the job application identifier can be used as a subdirectory, and the first local directory, the subdirectory, and the log file name can be concatenated to obtain the log path. For example, if the first local directory is / opt / log4j, the log file name A can be set for job execution container group 1, and the job application identifier is "application_16329876", then the concatenation can obtain the log path as

[0137] / opt / log4j / application_16329876 / application_16329876. In addition, you can also concatenate the log file extension to improve the accuracy of log file search. Using the above example, if the log file extension is .log, the concatenation can get the log path as

[0138] / opt / log4j / application_16329876 / application_16329876.log.

[0139] By obtaining the job application identifier corresponding to the job application object sent by the job server; concatenating the job application identifier and the first local directory to obtain the log path, the log path can be configured based on the job application identifier, so that different job application objects can store log information through different log paths, which helps to improve the query convenience of the logs generated by each job in the cluster, and can effectively isolate the operation path of each job application object in the network storage space, which helps to improve data security.

[0140] Optionally, sub-step 2032 may include:

[0141] Sub-step 20321: Obtain the container group identifier corresponding to the job execution container group.

[0142] In an embodiment of the present application, the container group identifier (HOSTNAME) corresponding to the job execution container group can also be obtained, and a log path can be further generated through the container group identifier, so that the log path can distinguish different job execution container groups, further improving the query convenience and security of log information recording.

[0143] Specifically, the container group identifier can be determined by the job name (appName) set by the job server for the target job and the container group type of the job execution container group. For example, for a Driver type job execution container group, its corresponding container group identifier can be ${appName}-driver$, and for an Executor type job execution container group, its corresponding container group identifier can be

[0144] ${appName}-exec-${n}, where exec is the abbreviation of Executor, and n is the ID or number of the Executor. The above container group identifier is generated by the job server and transmitted to the container orchestration platform. The container orchestration platform can also generate the above container group identifier based on the job name and container group type of the job execution container group obtained from the job server, as well as the ID or number of the job execution container group of the Executor type. After obtaining the above container group identifier, the container orchestration platform can define it in the environment variable corresponding to the job execution container group for future use.

[0145] Sub-step 20322, determining the log sub-directory according to the job application identifier, determining the log file name according to the container group identifier, concatenating the first local directory, the log sub-directory and the log file name to obtain the log path.

[0146] In an embodiment of the present application, the job application identifier can be used as the log subdirectory, the container group identifier can be used as the log file name, and the first local directory can be used as the log main directory. The above directories and log file names can be concatenated to obtain the log path.

[0147] For example, if the first local directory is / opt / log4j, the job application identifier is

[0148] ${“application_16329876”}, the container group ID corresponding to job execution container group 1 is

[0149] ${appName}-driver$, the container group identifier corresponding to job execution container group 2 is ${appName}-exec-${1}, then the log path corresponding to job execution container group 1 can be

[0150] / opt / log4j / ${“application_16329876”} / ${appName}-driver$, the log path corresponding to the job execution container group 2 can be

[0151] / opt / log4j / ${"application_16329876"} / ${appName}-exec-${1}. It should be noted that the above log file name can also be obtained by concatenating the container group identifier and the extension, for example

[0152] ${appName}-driver$.log.

[0153] By obtaining the container group identifier corresponding to the job execution container group; determining the log subdirectory according to the job application identifier, determining the log file name according to the container group identifier, concatenating the first local directory, the log subdirectory and the log file name, and obtaining the log path, the log path can be configured based on the job application identifier and the container group identifier, so that different job application objects can store log information through different log paths, and different job execution container groups can store log information through different log paths, which helps to improve the query convenience of the logs generated by each job execution container group in the cluster, and can effectively isolate the operation path of each job execution container group in the network storage space, which helps to improve data security.

[0154] Step 204: Generate a log record file corresponding to the job execution container group in the log path; wherein the log record file is used to record log information written by the job execution container group in the network storage space.

[0155] In an embodiment of the present application, a log record file corresponding to the job execution container group may be generated in a log path, and the log record file may be used to record log information written by the job execution container group in a network storage space.

[0156] By generating a log path corresponding to the job execution container group based on the first local directory, and generating a log record file corresponding to the job execution container group in the log path, the job execution container group can write log information to the log record file actually located in the network storage space through the log path containing the first local directory, which helps to improve the convenience of log information management, and the log information can continue to exist after the job execution container group is released, which helps to improve the information security of the log information.

[0157] Reference Figure 3 , Figure 3 A log record architecture provided by an embodiment of the present application is shown, such as Figure 3 As shown, the job execution container group can store log information in the network storage space through the local log path. The user client can query the log information uniformly managed in the network storage space directly or through the job server to obtain the log query results, which helps to improve the security of log records and the efficiency of log queries.

[0158] Optionally, an entry point script of the container image corresponding to the job execution container group is executed, and the entry point script generates the log record file based on the log configuration file and the log path, and stores the log configuration file based on the log path.

[0159] In the embodiment of the present application, the log record file can not only be directly generated by the container orchestration platform through its own program, but also an entry point script with log record file generation code can be written in advance, and the entry point script is used to make a container image, so that when the container orchestration platform executes the container group by executing the container image deployment job, the entry point script can be executed to complete the generation of the log record file.

[0160] For example, when the distributed computing engine is Spark, the following code may be defined in the entrypoint script entrypoint.sh:

[0161] log4jfile=" / opt / log4j / log4j.properties";

[0162] / / Set the file path of the log configuration file log4jfile;

[0163] newlog4jfile=" / opt / spark / work-dir / log4j.properties";

[0164] / / Set the file path of the logging file newlog4jfile.

[0165] if [-f "$log4jfile"]; then;

[0166] / / Check whether the file pointed to by the log configuration file log4jfile exists

[0167] cp"$log4jfile""$newlog4jfile";

[0168] / / If the file pointed to by the log configuration file log4jfile exists, copy the log configuration file log4jfile to the path specified by newlog4jfile

[0169] sed-i s / ${SPARK_APPLICATION_ID} / '"$SPARK_APPLICATION_ID"' / g'"$newlog4jfile";

[0170] / / Use sed command to replace the ${SPARK_APPLICATION_ID} placeholder in the newlog4jfile file with the environment variable

[0171] The actual value of SPARK_APPLICATION_ID.

[0172] fi

[0173] / / End the if statement code block.

[0174] Then use the above entrypoint.sh to create a container image.

[0175] It should be noted that in the above log template file log4j.properties, a file path template of the log record file can be defined, and the file path is based on the first local directory, the job application identifier and the container group identifier. Exemplarily, the following file path template can be defined in the log template file log4j.properties:

[0176] log4j.appender.file.File= / opt / / log4j / log / ${SPARK_APPLICATION_ID} / ${HOSTNAME}.log;

[0177] Where log4j.appender.file.File represents the file path of the log file generated based on the log configuration file, / opt / / log4j / log represents the first local directory,

[0178] ${SPARK_APPLICATION_ID} represents the job application ID, HOSTNAME}.log, and HOSTNAME represents the container group ID. Therefore, when using the log configuration file to generate a log record file, the job application ID and container group ID corresponding to the actual job execution container group can be used to replace SPARK_APPLICATION_ID and HOSTNAME in the above file path template respectively, and the file path of the log record file corresponding to the job execution container group can be obtained.

[0179] For example, when the distributed computing engine is Spark and the first local directory is / opt / / log4j / log, the overall log directory corresponding to a target job may be as follows:

[0180] / opt / / log4j / log;

[0181] / opt / / log4j / log / ${SPARK_APPLICATION_ID};

[0182] / opt / / log4j / log / ${SPARK_APPLICATION_ID} / appname-driver.log;

[0183] / opt / / log4j / log / ${SPARK_APPLICATION_ID} / appname-exec-1.log;

[0184] / opt / / log4j / log / ${SPARK_APPLICATION_ID} / appname-exec-n.log.

[0185] Another log collection method provided by an embodiment of the present application includes: obtaining a job application object corresponding to a target job sent by a job server and a parameter object corresponding to the job application object; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space; creating a job execution container group corresponding to the job application object based on the job application object, and setting the log configuration file path of the job execution container group based on the first local directory; mounting the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file. The log configuration file path corresponding to each job execution container group can be set based on the first local directory, and the first local directory can be mounted to the network storage space, so that the job execution container group can directly obtain the log configuration file from the network storage space based on the log configuration file path and record log information based on the log configuration file, so that the log recording method of multiple job execution container groups can be adjusted by maintaining the log configuration file of the network storage space, which helps to improve the convenience of configuring the log recording method of the job execution container group and improves the efficiency of collecting log information of the job execution container group.

[0186] Reference Figure 4 , Figure 4 is a flow chart of steps of another log collection method provided by an embodiment of the present application, such as Figure 4 As shown, the method includes:

[0187] Step 301, in response to a target job, generates a job application object corresponding to the target job, and configures a parameter object corresponding to the job application object; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space.

[0188] Step 302: Send the job application object and the parameter object to the container orchestration platform, so that the container orchestration platform creates a job execution container group corresponding to the job application object based on the job application object, sets the log configuration file path of the job execution container group based on the first local directory, and mounts the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file.

[0189] Optionally, the log path of the job execution container group can be set through the following steps A1 to A2:

[0190] Step A1: Generate a container group identifier corresponding to the job execution container group.

[0191] Step A2: sending the container group identifier to the container orchestration platform, so that the container orchestration platform determines the log subdirectory according to the job application identifier, determines the log file name according to the container group identifier, and concatenates the first local directory, the log subdirectory and the log file name to obtain the log path.

[0192] Optionally, the job application identifier is obtained from the container orchestration platform, and the job application identifier and the container group identifier are sent to a user client, so that the user client queries the log information in the network storage space based on the server address, the server path, the job application identifier and the container group identifier to obtain a log query result.

[0193] Another log collection method provided by an embodiment of the present application includes: in response to a target job, generating a job application object corresponding to the target job, and configuring a parameter object corresponding to the job application object; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space; sending the job application object and the parameter object to a container orchestration platform, so that the container orchestration platform creates a job execution container group corresponding to the job application object based on the job application object, sets a log configuration file path of the job execution container group based on the first local directory, and mounts the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file. It is possible to set the log configuration file path corresponding to each job execution container group based on the first local directory, and mount the first local directory to the network storage space, so that the job execution container group can directly obtain the log configuration file from the network storage space based on the log configuration file path and record log information based on the log configuration file. Therefore, by maintaining the log configuration file of the network storage space, the log recording method of multiple job execution container groups can be adjusted, which helps to improve the convenience of configuring the log recording method of the job execution container group and improves the efficiency of collecting log information of the job execution container group.

[0194] Corresponding to the method provided in the log collection method embodiment of the present application, see Figure 5 , the present application also provides a log collection system, such as Figure 5 As shown, in this embodiment, the log collection system 50 includes a job server 51 and a container orchestration platform 52:

[0195] The job server is used to generate a job application object corresponding to the target job in response to the target job, configure a parameter object corresponding to the job application object, and send the job application object and the parameter object to the container orchestration platform; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space.

[0196] The container orchestration platform is used to obtain a job application object corresponding to a target job sent by a job server and a parameter object corresponding to the job application object, create a job execution container group corresponding to the job application object based on the job application object, and set a log configuration file path of the job execution container group based on the first local directory, and mount the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file; wherein the parameter object includes the first local directory, the server address of the network storage space storing the log configuration file, and the server path of the network storage space.

[0197] The above-mentioned job server and container orchestration platform can also be used to execute other steps in the above-mentioned embodiments, which will not be described in detail in the embodiments of this application.

[0198] Reference Figure 6 , Figure 6 An interactive schematic diagram provided by an embodiment of the present application is shown, such as Figure 6 As shown in the figure, when the distributed computing engine is Apache Spark, the user initiates the target job to the job server (Job Server) through the user client. The job server generates the job application object (Spark App Object) corresponding to the target job and the parameter object corresponding to the job application object. The job server can also specify the job application identifier corresponding to the job application object and the container group identifier corresponding to the job execution container group in the parameter object. The job server then submits the job application object and parameter object to the container orchestration platform (K8Smaster), which operates the controller

[0199] (Spark Operator) generates a job application identifier (SPARK_APPLICATION_ID) for the job application object. The container orchestration platform can obtain the container group identifier from the job server, and then the container orchestration platform creates a job execution container group (driver pod and executor pod), so that the first local directory of the job execution container group is mounted to the network storage space, such as Figure 6The driver container group, executor container group 1 and executor container group n shown in the figure, the container orchestration platform generates a corresponding log path for each job execution container group according to the job application identifier, the first local directory in the parameter object, and the container group identifier, and establishes a log record file corresponding to each job execution container group in the network storage space. The container orchestration platform can also send the job application identifier to the job server, and the job server forwards the job application identifier to the user client. The user client can determine the log path to be queried based on the server address, the server path, the job application identifier and the container group identifier, and send a log query instruction to the job server based on the log path. The job server queries the network storage space according to the log query instruction and returns the log query result to the user client.

[0200] A log collection system provided by an embodiment of the present application can set a log configuration file path corresponding to each job execution container group based on a first local directory, and mount the first local directory to a network storage space, so that the job execution container group can directly obtain the log configuration file from the network storage space based on the log configuration file path and record log information based on the log configuration file, thereby adjusting the log recording method of multiple job execution container groups by maintaining the log configuration file of the network storage space, which helps to improve the convenience of configuring the log recording method of the job execution container group and improves the efficiency of collecting log information of the job execution container group.

[0201] Corresponding to the method provided in the log collection method embodiment of the present application, see Figure 7 , the present application also provides a log collection device structure diagram, which is applied to the container orchestration platform, such as Figure 7 As shown, in this embodiment, the log collection device 70 includes:

[0202] The acquisition module 71 is used to acquire the job application object corresponding to the target job sent by the job server and the parameter object corresponding to the job application object; wherein the parameter object includes the first local directory, the server address of the network storage space storing the log configuration file, and the server path of the network storage space;

[0203] A first path module 72, configured to create a job execution container group corresponding to the job application object based on the job application object, and set a log configuration file path of the job execution container group based on the first local directory;

[0204] The mounting module 73 is used to mount the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file.

[0205] Optionally, the device further comprises:

[0206] A second path module, used to generate a log path corresponding to the job execution container group based on the first local directory;

[0207] A log record file module is used to generate a log record file corresponding to the job execution container group in the log path; wherein the log record file is used to record the log information written by the job execution container group in the network storage space.

[0208] Optionally, the second path module includes:

[0209] A job application identification submodule, used to obtain the job application identification corresponding to the job application object;

[0210] The log path determination submodule is used to concatenate the job application identifier and the first local directory to obtain the log path.

[0211] Optionally, the log path determination submodule includes:

[0212] A container group identifier obtaining unit, used to obtain a container group identifier corresponding to the job execution container group;

[0213] The log path concatenation unit is used to determine the log subdirectory according to the job application identifier, determine the log file name according to the container group identifier, and concatenate the first local directory, the log subdirectory and the log file name to obtain the log path.

[0214] Optionally, the log file module includes:

[0215] The entry point script execution submodule is used to execute the entry point script of the container image corresponding to the job execution container group, and the entry point script generates the log record file based on the log configuration file and the log path, and stores the log configuration file based on the log path.

[0216] A log collection device provided in an embodiment of the present application can set a log configuration file path corresponding to each job execution container group based on a first local directory, and mount the first local directory to a network storage space, so that the job execution container group can directly obtain the log configuration file from the network storage space based on the log configuration file path and record log information based on the log configuration file, thereby adjusting the log recording method of multiple job execution container groups by maintaining the log configuration file of the network storage space, which helps to improve the convenience of configuring the log recording method of the job execution container group and improves the efficiency of collecting log information of the job execution container group.

[0217] Corresponding to the method provided in the log collection method embodiment of the present application, see Figure 8 , the present application also provides another log collection device structure diagram, which is applied to the job server, such as Figure 8 As shown, in this embodiment, the log collection device 80 includes:

[0218] A generating module 81 is used to generate a job application object corresponding to the target job in response to the target job, and configure a parameter object corresponding to the job application object; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space;

[0219] The sending module 82 is used to send the job application object and the parameter object to the container orchestration platform, so that the container orchestration platform creates a job execution container group corresponding to the job application object based on the job application object, sets the log configuration file path of the job execution container group based on the first local directory, and mounts the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file.

[0220] Optionally, the device further comprises:

[0221] A container group identifier generating module, used to generate a container group identifier corresponding to the job execution container group;

[0222] A container group identifier sending module is used to send the container group identifier to the container orchestration platform, so that the container orchestration platform determines the log subdirectory according to the job application identifier, determines the log file name according to the container group identifier, and concatenates the first local directory, the log subdirectory and the log file name to obtain the log path.

[0223] Optionally, the device further comprises:

[0224] A query module is used to obtain the job application identifier from the container orchestration platform, and send the job application identifier and the container group identifier to a user client, so that the user client queries the log information in the network storage space based on the server address, the server path, the job application identifier and the container group identifier to obtain a log query result.

[0225] Another log collection device provided by an embodiment of the present application is capable of setting a log configuration file path corresponding to each job execution container group based on a first local directory, and mounting the first local directory to a network storage space, so that the job execution container group can directly obtain the log configuration file from the network storage space based on the log configuration file path and record log information based on the log configuration file, thereby adjusting the log recording method of multiple job execution container groups by maintaining the log configuration file of the network storage space, which helps to improve the convenience of configuring the log recording method of the job execution container group and improves the efficiency of collecting log information of the job execution container group.

[0226] Fig. 9 6 is a logic block diagram of an electronic device 600 according to an embodiment of the present application. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0227] Reference Fig. 9 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0228] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0229] The memory 604 is used to store various types of data to support the operation of the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0230] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0231] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0232] The audio component 610 is used to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0233] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0234] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of the components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of contact between the user and the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0235] The communication component 616 is used to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0236] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement a log collection method provided in an embodiment of the present application.

[0237] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of an electronic device 600 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0238] Fig.10700 is a logic block diagram of an electronic device 700 according to an embodiment of the present application. For example, the electronic device 700 may be provided as a server. Fig.10 , the electronic device 700 includes a processing component 722, which further includes one or more processors, and a memory resource represented by a memory 732, for storing instructions that can be executed by the processing component 722, such as an application. The application stored in the memory 732 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 722 is configured to execute instructions to perform a log collection method provided in an embodiment of the present application.

[0239] The electronic device 700 may also include a power supply component 726 configured to perform power management of the electronic device 700, a wired or wireless network interface 750 configured to connect the electronic device 700 to a network, and an input / output (I / O) interface 758. The electronic device 700 may operate based on an operating system stored in the memory 732, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0240] An embodiment of the present application also provides a computer program product, including a computer program, which implements the log collection method when executed by a processor.

[0241] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0242] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A log collection method, characterized in that: Applied to a container orchestration platform, the method includes: Obtaining a job application object corresponding to a target job sent by a job server and a parameter object corresponding to the job application object; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space; Creating a job execution container group corresponding to the job application object based on the job application object, and setting a log configuration file path of the job execution container group based on the first local directory; Based on the server address and the server path, the first local directory of the job execution container group is mounted to the network storage space, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file.

2. The method according to claim 1, characterized in that: After creating a job execution container group corresponding to the job application object based on the job application object, the method includes: Generate a log path corresponding to the job execution container group based on the first local directory; A log record file corresponding to the job execution container group is generated in the log path; wherein the log record file is used to record log information written by the job execution container group in the network storage space.

3. The method according to claim 2, characterized in that The generating a log path corresponding to the job execution container group based on the first local directory includes: Obtaining a job application identifier corresponding to the job application object; The job application identifier and the first local directory are concatenated to obtain the log path.

4. The method according to claim 3, characterized in that The concatenating the job application identifier and the first local directory to obtain the log path includes: Obtaining a container group identifier corresponding to the job execution container group; A log subdirectory is determined according to the job application identifier, a log file name is determined according to the container group identifier, and the first local directory, the log subdirectory and the log file name are concatenated to obtain the log path.

5. The method according to claim 2, characterized in that: Generating a log record file corresponding to the job execution container group in the log path includes: An entry point script of the container image corresponding to the job execution container group is executed, the entry point script generates the log record file based on the log configuration file and the log path, and stores the log configuration file based on the log path.

6. A log collection method, characterized in that: Applied to a job server, the method comprises: In response to a target job, a job application object corresponding to the target job is generated, and a parameter object corresponding to the job application object is configured; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space; The job application object and the parameter object are sent to a container orchestration platform, so that the container orchestration platform creates a job execution container group corresponding to the job application object based on the job application object, sets a log configuration file path of the job execution container group based on the first local directory, and mounts the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file.

7. A log collection device, characterized in that: Applied to the container orchestration platform, the log collection device includes: An acquisition module, used to acquire a job application object corresponding to a target job sent by a job server and a parameter object corresponding to the job application object; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space; A first path module, configured to create a job execution container group corresponding to the job application object based on the job application object, and to set a log configuration file path of the job execution container group based on the first local directory; A mounting module is used to mount the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file.

8. A log collection device, characterized in that: Applied to a job server, the log collection device comprises: A generation module, configured to generate a job application object corresponding to the target job in response to the target job, and configure a parameter object corresponding to the job application object; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space; A sending module is used to send the job application object and the parameter object to a container orchestration platform, so that the container orchestration platform creates a job execution container group corresponding to the job application object based on the job application object, sets a log configuration file path of the job execution container group based on the first local directory, and mounts the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file.

9. A log collection system, characterized in that: The system includes a job server and a container orchestration platform; The job server is used to generate a job application object corresponding to the target job in response to the target job, configure a parameter object corresponding to the job application object, and send the job application object and the parameter object to the container orchestration platform; wherein the parameter object includes a first local directory, a server address of a network storage space storing a log configuration file, and a server path of the network storage space; The container orchestration platform is used to obtain a job application object corresponding to a target job sent by a job server and a parameter object corresponding to the job application object, create a job execution container group corresponding to the job application object based on the job application object, set a log configuration file path of the job execution container group based on the first local directory, and mount the first local directory of the job execution container group to the network storage space based on the server address and the server path, so that the job execution container group reads the log configuration file from the network storage space based on the log configuration file path, and records log information based on the log configuration file; wherein the parameter object includes the first local directory, the server address of the network storage space storing the log configuration file, and the server path of the network storage space.

10. An electronic device, characterized in that: include: The electronic device comprises a processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the log collection method according to any one of claims 1 to 6.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the log collection method as described in any one of claims 1 to 6.

Images