Data processing methods, apparatus, equipment, storage media and products

Abstract

This application discloses a data processing method, apparatus, device, storage medium, and product. The method includes: loading a configuration file after an agent process deployed on a network device starts; generating a shared class acquisition marker file containing acquisition parameters, which enables a task process to read the acquisition parameters and acquire class data from the configuration file based on the acquisition parameters; reading the class data acquired by the task; generating a shared class list file based on the read class data; and generating a shared class archive marker file containing usage parameters and class data based on the shared class list file, which enables the task process to read the usage parameters and load the class data contained in the shared class archive marker file based on the usage parameters. The technical solution of this application significantly optimizes the deployment of class data sharing and application class data sharing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] In related technologies, the normal deployment process for class data sharing and application class data sharing technologies consists of three steps: first, determining which types of data need to be archived; second, creating shared class archive files; and third, using the shared class archive files. For each task, each deployment step requires relevant operations and maintenance personnel to add corresponding parameters, resulting in low deployment efficiency and a high risk of errors, making it difficult to apply class data sharing and application class data sharing technologies to large-scale production environments.

[0003] It is evident that improving the deployment efficiency and accuracy of class data sharing and application-level data sharing is an urgent problem to be solved. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this application provide data processing methods, apparatuses, electronic devices, computer-readable storage media, and computer program products, which can at least to some extent improve the deployment efficiency and accuracy of class data sharing and application class data sharing.

[0005] According to one aspect of the embodiments of this application, a data processing method is provided, the method comprising: after an agent process deployed on a network device starts, loading a configuration file, the configuration file storing class data required by a task to be executed; generating a shared class collection marker file containing collection parameters; wherein the shared class collection marker file is used to enable a task process of the task to be executed to read the collection parameters and collect class data from the configuration file according to the collection parameters; reading the class data collected by the task to be executed; generating a shared class list file according to the read class data, and generating a shared class archive marker file containing usage parameters and the class data according to the shared class list file; wherein the shared class archive marker file is used to enable the task process to read the usage parameters and load the class data contained in the shared class archive marker file according to the usage parameters.

[0006] According to one aspect of the embodiments of this application, a data processing apparatus is provided, the apparatus comprising: a loading module configured to load a configuration file after an agent process deployed on a network device starts, the configuration file storing class data required by a task to be executed; a first generation module configured to generate a shared class collection marker file containing collection parameters; wherein the shared class collection marker file is used to enable the task process of the task to be executed to read the collection parameters contained in the shared class collection marker file and collect class data from the configuration file according to the collection parameters, the configuration file storing class data required by the task to be executed; a reading module configured to enable the agent process to read the class data collected by the task to be executed; and a second generation module configured to generate a shared class list file based on the read class data, and generate a shared class archive marker file containing usage parameters and the class data based on the shared class list file; wherein the shared class archive marker file is used to enable the task process of the task to be executed to read the usage parameters and class data contained in the shared class archive marker file and load the class data contained in the shared class archive marker file according to the usage parameters.

[0007] In some embodiments of this application, based on the foregoing scheme, the apparatus further includes: a determining module, configured to, after the agent process starts, read the class data related file corresponding to the task to be executed to determine whether the class data related file corresponding to the task to be executed has changed; a checking module, configured to, if the class data related file corresponding to the task to be executed has not changed, check whether there is a shared class archive marker file and a newly generated shared class list file; and an execution module, configured to, if it is found that there is no shared class archive marker file and no newly generated shared class list file, execute the step of loading the configuration file.

[0008] In some embodiments of this application, based on the foregoing scheme, the apparatus further includes: a deletion module configured to delete the soft link of the shared class archive marker file, the shared class list file, and the shared class collection marker file under a specified path if the class data related files corresponding to the task to be executed change; and the execution module configured to execute the step of loading the configuration file.

[0009] In some embodiments of this application, based on the foregoing scheme, the determining module includes: a first reading unit configured to read the class path contained in the configuration file; a first determining unit configured to determine that the class data related file corresponding to the task to be executed has changed if the class path contained in the configuration file has changed; the first determining unit is further configured to determine that the class data related file corresponding to the task to be executed has not changed if the class path contained in the configuration file has not changed.

[0010] In some embodiments of this application, based on the foregoing scheme, the determining module further includes: a second reading unit configured to read the classpath contained in the error log file; a second determining unit configured to determine that the class data related file corresponding to the task to be executed has changed if the number of error log files recording changes in classpath is greater than a preset number threshold; the second determining unit is further configured to determine that the class data related file corresponding to the task to be executed has not changed if the number of error log files recording changes in classpath is less than or equal to the preset number threshold.

[0011] In some embodiments of this application, based on the foregoing scheme, the apparatus further includes: a third generation module configured to generate a latest shared class archive tag file based on the existing shared class list file and the newly generated shared class list file if a newly generated shared class list file is detected; and the deletion module is further configured to delete the soft link of the shared class archive tag file under the specified path.

[0012] In some embodiments of this application, based on the foregoing scheme, the apparatus further includes: a fourth generation module, configured to generate an incremental shared class acquisition marker file containing incremental acquisition parameters if the shared class archive marker file is detected; wherein the incremental shared class acquisition marker file is used to enable the task process to read the incremental acquisition parameters and acquire newly added class data from the configuration file according to the incremental acquisition parameters.

[0013] In some embodiments of this application, based on the foregoing scheme, the apparatus further includes: a fifth generation module, configured to generate an agent process survival marker file when the agent process is first started; wherein the agent process survival marker file contains timestamp information when the agent process is first started, and the agent process survival marker file is used to enable the task process to read the timestamp information; and the execution module is further configured to perform the step of reading the collection parameters when it is determined from the timestamp information that the survival time of the agent process is less than a preset duration threshold.

[0014] In some embodiments of this application, based on the foregoing scheme, the apparatus further includes: the running module, which is further configured to run the task process of the task to be executed if the task process successfully loads the class data contained in the shared class archive marker file into memory according to the usage parameters; and an output module, which is configured to output failure information if the task process fails to successfully load the class data contained in the shared class archive marker file into memory according to the usage parameters.

[0015] In some embodiments of this application, based on the foregoing scheme, the network device includes a network device in standalone mode; wherein the agent process and the task process are deployed on the network device in standalone mode, and the task process includes a short task process.

[0016] In some embodiments of this application, based on the foregoing scheme, the network device includes a central node network device in a cluster mode and various sub-node network devices under the central node network device; wherein, the proxy process is deployed on the central node network device and the various sub-node network devices respectively, the task process is deployed on the various sub-node network devices, and the task process includes a long-task task process.

[0017] In some embodiments of this application, based on the foregoing scheme, the agent process deployed on the central node network device is used to issue class data collection instructions to the agent processes deployed on each child node network device. The agent processes deployed on each child node network device are used to generate a shared class collection tag file according to the class data collection instructions to obtain a shared class list file, and report the obtained shared class list file to the agent process deployed on the central node network device. The agent process deployed on the central node network device is also used to generate a shared class archive tag file according to the received shared class list file, and issue the generated shared class archive tag file to the agent processes deployed on each child node network device.

[0018] According to one aspect of the embodiments of this application, an electronic device is provided, including one or more processors; and a storage device for storing one or more programs, which, when executed by the electronic device, cause the electronic device to implement the data processing method as described above.

[0019] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the data processing method described above.

[0020] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions that, when executed by a processor, implement the data processing method as described above.

[0021] In the technical solutions provided by the embodiments of this application, on the one hand, after the agent process deployed on the network device starts, it loads the configuration file and generates a shared class collection tag file containing collection parameters. In this way, the task process of the task to be executed can read the collection parameters contained in the shared class collection tag file and collect class data from the loaded configuration file according to the collection parameters, thereby realizing automated collection of class data. The process of collecting class data is the process of determining which class data needs to be archived. On the other hand, the agent process deployed on the network device can read the class data collected by the task to be executed and generate a shared class list file according to the read class data. Then, a shared class archive tag file containing usage parameters and class data is generated according to the shared class list file. In this way, the task process of the task to be executed can read the usage parameters contained in the shared class archive tag file and load the class data contained in the shared class archive tag file according to the usage parameters, thereby realizing automated creation of shared class archive tag files and automated use of the class data contained in the shared class archive tag files. The process of generating the shared class archive tag file is the process of creating a shared class archive file, and loading the class data contained in the shared class archive tag file is the process of using the class data contained in the shared class archive tag file.

[0022] In other words, the technical solution provided in the embodiments of this application generates a shared class collection tag file containing collection parameters and a shared class archive tag file containing usage parameters and class data by an agent process deployed on a network device. In this way, before the task process actually runs, it can automatically complete the collection or loading of class data by reading the corresponding tag file, realizing the automated deployment of class data sharing and application class data sharing. This avoids the problems of low efficiency and easy errors in deployment by operation and maintenance personnel adding corresponding parameters, greatly improving deployment efficiency and accuracy. Furthermore, due to the high deployment efficiency, class data sharing and application class data sharing technologies can be applied to large-scale production environments, making it easy to promote and use class data sharing and application class data sharing technologies, while bringing more development space to large-scale production environments.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0025] Figure 1 This is a schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied;

[0026] Figure 2 This is a schematic diagram illustrating a standalone mode in an exemplary embodiment of this application;

[0027] Figure 3 This is a schematic diagram illustrating a cluster mode in an exemplary embodiment of this application;

[0028] Figure 4 This is a flowchart illustrating a data processing method in an exemplary embodiment of this application;

[0029] Figure 5 This is a flowchart illustrating a data processing method in an exemplary embodiment of this application;

[0030] Figure 6 yes Figure 5 A flowchart of step S501 in an exemplary embodiment shown in the illustrated example;

[0031] Figure 7 yes Figure 5 A flowchart of step S501 in an exemplary embodiment shown in the illustrated example;

[0032] Figure 8 This is a flowchart illustrating a data processing method in an exemplary embodiment of this application;

[0033] Figure 9 This is a schematic diagram illustrating a data processing method in an exemplary embodiment of this application;

[0034] Figure 10 This is a flowchart illustrating a data processing method in an exemplary embodiment of this application;

[0035] Figure 11 This is a flowchart illustrating a data processing method in an exemplary embodiment of this application;

[0036] Figure 12 This is a block diagram illustrating a data processing apparatus according to an exemplary embodiment of this application;

[0037] Figure 13This is a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application embodiments. Detailed Implementation

[0038] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments identical to those described in this application. Rather, they are merely examples of apparatuses and methods identical to some aspects of this application as detailed in the appended claims.

[0039] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0040] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0041] It should be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0042] Before introducing the technical solutions of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0043] The Java Virtual Machine (JVM) is an abstraction of a computer, implemented by simulating various computer functions on a real computer. The JVM has its own complete hardware architecture, such as a processor, stack, and registers, and also has a corresponding instruction set. The JVM can be simply understood as a translator; it shields the program from information specific to the operating system platform, allowing Java programs to run on multiple platforms without modification by generating target code (bytecode) that runs on the JVM. Therefore, the Java language is cross-platform compatible. In essence, one Java process corresponds to one JVM, and Java programs are executed by the JVM.

[0044] Class Data Sharing (CDS) refers to the preprocessing of class data and placing it into a shared class archive. Specifically, the installer loads a series of class files (i.e., class data) from the system JAR file into a private internal representation and stores this internal representation in a file called the shared archive. When a Java program starts, the Java Virtual Machine loads / maps the shared class archive into memory, optimizing the loading speed of class data, saving the cost of loading class data, and thus reducing the startup time of the Java program.

[0045] Application Class-Data Sharing (AppCDS) is similar to Class Data Sharing, but the difference lies in that it extends CDS technology to the class data used by the application. AppCDS is particularly effective for smaller applications because it eliminates the fixed cost of loading certain core classes. The smaller the application is relative to the number of core classes loaded, the greater the time savings in startup.

[0046] In related technologies, the normal deployment process for class data sharing and application class data sharing technologies consists of three steps: first, determining which types of data need to be archived; second, creating shared class archive files; and third, using the shared class archive files. For each task, each deployment step requires operations personnel to add corresponding parameters, resulting in low deployment efficiency and making it difficult to apply class data sharing and application class data sharing technologies to large-scale production environments. Furthermore, each step requires a restart when creating a shared class archive file for the first time, further reducing deployment efficiency. Additionally, the workload for operations personnel is considerable, requiring them to have a thorough understanding of deployment procedures, thus placing high demands on their skills.

[0047] Based on this, embodiments of this application propose a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which improves the deployment efficiency of class data sharing and application class data sharing, enabling its application in large-scale production environments, reducing the pressure on maintenance personnel, saving manpower and resources, and avoiding the error-prone phenomenon of manual deployment, thereby improving the deployment accuracy of class data sharing and application class data sharing.

[0048] Please see Figure 1 , Figure 1 This is a schematic diagram of an exemplary system architecture of this application; the system architecture includes a terminal device 110 and a server 120, and the terminal device 110 and the server 120 communicate with each other through a wired or wireless network.

[0049] The terminal device 110 can be an electronic device such as a smartphone, tablet, laptop, computer, or vehicle terminal, and the present invention does not limit this.

[0050] In one embodiment of this application, server 120 can be a big data processing center built on Hadoop. It is understood that Hadoop is a distributed infrastructure framework comprising three main parts: HDFS (Hadoop Distributed File System), YARN (Yet Another Resource Negotiator), and MapReduce. HDFS handles storage, YARN handles resource scheduling, and MapReduce handles computation. Within the compute nodes, each physical machine can launch a large number of task processes to complete computational tasks. Optionally, each physical machine can be a standalone network device or a clustered network device.

[0051] Please see Figure 2 This is an example of a network device in standalone mode. Figure 2 This example only illustrates one network device. In other embodiments of this application, any number of network devices can be configured according to actual needs. Each network device (i.e., a single machine) deploys an agent process, which simultaneously handles three types of tasks: MapReduce (MR) tasks (tasks implemented using MR technology, referred to as MR tasks; MR technology is used for parallel computation of large-scale datasets, including Map and Reduce functions. The Map function processes a key-value pair-based dataset, outputting intermediate key-value pair-based datasets, while the Reduce function merges all intermediate values ​​with the same intermediate key), and Spark tasks (tasks implemented using Spark, referred to as Spark tasks; Spark is a technology developed by the AMP Lab (Algorithms, Machines, and...) at UC Berkeley). Developed by PeopleLab, Spark is a general-purpose in-memory parallel computing framework used to build large-scale, low-latency data analysis applications. It's an open-source cluster computing framework, a fast and general-purpose computing engine designed for large-scale data processing. It can be used to perform various operations, including querying, text processing, and machine learning. Developers need to receive, process, and output data according to Spark's specific programming methods, then manually generate a JAR file and submit it to the corresponding Spark cluster so that Spark will allocate cluster resources to run the task. There are also SQL (Structured Query Language) type tasks (tasks implemented using SQL statements; SQL is a database query and programming language used to access, query, update, and manage relational database systems). Therefore, there are three types of task working directories: MapReduce (MR) task working directory, Spark task working directory, and SQL task working directory. Each type of task working directory is further subdivided into multiple tasks; for example, the MR task working directory is subdivided into multiple MR tasks, the Spark task working directory into multiple Spark tasks, and the SQL task working directory into multiple SQL tasks. It is understandable that for tasks of the same type, their corresponding classpaths have some consistency. Therefore, AppCDS can function in this standalone mode.

[0052] In standalone mode, tasks on network devices are typically short tasks (but can also be long tasks). Correspondingly, the task processes deployed on the network device are the task processes for short tasks. The agent processes deployed on the network device can adopt a timed startup mode. First, a shared class collection tag file containing collection parameters is generated. In this way, the task process of the task to be executed can read the collection parameters contained in the shared class collection tag file and collect class data from the configuration file according to the collection parameters. Then, the agent process generates a shared class list file by reading the class data collected by the task to be executed, and generates a shared class archive tag file containing usage parameters and class data according to the shared class list file. In this way, the task process of the task to be executed can read the usage parameters contained in the shared class archive tag file and load the class data contained in the shared class archive tag file according to the usage parameters.

[0053] Please see Figure 3 This is an example of a network device in a cluster mode. The central node network device (i.e., the central node) and each of its sub-node network devices (i.e., individual machines) are deployed with agent processes. The agent processes deployed on the central node network device manage the agent processes deployed on each sub-node network device. Each sub-node network device has one type of task to execute, such as Agent1 having type 1 tasks to execute, Agent2 having type 2 tasks to execute, and Agent3 having type 3 tasks to execute. These correspond to task working directories: Agent1 corresponds to task working directory 1, Agent2 to task working directory 2, and Agent3 to task working directory 3. Each type of task working directory is further subdivided into multiple tasks; for example, task working directory 1 is subdivided into tasks 11-13, task working directory 2 into tasks 21-23, and task working directory 3 into tasks 31-33. It is understandable that for tasks of the same type, their corresponding classpaths have some consistency. Therefore, AppCDS can function in this cluster mode.

[0054] In cluster mode, tasks on network devices are typically long tasks (but can also be short tasks). Correspondingly, the task processes deployed on the network devices are the task processes corresponding to the long tasks. The agent processes deployed on the network devices can adopt an intermittent startup mode. First, the agent process deployed on the central node network device sends class data collection instructions to the agent processes deployed on each child node network device. Each child node agent process generates a shared class data collection marker file containing collection parameters based on the class data collection instructions. In this way, the task processes on each child node network device can read the collection parameters contained in the shared class data collection marker file and collect class data from the configuration file based on these parameters. Then, each child node network... The agent process deployed on the device generates a shared class list file by reading the class data collected by the task to be executed, and reports the generated shared class list file to the agent process deployed on the central node network device. The agent process deployed on the central node network device generates a shared class archive marker file containing usage parameters and class data according to the shared class list file reported by the agent processes of each child node network device, and sends the generated shared class archive marker file to the corresponding agent processes deployed on each child node network device. In this way, the task process of the task to be executed on each child node network device can read the usage parameters contained in the shared class archive marker file and load the class data contained in the shared class archive marker file according to the usage parameters.

[0055] It should be noted that, regardless of whether it is in standalone mode or cluster mode, the Agent process deployed on the network device generates corresponding shared class collection tag files, shared class list files, and shared class archive tag files containing collection parameters based on the same type of task.

[0056] Among them, Figure 2 In the single-machine mode shown, since there are three types of tasks to be executed on the single-machine Agent, the single-machine Agent will generate a shared class collection marker file, a shared class list file, and a shared class archive marker file for each type of task. That is, for MR type tasks, shared class collection marker files, shared class list files, and shared class archive marker files will be generated, and the task process of the MR task will read the corresponding shared class collection marker files and shared class archive marker files. Similarly, for Spark type tasks, shared class collection marker files and shared class archive marker files will be generated, and the task process of the Spark task will read the corresponding shared class collection marker files and shared class archive marker files.

[0057] Among them, Figure 3 In the cluster mode shown, since there are three single-machine agents under the central node agent, and each single-machine agent has one type of task to be executed, the central node agent generates a shared class collection marker file, a shared class list file, and a shared class archive marker file for each type of task. That is, for type 1 tasks, a shared class collection marker file, a shared class list file, and a shared class archive marker file are generated, and the task process of type 1 tasks reads the corresponding shared class collection marker file and shared class archive marker file; correspondingly, for type 2 tasks, a shared class collection marker file and a shared class archive marker file are generated, and the task process of type 3 tasks reads the corresponding shared class collection marker file and shared class archive marker file.

[0058] In one embodiment of this application, server 120 may be a serverless center in the cloud. It is understood that serverless computing in the cloud is a method of providing backend services on demand; serverless providers allow users to write and deploy code without worrying about the underlying infrastructure. Companies obtaining backend services from serverless providers will pay for their computing services without having to retain and pay for a fixed amount of bandwidth or servers, as the service is automatically scalable. Although called serverless, physical servers are still used, but the relevant developers do not need to understand them. It should be noted that the serverless center in the cloud can also adopt the single-machine mode or cluster mode described above. An agent process is deployed on the network device in single-machine or cluster mode. The agent process generates a shared class collection tag file, a shared class list file, and a shared class archive tag file. The task process to be executed then reads the collection parameters contained in the shared class collection tag file to implement the collection of class data, and reads the usage parameters contained in the shared class archive tag file to implement the loading of class data. For details, please refer to the foregoing description; it will not be repeated here.

[0059] In one embodiment of this application, server 120 may be an e-commerce service center. It is understood that the e-commerce service center may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. It should be noted that the e-commerce service center may also adopt the single-machine mode or cluster mode described above. An agent process is deployed on the network device in the single-machine mode or cluster mode. The agent process generates a shared class collection marker file, a shared class list file, and a shared class archive marker file. The task process to be executed then reads the collection parameters contained in the shared class collection marker file to implement the collection of class data, and reads the usage parameters contained in the shared class archive marker file to implement the loading of class data. For details, please refer to the foregoing description; further details will not be repeated here.

[0060] based on Figure 1 The application scenario shown illustrates that, after adopting the technical solution of this application embodiment, a shared class data collection marker file containing collection parameters is generated by an agent process. The task process of the task to be executed reads the collection parameters contained in the shared class data collection marker file to realize the collection of class data. Similarly, the agent process generates a shared class archive marker file containing usage parameters and class data, and the task process of the task to be executed reads the usage parameters contained in the shared class archive marker file to realize the loading of class data. This achieves automatic deployment of class data sharing and application class data sharing, eliminating the need for manual deployment by operations and maintenance personnel. On one hand, it improves the deployment efficiency of class data sharing and application class data sharing, making it applicable to large-scale production environments, such as big data processing centers, cloud serverless centers, and e-commerce service centers, with simple implementation logic. On the other hand, it reduces the pressure on operations and maintenance personnel, saves manpower and resources, and avoids the errors that are prone to occur during manual deployment, thus improving the deployment accuracy of class data sharing and application class data sharing.

[0061] The following details the various implementation details of the technical solutions in the embodiments of this application:

[0062] Please see Figure 4 , Figure 4 This is a flowchart illustrating a data processing method according to an embodiment of this application.

[0063] like Figure 4 As shown, the data processing method includes at least steps S401 to S404, which are described in detail below:

[0064] Step S401: After the agent process deployed on the network device starts, it loads the configuration file, which stores the class data required by the task to be executed.

[0065] In this embodiment of the application, the proxy process corresponds to the proxy service. The proxy service is a service developed by the software development engineer to generate a shared class collection tag file containing collection parameters, and to generate a shared class list file based on the collected class data, and then generate a shared class archive tag file containing usage parameters and class data based on the shared class list file. After development is completed, it is pre-installed in the network device. It can be understood that the proxy service is called the proxy process when it runs.

[0066] In an optional embodiment, the proxy service can be developed separately, so it only needs to be deployed in the network device without adjusting or modifying other services in the network device. This results in low coupling and facilitates later maintenance, making it highly maintainable.

[0067] In this embodiment, the task to be executed refers to a task of the same type to be executed, such as an MapReduce task, a Spark task, an SQL task, etc. When the task to be executed is executed, a corresponding task process is run to achieve the execution of the task. The task process of the task to be executed includes a Java process and the Java Virtual Machine corresponding to the Java process.

[0068] In an optional embodiment, the automated deployment process for class data sharing and application class data sharing can be executed by a Java Virtual Machine (JVM). Specifically, the task process of the task to be executed (hereinafter referred to as the task process) reads the collection parameters and collects class data from the configuration file according to the collection parameters. This can be achieved by the JVM reading the collection parameters and collecting class data from the configuration file accordingly. Furthermore, the task process of the task to be executed (hereinafter referred to as the task process) reads the usage parameters and loads the class data contained in the shared class archive tag file according to the usage parameters. This can be achieved by the JVM reading the usage parameters and loading the class data contained in the shared class archive tag file accordingly. Additionally, the task process of the task to be executed (hereinafter referred to as the task process) reads the incremental collection parameters and collects newly added class data from the configuration file according to the incremental collection parameters. This can be achieved by the JVM reading the incremental collection parameters and collecting newly added class data from the configuration file accordingly. Finally, the task process of the task to be executed (hereinafter referred to as the task process) reads timestamp information and, when it is determined based on the timestamp information that the lifetime of the proxy process is less than a preset lifetime threshold, reads the collection parameters. This can be achieved by the JVM reading the timestamp information and, when it is determined based on the timestamp information that the lifetime of the proxy process is less than a preset lifetime threshold, reads the collection parameters.

[0069] In this embodiment of the application, the configuration file refers to a file that records the working directory (i.e., the working directory) corresponding to the task to be executed. The working directory of the task to be executed contains a classpath, and the class path contains corresponding class data. It can be understood that the class data corresponding to the classpath is the class data required by the task to be executed when it is executed.

[0070] It is understood that after the agent process deployed on the network device in this embodiment starts, it first loads the configuration file. This allows the task process to collect class data from the loaded configuration file based on the collection parameters contained in the shared class collection marker file. It should be noted that the agent process startup mentioned here refers to startup in a system environment where there is no shared class collection marker file, no shared class list file, and no shared class archive marker file (i.e., the first startup of the agent process), or startup in a system environment where there is no newly generated shared class list file and no shared class archive marker file (i.e., a subsequent "first" startup of the agent process). In this case, loading the configuration file is necessary; in other system environments, loading the configuration file is not required.

[0071] In one embodiment of this application, please refer to Figure 5 Before loading the configuration file in step S401, the data processing method may also include steps S501 to S503, which are described in detail below:

[0072] Step S501: After the agent process starts, it reads the class data related files corresponding to the task to be executed to determine whether the class data related files corresponding to the task to be executed have changed.

[0073] Step S502: If the class data related files corresponding to the task to be executed have not changed, check whether there is a shared class archive marker file and a newly generated shared class list file;

[0074] Step S503: If it is found that there is no shared class archive tag file and no newly generated shared class list file, then the step of loading configuration file is executed.

[0075] In other words, in the optional embodiment, before generating the shared class acquisition marker file containing acquisition parameters, the agent process first reads the class data related file corresponding to the task to be executed after startup to determine whether the class data related file corresponding to the task to be executed has changed. If the agent process determines that the class data related file corresponding to the task to be executed has not changed after reading it, it indicates that there is no need to load the configuration file at this time. If the agent process determines that the class data related file corresponding to the task to be executed has changed after reading it, it indicates that there is a need to load the configuration file at this time. Furthermore, the agent process needs to check whether the shared class archive marker file and the newly generated shared class list file exist in the system environment. If the agent process finds that neither the shared class archive marker file nor the newly generated shared class list file exists in the system environment, it indicates that there is a need to load the configuration file at this time.

[0076] In this way, the agent process reads the class data related files corresponding to the task to be executed, confirms that the class data related files corresponding to the task to be executed have not changed, and further checks that there is no shared class archive marker file and no newly generated shared class list file before executing the step of loading the configuration file. This ensures that the configuration file is loaded at the appropriate time, avoiding unnecessary resource overhead caused by loading the configuration file at all times, and saving system resources.

[0077] In an optional embodiment, if the agent process determines that the class data related files corresponding to the task to be executed have changed after reading the class data related files corresponding to the task to be executed, it is necessary to delete the soft link of the shared class archive marker file, the shared class list file, and the shared class collection marker file under the specified path, and then execute the step of loading the configuration file.

[0078] In one optional embodiment, a soft link, also known as a symbolic link, is similar to a shortcut in Windows. When a soft link is created, a new file is created, and the operating system allocates an inode to it. The purpose of this file is to provide a shortcut to another file located somewhere in the system. The counterpart to a soft link is a hard link. Specifically, a soft link to a shared archive marker file refers to a soft link to a shared archive marker file, which is a special file (i.e., a text file) containing the location information of another file (which can be a shared archive marker file).

[0079] In one optional embodiment, when the agent process determines that the class data-related files corresponding to the task to be executed have changed, the soft links to the shared class archive marker files, shared class list files, and shared class acquisition marker files in the specified path that are deleted accordingly refer to the existing soft links to the shared class archive marker files, existing shared class list files, and existing shared class acquisition marker files in the specified path. It is understood that "existing" refers to what currently exists in the system environment, but was generated in the past, which is the same as the "existing" mentioned below.

[0080] In this way, when the agent process determines that the class data-related files corresponding to the task to be executed have changed, deleting the symbolic link of the shared class archive marker file in the specified path will not affect the shared class archive marker file, that is, it will not affect the tasks using the class data contained in the shared class archive marker file, ensuring the normal execution of the task. However, if the shared class archive marker file is deleted directly, it will affect the tasks using the class data contained in the shared class archive marker file, which may cause the task execution to fail. At the same time, when it is determined that the class data-related files corresponding to the task to be executed have changed, it is also necessary to delete the shared class collection marker file and the shared class list file. This ensures that after the class data-related files corresponding to the task to be executed change, the configuration file is reloaded to generate the latest shared class collection marker file and the latest shared class list file.

[0081] In an optional embodiment, if the agent process detects the existence of a newly generated shared class list file, it generates the latest shared class archive tag file based on the existing shared class list file and the newly generated shared class list file, and deletes the symbolic link of the shared class archive tag file under the specified path.

[0082] In one of the optional embodiments, when the agent process detects that both an existing shared class list file and a newly generated shared class list file exist in the system environment, the agent process needs to generate the latest shared class archive marker file based on the existing shared class list file and the newly generated shared class list file. That is, the existing shared class list file is updated based on the newly generated shared class list file to generate the latest shared class archive marker file.

[0083] In one optional embodiment, when the agent process detects the existence of a newly generated shared class list file, the soft link to the shared class archive marker file in the specified path that is deleted refers to the soft link to the existing shared class archive marker file in the specified path.

[0084] In this way, when the agent process detects a newly generated shared class list file, it only needs to update the existing shared class list file based on the newly generated file to generate the latest shared class archive tag file, without having to load the configuration file to generate the latest shared class list file again, thus improving the speed of generating the latest shared class archive tag file. Simultaneously, when the agent process detects a newly generated shared class list file, it also needs to delete the symbolic link to the shared class archive tag file in the specified path. This does not affect the shared class archive tag file itself, meaning it does not affect tasks that use the class data contained in the shared class archive tag file, ensuring the normal execution of tasks.

[0085] In an optional embodiment, if the agent process detects the existence of a shared class archive marker file, it generates an incremental shared class acquisition marker file containing incremental acquisition parameters; wherein, the incremental shared class acquisition marker file is used to enable the task process to read the incremental acquisition parameters and acquire newly added class data from the configuration file according to the incremental acquisition parameters.

[0086] In one of the optional embodiments, when the agent process detects that a shared class archive marker file exists in the system environment, the agent process can generate an incremental shared class acquisition marker file containing incremental acquisition parameters. The incremental shared class acquisition marker file contains incremental acquisition parameters and is used to enable the task process to read the incremental acquisition parameters and acquire newly added class data from the configuration file based on the read incremental acquisition parameters to obtain the newly added class data.

[0087] In one optional embodiment, the incremental shared collection marker file containing incremental collection parameters can be generated by the agent process receiving an instruction carrying incremental collection parameters. The instruction carrying incremental collection parameters is issued by the operation and maintenance personnel according to the specific application scenario requirements.

[0088] In this way, when the agent process detects a newly generated shared class list file, it can generate an incremental shared class acquisition marker file containing incremental acquisition parameters. The task process to be executed can then read the incremental acquisition parameters contained in the incremental shared class acquisition marker file and acquire the newly added class data from the configuration file (the updated configuration file) based on the incremental acquisition parameters. The agent process can then read the newly added class data acquired by the task to be executed and generate a new shared class list file based on the read new class data (i.e., obtain the newly generated shared class list file). The existing shared class list file is then updated based on the newly generated shared class list file to generate the latest shared class archive marker file. The agent process can also update the identifiers of the latest shared class archive marker file, such as the identifier number, without having to load the configuration file to generate the latest shared class list file again, thus improving the speed of generating the latest shared class archive marker file.

[0089] In one embodiment of this application, please refer to Figure 6 The class data related files include configuration files; the process of reading the class data related files corresponding to the task to be executed in step S501 to determine whether the class data related files corresponding to the task to be executed have changed may include steps S601 to S603, which are described in detail below:

[0090] Step S601: Read the classpath contained in the configuration file;

[0091] Step S602: If the classpath contained in the configuration file changes, it is determined that the class data related files corresponding to the task to be executed have changed.

[0092] Step S603: If the classpath contained in the configuration file has not changed, then it is determined that the class data related files corresponding to the task to be executed have not changed.

[0093] In an optional embodiment, since the configuration file contains a classpath, the configuration file is a type of class data related file. Therefore, the proxy process can determine whether the class data related file corresponding to the task to be executed has changed by reading the classpath contained in the configuration file. Specifically, it can read the classpath contained in the configuration file. If the proxy process finds that the classpath contained in the configuration file has changed, it can be determined that the class data related file corresponding to the task to be executed has changed. If the proxy process finds that the classpath contained in the configuration file has not changed, it can be determined that the class data related file corresponding to the task to be executed has not changed.

[0094] For example, suppose the original classpath in the configuration file is A. If the proxy process reads that the classpath in the configuration file has changed from A to B, then it can be determined that the class data related files corresponding to the task to be executed have changed. If the proxy process reads that the classpath in the configuration file is A, then it can be determined that the class data related files corresponding to the task to be executed have not changed.

[0095] In this way, since the configuration file is an easily accessible class data file, the proxy process can easily and conveniently determine whether the class data file corresponding to the task to be executed has changed by reading the class path contained in the configuration file, thus improving the efficiency of the proxy process in determining whether the class data file corresponding to the task to be executed has changed.

[0096] In one embodiment of this application, please refer to Figure 7 The class data related files include an error log file, which records changes to the classpath. Step S501, which involves reading the class data related files corresponding to the task to be executed to determine whether these files have changed, may include steps S701 to S703, detailed below:

[0097] Step S701: Read the classpath contained in the error log file;

[0098] Step S702: If the number of error log files recording changes in classpath exceeds a preset threshold, it is determined that the class data related files corresponding to the task to be executed have changed.

[0099] Step S703: If the number of error log files recording changes in classpath is less than or equal to a preset threshold, it is determined that the class data related files corresponding to the task to be executed have not changed.

[0100] In an optional embodiment, since the error log file records changes to the classpath, it is a type of class data related file. Therefore, the agent process can determine whether the class data related file corresponding to the task to be executed has changed by reading the number of error log files that record changes to the classpath. Specifically, it can read the classpath contained in the error log file. If the number of error log files that record changes to the classpath read by the agent process is greater than a preset threshold, it can be determined that the class data related file corresponding to the task to be executed has changed. If the number of error log files that record changes to the classpath read by the agent process is less than or equal to the preset threshold, it can be determined that the class data related file corresponding to the task to be executed has not changed.

[0101] For example, assuming a preset threshold of 10, if the agent process reads more than 10 error log files showing changes to the class path, it can be determined that the class data related files for the task to be executed have changed. If the agent process reads less than or equal to 10 error log files showing changes to the class path, it can be determined that the class data related files for the task to be executed have not changed. In practical applications, the preset threshold can be flexibly adjusted according to the specific application scenario.

[0102] In this way, since the error log file is an easily accessible class data related file, the agent process can easily and conveniently determine whether the class data related file corresponding to the task to be executed has changed by reading the number of error log files that record changes in the classpath, thus improving the efficiency of the agent process in determining whether the class data related file corresponding to the task to be executed has changed.

[0103] It should be noted that the above describes two methods for determining whether the class data related files corresponding to the task to be executed have changed. These two methods can be executed in combination or separately.

[0104] In an optional embodiment, during combined execution, it can be determined that the class data related files corresponding to the task to be executed have changed when one of the following conditions is met: for example, when the class path contained in the configuration file changes, and the number of error log files recording the class path change is less than or equal to a preset number threshold, it can be determined that the class data related files corresponding to the task to be executed have changed; or when the number of error log files recording the class path change is greater than the preset number threshold, and the class path contained in the configuration file has not changed, it can be determined that the class data related files corresponding to the task to be executed have changed.

[0105] In an optional embodiment, during combined execution, it can also be determined that the class data related files corresponding to the task to be executed have changed when both conditions are met; for example, it can be determined that the class data related files corresponding to the task to be executed have changed only when the class path contained in the configuration file changes and the number of error log files recording the class path change is greater than a preset number threshold.

[0106] In an optional embodiment, when the agent process is started for the first time, an agent process survival marker file is generated; wherein, the agent process survival marker file contains timestamp information when the agent process is started for the first time, the agent process survival marker file is used to enable the task process to read the timestamp information, and when it is determined from the timestamp information that the survival time of the agent process is less than a preset duration threshold, the step of reading the collection parameters is executed.

[0107] The agent process survival flag file contains a timestamp information of the agent process's first startup. This file is used by task processes to read the timestamp information, determine the agent process's survival duration based on the timestamp, and compare the agent process's survival duration with a preset duration threshold. If the agent process's survival duration is less than the preset duration threshold, the task process can read the acquisition parameters; if the agent process's survival duration is greater than or equal to the preset duration threshold, the task process does not need to read the acquisition parameters. It is understood that the timestamp information of the agent process's first startup contained in the agent process survival flag file refers to the agent process's true first startup, not the aforementioned subsequent first startup.

[0108] For example, if the agent process survival flag file contains the timestamp of the agent process's first startup as 2021-09-01, and the current time is 2021-09-28, then the agent process's survival time can be determined to be 28 days. If the preset duration threshold is set to 30 days, then the agent process's survival time of 28 days is less than the preset duration threshold of 30 days, and the task process to be executed can read the collection parameters. Conversely, if the agent process survival flag file contains the timestamp of the agent process's first startup as 2021-09-01, and the current time is 2021-10-01, then the agent process's survival time can be determined to be 31 days. If the preset duration threshold is set to 30 days, then the agent process's survival time of 31 days is greater than the preset duration threshold of 31 days, and the task process to be executed does not need to read the collection parameters. In practical applications, the preset duration threshold can be flexibly adjusted according to the specific application scenario.

[0109] In this way, the lifespan of the agent process can be easily and conveniently determined by generating an agent process survival tag file containing the timestamp information of the agent process when it was first started. Furthermore, the task process to be executed will only read the collection parameters when the agent process is within the specified lifespan, which provides a fault tolerance mechanism. If the agent process is not within the specified lifespan, it is determined to be an older agent process, and its generated tag file will no longer be read.

[0110] Step S402: Generate a shared class acquisition marker file containing acquisition parameters; wherein, the shared class acquisition marker file is used to enable the task process of the task to be executed to read the acquisition parameters and acquire class data from the configuration file according to the acquisition parameters.

[0111] In this embodiment of the application, after the agent process deployed on the network device starts and loads the configuration file, it can generate a shared class collection marker file containing collection parameters. In this way, the task process to be executed can read the collection parameters contained in the shared class collection marker file and collect class data from the configuration file according to the read collection parameters to obtain class data.

[0112] In an optional embodiment, the shared class collection marker file containing collection parameters can be generated by the agent process receiving an instruction carrying collection parameters. The instruction carrying collection parameters is issued by the operation and maintenance personnel according to the specific application scenario requirements.

[0113] Step S403: Read the class data collected by the task to be executed.

[0114] In this embodiment of the application, after the agent process generates a shared class collection marker file containing collection parameters, it can read the class data collected by the task to be executed.

[0115] Step S404: Generate a shared class list file based on the read class data, and generate a shared class archive marker file containing usage parameters and class data based on the shared class list file; wherein, the shared class archive marker file is used to enable the task process to read the usage parameters and load the class data contained in the shared class archive marker file according to the usage parameters.

[0116] In this embodiment of the application, after the agent process reads the class data collected by the task to be executed, it can generate a shared class list file based on the read class data, and generate a shared class archive tag file containing usage parameters and class data based on the shared class list file. In this way, the task process of the task to be executed can read the usage parameters contained in the shared class archive tag file and load the class data contained in the shared class archive tag file based on the read usage parameters.

[0117] In an optional embodiment, a shared class archive marker file containing usage parameters and class data is generated based on the shared class list file. This can be achieved by the agent process receiving an instruction to generate a shared class archive marker file, which in turn generates the shared class archive marker file containing usage parameters and class data. The instruction to generate the shared class archive marker file is issued by the operation and maintenance personnel according to the specific application scenario requirements.

[0118] In an optional embodiment, if the task process successfully loads the class data contained in the shared class archive tag file into memory according to the usage parameters, then the task process running the task to be executed, i.e. the Java process included in the task process, runs the task to be executed. In this way, since the class data required by the task to be executed has been successfully loaded into memory, the task process of the task to be executed, such as the Java process, can directly use the class data loaded into memory when it runs, thus improving the running speed.

[0119] In an optional embodiment, if the task process fails to successfully load the class data contained in the shared class archive marker file into memory according to the usage parameters, a failure message is output. In this way, when the class data required by the task to be executed is not successfully loaded into memory, the corresponding failure message is output, which allows the relevant operation and maintenance personnel to determine that the class data required by the task to be executed has not been successfully loaded into memory. Furthermore, the failure message may contain the reason for the failure, which facilitates the relevant operation and maintenance personnel to make adjustments and modifications based on the reason for the failure.

[0120] In this embodiment, a shared class collection tag file containing collection parameters and a shared class archive tag file containing usage parameters and class data are generated by an agent process deployed on a network device. Before the task process actually runs, it can automatically complete the collection or loading of class data by reading the corresponding tag file, thus achieving automated deployment of class data sharing and application class data sharing. This avoids the low efficiency and error-prone nature of deployment by maintenance personnel adding corresponding parameters, greatly improving deployment efficiency and accuracy. Furthermore, due to the high deployment efficiency, class data sharing and application class data sharing technologies can be applied to large-scale production environments, facilitating their widespread adoption and providing more development opportunities for large-scale production environments.

[0121] The following provides a detailed description of a specific application scenario of this application:

[0122] In the embodiments of this application, using Figure 2 The example shown illustrates the deployment of the Agent process on a standalone network device, and uses the presence of MR type tasks (including multiple MR tasks) on the standalone network device as an example.

[0123] Please see Figure 8 The above is a flowchart of an exemplary data processing method, and please also refer to [the flowchart]. Figure 9 This is a schematic diagram of an exemplary data processing method; the data processing method may include the following steps, detailed below:

[0124] Step S801: When the agent process deployed on the network device starts for the first time, an agent process survival marker file is generated; wherein, the agent process survival marker file contains the timestamp information when the agent process starts for the first time. The agent process survival marker file is used to enable the task process to read the timestamp information and read the collection parameters when it is determined from the timestamp information that the survival time of the agent process is less than the preset duration threshold.

[0125] Optionally, the Java Virtual Machine corresponding to the Java process of the MR task reads the timestamp information contained in the agent process survival tag file, and reads the collection parameters contained in the shared class collection tag file when it is determined from the timestamp information that the survival time of the agent process is less than a preset duration threshold.

[0126] In this way, by generating a proxy process survival tag file containing the timestamp information of the first startup of the proxy process, the survival time of the proxy process can be determined easily and conveniently. Only when the proxy process is within the specified survival time will the task process to be executed, such as the Java Virtual Machine corresponding to the Java process, read the collection parameters contained in the shared class collection tag file. This is a fault tolerance mechanism. If the proxy process is not within the specified survival time, it is determined to be an older proxy process, and the collection parameters contained in the shared class collection tag file will no longer be read.

[0127] Optionally, the specific implementation process of step S801 can refer to the technical solution of the foregoing embodiments.

[0128] Step S802: After the agent process starts, it reads the class data related files corresponding to the task to be executed to determine whether the class data related files corresponding to the task to be executed have changed.

[0129] The agent process reads the class data related files corresponding to the task to be executed to determine whether these files have changed. Specifically, it can read the classpath contained in the configuration file. If the classpath in the configuration file has changed, it is determined that the class data related files corresponding to the task to be executed have changed; otherwise, it is determined that the class data related files corresponding to the task to be executed have not changed. Optionally, after the agent process starts, it reads the configuration file corresponding to the MR task to determine whether the configuration file corresponding to the MR task has changed. The configuration file read by the agent process can be, for example,... Figure 9 The task configuration file shown.

[0130] The agent process reads the class data related files corresponding to the task to be executed to determine whether these files have changed. Specifically, it can read the classpaths contained in the error log file. If the number of error log files recording classpath changes is greater than a preset threshold, it is determined that the class data related files corresponding to the task to be executed have changed; if the number of error log files recording classpath changes is less than or equal to the preset threshold, it is determined that the class data related files corresponding to the task to be executed have not changed. Optionally, after the agent process starts, it reads the error log file corresponding to the MR task to determine whether the error log file corresponding to the MR task has changed; the error log file read by the agent process can be, for example, the classpaths contained in the error log file. Figure 9 The single-task error log file shown.

[0131] Optionally, the specific implementation process of step S802 can refer to the technical solution of the foregoing embodiments.

[0132] In step S803, if the class data related files corresponding to the task to be executed have not changed, the agent process checks whether there is a shared class archive marker file and a newly generated shared class list file.

[0133] Optionally, if the agent process reads the class data related files corresponding to the MR task and determines that the class data related files corresponding to the task to be executed have not changed, it means that there is no need to load the configuration file at this time.

[0134] Optionally, the specific implementation process of step S803 can refer to the technical solution of the foregoing embodiments.

[0135] Step S804: If the agent process detects that there is no shared class archive tag file and no newly generated shared class list file, then load the configuration file, which stores the class data required by the task to be executed.

[0136] After the agent process determines that the class data related files corresponding to the task to be executed have not changed, it needs to check whether there is a shared class archive marker file and a newly generated shared class list file in the system environment. If it finds that there is neither a shared class archive marker file nor a newly generated shared class list file in the system environment, it indicates that it is necessary to load the configuration file at this time.

[0137] Optionally, the proxy process loads the configuration file corresponding to the MR task, which stores the class data required by the MR task during execution; wherein, the configuration file loaded by the proxy process can be as follows: Figure 9 The task configuration file shown.

[0138] Optionally, the specific implementation process of step S804 can refer to the technical solution of the foregoing embodiments.

[0139] Step S805: The agent process generates a shared class acquisition marker file containing acquisition parameters; wherein, the shared class acquisition marker file is used to enable the task process of the task to be executed to read the acquisition parameters and acquire class data from the configuration file according to the acquisition parameters.

[0140] Optionally, the agent process generates a shared class acquisition marker file containing acquisition parameters. In this way, the Java Virtual Machine corresponding to the Java process of each MR task can read the acquisition parameters contained in the shared class acquisition marker file and acquire class data from the configuration file according to the read acquisition parameters to obtain class data.

[0141] Optionally, the specific implementation process of step S805 can refer to the technical solution of the foregoing embodiments.

[0142] In step S806, the agent process reads the class data collected by the task to be executed and generates a shared class list file based on the read class data.

[0143] Optionally, the agent process can read the class data collected by each MR task and generate a shared class list file based on the read class data; wherein, the agent process generating the shared class list file based on the read class data can be as follows: Figure 9 The file showing the list of single-task shared classes.

[0144] Optionally, the specific implementation process of step S806 can refer to the technical solution of the foregoing embodiments.

[0145] In step S807, the agent process generates a shared class archive marker file containing usage parameters and class data based on the shared class list file; wherein, the shared class archive marker file is used to enable the task process to read the usage parameters and load the class data contained in the shared class archive marker file according to the usage parameters.

[0146] Optionally, the agent process reads the shared class list file and generates a shared class archive tag file containing usage parameters and class data based on the read shared class list file. In this way, the Java Virtual Machine corresponding to the Java process of each MR task can read the usage parameters contained in the shared class archive tag file and load the class data contained in the shared class archive tag file based on the read usage parameters.

[0147] Optionally, the specific implementation process of step S807 can refer to the technical solution of the foregoing embodiments.

[0148] Step S808: If the task process successfully loads the class data contained in the shared class archive marker file into memory according to the usage parameters, then the task process to be executed will run.

[0149] Optionally, if the Java Virtual Machine corresponding to the Java process of the MR task successfully loads the class data contained in the shared class archive tag file into memory according to the usage parameters, then the Java process of the MR task is run, thereby improving the running speed of the Java process of the MR task.

[0150] Optionally, the specific implementation process of step S808 can refer to the technical solution of the foregoing embodiments.

[0151] In step S809, if the task process fails to load the class data contained in the shared class archive marker file into memory according to the usage parameters, a failure message is output.

[0152] Optionally, if the Java Virtual Machine corresponding to the Java process of the MR task fails to successfully load the class data contained in the shared class archive tag file into memory according to the usage parameters, a failure message is output, which facilitates the operation and maintenance personnel to make adjustments and modifications based on the failure message.

[0153] Optionally, the specific implementation process of step S809 can refer to the technical solution of the foregoing embodiments.

[0154] This application embodiment realizes the automatic deployment of class data sharing and application class data sharing, eliminating the need for manual deployment by operation and maintenance personnel. On the one hand, it improves the deployment efficiency of class data sharing and application class data sharing, making it applicable to large-scale production environments and bringing more development space to large-scale production environments. On the other hand, it reduces the pressure on operation and maintenance personnel, saves manpower and material resources, and also avoids the error-prone phenomenon of manual deployment, thereby improving the deployment accuracy of class data sharing and application class data sharing.

[0155] The following provides a detailed description of another specific application scenario of this application:

[0156] In this embodiment, the example is of deploying an agent process on a network device in standalone mode, and of having tasks to be executed on the network device in standalone mode:

[0157] First, let's introduce the proxy process side. Please refer to [link / reference]. Figure 10 This is a flowchart of an exemplary data processing method; the data processing method may include the following steps, detailed below:

[0158] Step S1001: The agent process deployed on the network device starts at regular intervals.

[0159] Optionally, if the example is that an agent process is deployed on the central node network device and each child node network device under the central node network device in cluster mode, and there are tasks to be executed on each child node network device, the agent process can be started when a start command is received.

[0160] Optionally, the specific implementation process of step S1001 can refer to the technical solution of the foregoing embodiments.

[0161] Step S1002: Determine whether the agent process is being started for the first time; if yes, proceed to step S1003; otherwise, proceed to step S1004.

[0162] If the agent process is starting for the first time, there are no shared class collection marker files, shared class list files, or shared class archive marker files in the current system environment. If the agent process is not starting for the first time, there are shared class collection marker files, shared class list files, and shared class archive marker files in the current system environment (except in the case of deletion).

[0163] Optionally, the specific implementation process of step S1002 can refer to the technical solution of the foregoing embodiments.

[0164] In step S1003, the agent process loads the configuration file, generates a shared class collection tag file containing collection parameters, generates a shared class list file based on the read class data, and generates a shared class archive tag file containing usage parameters and class data based on the shared class list file.

[0165] Among them, the shared class acquisition tag file is used to enable the task process of the task to be executed to read the acquisition parameters and acquire class data from the configuration file according to the acquisition parameters; the shared class archive tag file is used to enable the task process to read the usage parameters and load the class data contained in the shared class archive tag file according to the usage parameters.

[0166] Optionally, the specific implementation process of step S1003 can refer to the technical solution of the foregoing embodiments.

[0167] In step S1004, the agent process reads the class data related file corresponding to the task to be executed and determines whether the class data related file corresponding to the task to be executed has changed; if yes, then proceed to step S1005; if no, then proceed to step S1006.

[0168] Among them, class data-related files include configuration files for storing class data required by tasks to be executed, and error log files for recording changes to the classpath.

[0169] Optionally, the agent process reads the class path contained in the configuration file and performs a consistency check on it. This check may include the size of the JAR files under the class path, timestamp information, etc. If the class path contained in the configuration file is found to have changed, it is determined that the class data related files corresponding to the task to be executed have changed.

[0170] Optionally, the agent process reads the classpath contained in the error log file. This is a classpath change check of the running process. It can check whether the number of error log files is greater than a preset threshold, such as whether the number of error log files is greater than 0.5% of the total number of log files corresponding to the total number of tasks in the agent process's running cycle. If the number of error log files is found to be greater than the preset threshold, it is determined that the class data related files corresponding to the task to be executed have changed.

[0171] Optionally, the specific implementation process of step S1004 can refer to the technical solution of the foregoing embodiments.

[0172] In step S1005, the agent process deletes the symbolic links to the shared class archive marker files, the shared class list files, and the shared class collection marker files under the specified path, and updates the configuration file.

[0173] Specifically, when the agent process determines that the class data related files corresponding to the task to be executed have changed, it deletes the symbolic link of the shared class archive marker file in the specified path. This will not affect the task that uses the class data contained in the shared class archive marker file, thus ensuring the normal execution of the task. At the same time, it deletes the shared class collection marker file and the shared class list file, so that after the class data related files corresponding to the task to be executed change, the updated configuration file is reloaded to generate the latest shared class collection marker file and the latest shared class list file.

[0174] Optionally, the specific implementation process of step S1005 can refer to the technical solution of the foregoing embodiments.

[0175] In step S1006, the agent process checks whether a newly generated shared class list file exists; if yes, proceed to step S1007; otherwise, proceed to step S1008.

[0176] If the agent process determines that the class data related files for the task to be executed have not changed, it will further check whether a newly generated shared class list file exists in the system environment.

[0177] Optionally, the specific implementation process of step S1006 can refer to the technical solution of the foregoing embodiments.

[0178] In step S1007, the agent process generates the latest shared class archive tag file based on the existing shared class list file and the newly generated shared class list file, and deletes the symbolic link of the shared class archive tag file in the specified path.

[0179] If the agent process detects that both an existing shared class list file and a newly generated shared class list file exist in the system environment, it can update the existing shared class list file based on the newly generated file, generating the latest shared class archive tag file. Simultaneously, it deletes the symbolic link to the shared class archive tag file in the specified path. This does not affect tasks using the class data contained in the shared class archive tag file, ensuring normal task execution. It also generates a symbolic link to the latest shared class archive tag file in the specified path, or alternatively, it can replace the existing symbolic link with the latest shared class archive tag file.

[0180] Optionally, the specific implementation process of step S1007 can refer to the technical solution of the foregoing embodiments.

[0181] In step S1008, the agent process checks whether a shared class archive tag file exists; if yes, proceed to step S1009; otherwise, proceed to step S1010.

[0182] If the agent process detects a newly generated shared class list file in the system environment, it will further check whether a shared class archive tag file exists in the system environment.

[0183] Optionally, the specific implementation process of step S1008 can refer to the technical solution of the foregoing embodiments.

[0184] Step S1009: The agent process updates the identifier of the shared class archive tag file.

[0185] If the agent process detects a shared class archive tag file in the system environment, this shared class archive tag file may be unchanged or newly generated after changes. In this case, the identifier of the shared class archive tag file needs to be updated. It is understood that the identifier of a shared class archive tag file is unique.

[0186] Optionally, the specific implementation process of step S1009 can refer to the technical solution of the foregoing embodiments.

[0187] In step S1010, the agent process determines whether an incremental shared class collection marker file is needed; if yes, then step S1011 is executed; otherwise, step S1012 is executed.

[0188] The incremental shared class acquisition marker file is used to enable the task process to read the incremental acquisition parameters and acquire newly added class data from the updated configuration file based on the incremental acquisition parameters.

[0189] Optionally, the specific implementation process of step S1010 can refer to the technical solution of the foregoing embodiments.

[0190] Step S1011: The agent process generates an incremental shared class acquisition marker file containing incremental acquisition parameters.

[0191] If the agent process determines that an incremental shared class acquisition marker file is needed, then an incremental shared class acquisition marker file containing incremental acquisition parameters will be generated at this time.

[0192] Optionally, the specific implementation process of step S1011 can refer to the technical solution of the foregoing embodiments.

[0193] Step S1012: The agent process updates the status of the incremental shared class collection marker file.

[0194] Specifically, if the agent process determines that the incremental shared class collection marker file is not needed in the system environment, then the status of the incremental shared class collection marker file will be updated. Specifically, when the incremental shared class collection marker file is needed, the status of the incremental shared class collection marker file will be enabled; when the incremental shared class collection marker file is not needed, the status of the incremental shared class collection marker file will be maintained.

[0195] Optionally, the specific implementation process of step S1012 can refer to the technical solution of the foregoing embodiments.

[0196] Next, we will introduce the task to be performed. Please refer to [link / reference]. Figure 11 This is a flowchart of an exemplary data processing method; the data processing method may include the following steps, detailed below:

[0197] Step S1101: Start the Java process for the task to be executed.

[0198] Optionally, the Java process for the task to be executed can be started by receiving a start command, such as by adding the parameter -XX:WorkDirForAppCDS= <path>The working directory to which the process belongs.

[0199] Optionally, the Java process of the task to be executed can be any Java process of the task to be executed, such as the Java process of MR type task, Spark type task, or SQL type task.

[0200] In step S1102, the Java Virtual Machine corresponding to the Java process checks whether a working directory exists; if not, proceed to step S1103; if yes, proceed to step S1104.

[0201] The working directory (also a specified path) is used to store various tag files generated by the agent process, such as shared class collection tag files, shared class archive tag files, and agent process liveness tag files.

[0202] In step S1103, the Java Virtual Machine corresponding to the Java process outputs a message that a working directory does not exist, and then jumps to step S1112.

[0203] If the Java Virtual Machine corresponding to the Java process detects that the working directory does not exist, that is, the various marker files generated by the agent process do not exist, then the message "no working directory" can be output directly.

[0204] In step S1104, the Java Virtual Machine corresponding to the Java process checks whether a shared class archive tag file exists; if yes, proceed to step S1105; otherwise, proceed to step S1106.

[0205] If the Java Virtual Machine corresponding to the Java process detects the existence of a working directory, it further checks whether a shared class archive tag file exists in that working directory.

[0206] Step S1105: The Java Virtual Machine corresponding to the Java process reads the usage parameters contained in the shared class archive tag file.

[0207] If the Java Virtual Machine corresponding to the Java process detects the existence of a shared class archive tag file in the working directory, it reads the usage parameters contained in the shared class archive tag file.

[0208] In step S1106, the Java Virtual Machine corresponding to the Java process checks whether an incremental shared class collection marker file exists; if yes, proceed to step S1107; otherwise, proceed to step S1108.

[0209] Specifically, the Java Virtual Machine corresponding to the Java process checks if there is a shared class archive marker file in the working directory, and then further checks if there is an incremental shared class collection marker file.

[0210] In step S1107, the Java Virtual Machine corresponding to the Java process reads the incremental acquisition parameters contained in the incremental shared class acquisition marker file, and acquires the newly added class data from the configuration file according to the incremental acquisition parameters.

[0211] Specifically, if the Java Virtual Machine corresponding to the Java process detects the existence of an incremental shared class collection marker file, it reads the incremental collection parameters contained in the incremental shared class collection marker file and collects the newly added class data from the configuration file according to the incremental collection parameters.

[0212] In step S1108, the Java Virtual Machine corresponding to the Java process preprocesses the parameters.

[0213] The Java Virtual Machine corresponding to the Java process can preprocess the usage parameters so that the class data contained in the shared class archive tag file can be loaded later based on the preprocessed usage parameters.

[0214] In step S1109, the Java Virtual Machine corresponding to the Java process loads the class data contained in the shared class archive tag file according to the preprocessed usage parameters.

[0215] In this context, the Java Virtual Machine corresponding to the Java process preprocesses the usage parameters and then loads the class data contained in the shared class archive tag file based on the preprocessed usage parameters.

[0216] In step S1110, the Java Virtual Machine corresponding to the Java process determines whether the class data contained in the shared class archive tag file has been successfully loaded; if not, step S1111 is executed, and if yes, step S1112 is executed.

[0217] In this process, after the Java Virtual Machine corresponding to the Java process loads the class data contained in the shared class archive tag file according to the preprocessed usage parameters, it needs to further determine whether the class data contained in the shared class archive tag file has been successfully loaded.

[0218] Step S1111: The Java Virtual Machine corresponding to the Java process outputs a failure message.

[0219] If the Java Virtual Machine corresponding to the Java process fails to load the class data contained in the shared class archive tag file into memory, a failure message needs to be output so that maintenance personnel can make adjustments and modifications based on the failure message. Optionally, the Java Virtual Machine corresponding to the Java process can also output runtime-related data and parameters.

[0220] Step S1112: Run the Java process.

[0221] If the Java Virtual Machine corresponding to the Java process successfully loads the class data contained in the shared class archive tag file into memory, then the Java process can be run directly.

[0222] In this embodiment, the agent process deployed on the network device interacts with the task to be executed. The agent process generates a shared class collection tag file containing collection parameters and a shared class archive tag file containing usage parameters and class data. In this way, before the task process actually runs, it reads the corresponding tag file to automatically complete the collection or loading of class data. This realizes the automated deployment of class data sharing and application class data sharing. The deployment efficiency and accuracy are both high. Furthermore, class data sharing and application class data sharing technologies can be applied to large-scale production environments, making it easy to promote and use class data sharing and application class data sharing technologies, while bringing more development space to large-scale production environments.

[0223] Figure 12 This is a block diagram illustrating a data processing apparatus according to an embodiment of this application. Figure 12 As shown, the data processing apparatus includes:

[0224] Load module 1201 is configured to load the configuration file after the agent process deployed on the network device starts. The configuration file stores the class data required by the task to be executed.

[0225] The first generation module 1202 is configured to generate a shared class acquisition marker file containing acquisition parameters; wherein, the shared class acquisition marker file is used to enable the task process of the task to be executed to read the acquisition parameters contained in the shared class acquisition marker file, and to acquire class data from the configuration file according to the acquisition parameters, and the configuration file stores the class data required for the execution of the task to be executed;

[0226] The reading module 1203 is configured to allow the proxy process to read class data collected by the task to be executed;

[0227] The second generation module 1204 is configured to generate a shared class list file based on the read class data, and to generate a shared class archive marker file containing usage parameters and class data based on the shared class list file; wherein, the shared class archive marker file is used to enable the task process of the task to be executed to read the usage parameters and class data contained in the shared class archive marker file, and to load the class data contained in the shared class archive marker file according to the usage parameters.

[0228] In this embodiment, a shared class collection tag file containing collection parameters and a shared class archive tag file containing usage parameters and class data are generated by an agent process deployed on a network device. Before the task process actually runs, it can automatically complete the collection or loading of class data by reading the corresponding tag file, thus achieving automated deployment of class data sharing and application class data sharing. This avoids the low efficiency and error-prone nature of deployment by maintenance personnel adding corresponding parameters, greatly improving deployment efficiency and accuracy. Furthermore, due to the high deployment efficiency, class data sharing and application class data sharing technologies can be applied to large-scale production environments, facilitating their widespread adoption and providing more development opportunities for large-scale production environments.

[0229] In one embodiment of this application, the data processing apparatus further includes:

[0230] The module is configured to read the class data related files corresponding to the task to be executed after the proxy process starts, in order to determine whether the class data related files corresponding to the task to be executed have changed.

[0231] The inspection module is configured to check for the existence of a shared class archive marker file and a newly generated shared class list file if the class data related files corresponding to the task to be executed have not changed.

[0232] The execution module is configured to perform the step of loading the configuration file if it is found that there is no shared class archive tag file or no newly generated shared class list file.

[0233] In one embodiment of this application, the data processing apparatus further includes:

[0234] The deletion module is configured to delete the symbolic link of the shared class archive marker file, the shared class list file, and the shared class collection marker file in the specified path if the class data related files corresponding to the task to be executed change. The execution module is configured to perform the step of loading the configuration file.

[0235] In one embodiment of this application, the determining module includes:

[0236] The first reading unit is configured to read the classpath contained in the configuration file;

[0237] The first determining unit is configured to determine that the class data related files corresponding to the task to be executed have changed if the class path contained in the configuration file changes.

[0238] The first determining unit is further configured to determine that the class data related files corresponding to the task to be executed have not changed if the class path contained in the configuration file has not changed.

[0239] In one embodiment of this application, the determining module further includes:

[0240] The second reading unit is configured to read the classpath contained in the error log file;

[0241] The second determining unit is configured to determine that the class data related files corresponding to the task to be executed have changed if the number of error log files recording changes in the class path is greater than a preset number threshold.

[0242] The second determining unit is further configured to determine that the class data related files corresponding to the task to be executed have not changed if the number of error log files recording changes in classpath is less than or equal to a preset number threshold.

[0243] In one embodiment of this application, the data processing apparatus further includes:

[0244] The third generation module is configured to generate the latest shared class archive tag file based on the existing shared class list file and the newly generated shared class list file if a newly generated shared class list file is detected. The deletion module is also configured to delete the symbolic links of the shared class archive tag files under the specified path.

[0245] In one embodiment of this application, the data processing apparatus further includes:

[0246] The fourth generation module is configured to generate an incremental shared class acquisition marker file containing incremental acquisition parameters if a shared class archive marker file is detected. The incremental shared class acquisition marker file is used to enable the task process to read the incremental acquisition parameters and acquire newly added class data from the configuration file according to the incremental acquisition parameters.

[0247] In one embodiment of this application, the data processing apparatus further includes:

[0248] The fifth generation module is configured to generate a liveness marker file for the agent process when it is first started. The liveness marker file contains the timestamp information of the agent process when it is first started. The liveness marker file is used to enable the task process to read the timestamp information and the execution module. It is also configured to execute the step of reading the collection parameters when the survival time of the agent process is determined to be less than the preset duration threshold based on the timestamp information.

[0249] In one embodiment of this application, the data processing apparatus further includes:

[0250] The running module is also configured to run the task process of the task to be executed if the task process successfully loads the class data contained in the shared class archive tag file into memory according to the usage parameters.

[0251] The output module is configured to output a failure message if the task process fails to load the class data contained in the shared class archive tag file into memory according to the usage parameters.

[0252] In one embodiment of this application, the network device includes a network device in standalone mode; wherein, the agent process and the task process are deployed on the network device in standalone mode, and the task process includes a short task task process.

[0253] In one embodiment of this application, the network device includes a central node network device in a cluster mode and various sub-node network devices under the central node network device; wherein, the agent process is deployed on the central node network device and various sub-node network devices respectively, the task process is deployed on various sub-node network devices, and the task process includes the task process of long task.

[0254] Among them, the agent process deployed on the central node network device is used to issue class data collection instructions to the agent processes deployed on each sub-node network device. The agent process deployed on each sub-node network device is used to generate a shared class collection marker file according to the class data collection instructions to obtain a shared class list file, and report the obtained shared class list file to the agent process deployed on the central node network device.

[0255] The agent process deployed on the central node network device is also used to generate a shared class archive marker file based on the received shared class list file, and to distribute the generated shared class archive marker file to the agent processes deployed on each child node network device.

[0256] It should be noted that the data processing apparatus provided in the above embodiments and the data processing method provided in the above embodiments belong to the same concept. The specific way in which each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0257] Embodiments of this application also provide an electronic device, including one or more processors and a storage device, wherein the storage device is used to store one or more programs, which, when executed by the electronic device, implement the aforementioned data processing method.

[0258] Figure 13 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0259] It should be noted that, Figure 13 The computer system 1300 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0260] like Figure 13 As shown, the computer system 1300 includes a Central Processing Unit (CPU) 1301, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1302 or programs loaded from Storage Unit 1308 into Random Access Memory (RAM) 1303. The RAM 1303 also stores various programs and data required for system operation. The CPU 1301, ROM 1302, and RAM 1303 are interconnected via a bus 1304. An Input / Output (I / O) interface 1305 is also connected to the bus 1304.

[0261] The following components are connected to I / O interface 1305: an input section 1306 including a keyboard, mouse, etc.; an output section 1307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to I / O interface 1305 as needed. Removable media 1311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1310 as needed so that computer programs read from them can be installed into storage section 1308 as needed.

[0262] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1309, and / or installed from removable medium 1311. When the computer program is executed by central processing unit (CPU) 1301, it performs various functions defined in the system of this application.

[0263] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0264] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0265] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0266] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned data processing method. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0267] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing methods provided in the various embodiments described above.

[0268] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.< / path>

Claims

1. A data processing method, characterized by, The method comprises: After the agent process deployed on the network device is started, a configuration file is loaded, and the configuration file stores class data required by a to-be-executed task when the to-be-executed task is executed; A shared class collection marker file containing collection parameters is generated; wherein the shared class collection marker file is used to enable a task process of the to-be-executed task to read the collection parameters and collect class data from the configuration file according to the collection parameters; Class data collected by the to-be-executed task is read; A shared class list file is generated according to the read class data, and a shared class archive marker file containing usage parameters and the class data is generated according to the shared class list file; wherein the shared class archive marker file is used to enable the task process to read the usage parameters and load the class data contained in the shared class archive marker file according to the usage parameters; Before the configuration file is loaded, the method further comprises: After the agent process is started, a class data related file corresponding to the to-be-executed task is read to determine whether the class data related file corresponding to the to-be-executed task has changed; If the class data related file corresponding to the to-be-executed task has not changed, it is checked whether a shared class archive marker file and a newly generated shared class list file exist; If it is checked that the shared class archive marker file does not exist and the newly generated shared class list file does not exist, the step of loading the configuration file is performed; If the class data related file corresponding to the to-be-executed task has changed, a soft link of the shared class archive marker file, the shared class list file and the shared class collection marker file under a specified path are deleted, and the step of loading the configuration file is performed.

2. The data processing method of claim 1, wherein, The class data related file comprises the configuration file; the reading of the class data related file corresponding to the to-be-executed task to determine whether the class data related file corresponding to the to-be-executed task has changed comprises: A class path contained in the configuration file is read; If the class path contained in the configuration file has changed, it is determined that the class data related file corresponding to the to-be-executed task has changed; If the class path contained in the configuration file has not changed, it is determined that the class data related file corresponding to the to-be-executed task has not changed.

3. The data processing method of claim 1, wherein, The class data related file comprises an error log file, wherein the error log file is used to record changes of the class path; the reading of the class data related file corresponding to the to-be-executed task to determine whether the class data related file corresponding to the to-be-executed task has changed comprises: A class path contained in the error log file is read; If a number of error log files recording changes of the class path is greater than a preset number threshold, it is determined that the class data related file corresponding to the to-be-executed task has changed; If the number of error log files recording changes of the class path is less than or equal to the preset number threshold, it is determined that the class data related file corresponding to the to-be-executed task has not changed.

4. The data processing method of claim 1, wherein, The method further comprises: If it is checked that the newly generated shared class list file exists, then according to the existing shared class list file and the newly generated shared class list file, the latest shared class archiving mark file is generated, and the soft link of the shared class archiving mark file in the specified path is deleted.

5. The data processing method of claim 1, wherein, The method further comprises: If it is checked that the shared class archiving mark file exists, then a shared class incremental collection mark file containing an incremental collection parameter is generated; wherein the shared class incremental collection mark file is used to enable the task process to read the incremental collection parameter and collect newly added class data from the configuration file according to the incremental collection parameter.

6. The data processing method according to any one of claims 1 to 5, wherein, The method further comprises: When the agent process is started for the first time, an agent process survival mark file is generated; wherein the agent process survival mark file contains a time stamp information of the first start of the agent process, and the agent process survival mark file is used to enable the task process to read the time stamp information and execute the step of reading the collection parameter when it is determined according to the time stamp information that the survival duration of the agent process is less than a preset duration threshold.

7. The data processing method according to any one of claims 1 to 5, wherein, The method further comprises: If the task process successfully loads the class data contained in the shared class archiving mark file into the memory according to the use parameter, then the task process of the to-be-executed task is run; If the task process fails to successfully load the class data contained in the shared class archiving mark file into the memory according to the use parameter, then failure information is output.

8. The data processing method of any one of claims 1 to 5, wherein the network device comprises a single-machine-mode network device; wherein the agent process and the task process are deployed on the single-machine-mode network device, and the task process comprises a short-task task process; or, the network device comprises a center node network device in a cluster mode and each sub node network device under the center node network device; wherein the agent process is respectively deployed on the center node network device and each sub node network device, the task process is deployed on each sub node network device, and the task process comprises a long-task task process.

9. The data processing method of claim 8, wherein the agent process deployed on the center node network device is used to issue a class data collection instruction to the agent process deployed on each sub node network device, the agent process deployed on each sub node network device is used to generate a shared class collection mark file according to the class data collection instruction to obtain a shared class list file, and the obtained shared class list file is reported to the agent process deployed on the center node network device; the agent process deployed on the center node network device is further used to generate a shared class archiving mark file according to the received shared class list file, and issue the generated shared class archiving mark file to the agent process deployed on each sub node network device.

10. A data processing apparatus, characterized by, The device comprises: a loading module configured to load a configuration file after the agent process deployed on the network device is started, wherein the configuration file stores class data required by a to-be-executed task when the to-be-executed task is executed. The first generation module is configured to generate a shared collection mark file containing collection parameters; wherein the shared collection mark file is used to enable a task process of the to-be-executed task to read the collection parameters contained in the shared collection mark file, collect class data from the configuration file according to the collection parameters, and the configuration file stores class data required for execution of the to-be-executed task; The reading module is configured to enable the agent process to read the class data collected by the to-be-executed task; The second generation module is configured to generate a shared class list file according to the read class data, and generate a shared class archive mark file containing usage parameters and the class data according to the shared class list file; wherein the shared class archive mark file is used to enable the task process of the to-be-executed task to read the usage parameters and the class data contained in the shared class archive mark file, and load the class data contained in the shared class archive mark file according to the usage parameters; Before the loading of the configuration file, the determination module is configured to enable the agent process to read class data related files corresponding to the to-be-executed task after starting, to determine whether the class data related files corresponding to the to-be-executed task have changed; The checking module is configured to check whether there is a shared class archive mark file and a newly generated shared class list file if the class data related files corresponding to the to-be-executed task have not changed; The execution module is configured to execute the step of loading the configuration file if it is checked that there is no shared class archive mark file and no newly generated shared class list file; The deletion module is configured to delete a soft link of a shared class archive mark file, a shared class list file and a shared collection mark file under a specified path, and execute the step of loading the configuration file if the class data related files corresponding to the to-be-executed task have changed. The class data related files include the configuration file; the determination module includes:

11. The data processing apparatus of claim 10, wherein, The first reading unit is configured to read a class path contained in the configuration file; The first determination unit is configured to determine that the class data related files corresponding to the to-be-executed task have changed if the class path contained in the configuration file has changed; The first determination unit is further configured to determine that the class data related files corresponding to the to-be-executed task have not changed if the class path contained in the configuration file has not changed. The class data related files include an error log file, wherein the error log file is used to record changes of the class path; the determination module further includes:

12. The data processing apparatus of claim 10, wherein, The second reading unit is configured to read a class path contained in the error log file; The second determination unit is configured to determine that the class data related files corresponding to the to-be-executed task have changed if a number of error log files recording changes of the class path is greater than a preset number threshold; The second determination unit is further configured to determine that the class data related files corresponding to the to-be-executed task have not changed if the number of error log files recording changes of the class path is less than or equal to the preset number threshold. The apparatus further includes:

13. The data processing apparatus of claim 10, wherein, ​ The third generating module is configured to generate a latest shared class archiving mark file according to the existing shared class list file and the newly generated shared class list file if it is checked that the newly generated shared class list file exists, and the deleting module is further configured to delete the soft link of the shared class archiving mark file in the specified path.

14. The data processing apparatus of claim 10, wherein, The apparatus further comprises: The fourth generating module is configured to generate an incremental shared class collecting mark file containing an incremental collecting parameter if it is checked that the shared class archiving mark file exists; wherein the incremental shared class collecting mark file is used to enable the task process to read the incremental collecting parameter and collect the newly added class data from the configuration file according to the incremental collecting parameter.

15. The data processing apparatus of any one of claims 10 to 14, wherein, The apparatus further comprises: The fifth generating module is configured to generate an agent process survival mark file when the agent process is started for the first time; wherein the agent process survival mark file contains a time stamp information when the agent process is started for the first time, and the agent process survival mark file is used to enable the task process to read the time stamp information, and the executing module is further configured to execute the step of reading the collecting parameter when it is determined according to the time stamp information that the survival duration of the agent process is less than a preset duration threshold.

16. The data processing apparatus of any one of claims 10 to 14, wherein, The apparatus further comprises: The running module is further configured to run the task process of the to-be-executed task if the task process successfully loads the class data contained in the shared class archiving mark file into the memory according to the usage parameter; The output module is configured to output a failure information if the task process fails to load the class data contained in the shared class archiving mark file into the memory according to the usage parameter.

17. The data processing apparatus of any one of claims 10 to 14, wherein The network device comprises a single-machine-mode network device; wherein the agent process and the task process are deployed on the single-machine-mode network device, and the task process comprises a task process of a short task; Alternatively, The network device comprises a center-node network device in a cluster mode and each sub-node network device under the center-node network device; wherein the agent process is respectively deployed on the center-node network device and each sub-node network device, and the task process is deployed on each sub-node network device, and the task process comprises a task process of a long task.

18. The data processing apparatus of claim 17, wherein The agent process deployed on the center-node network device is used to issue a class data collecting instruction to the agent process deployed on each sub-node network device, the agent process deployed on each sub-node network device is used to generate a shared class collecting mark file according to the class data collecting instruction to obtain a shared class list file, and report the obtained shared class list file to the agent process deployed on the center-node network device; The proxy process deployed on the central node network device is further configured to generate a shared class archiving mark file according to the received shared class list file, and distribute the generated shared class archiving mark file to the proxy process deployed on each sub-node network device.

19. An electronic device, comprising: Comprise: one or more processors; a storage device for storing one or more programs, which, when executed by the electronic device, cause the electronic device to implement the data processing method of any one of claims 1 to 9.

20. A computer readable medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the data processing method of any one of claims 1 to 9.

21. A computer program product comprising computer instructions, characterized in that, The computer program is executed by the processor to implement the data processing method of any one of claims 1 to 9. The computer program is executed by the processor to implement the data processing method of any one of claims 1 to 9.

Images