Reproduction method, re-run method, data system, electronic device, and storage medium

Abstract

A method for reproducing and re-running a target run is disclosed, along with a data system, an electronic device, and a storage medium. The method for reproducing a target run includes: obtaining a target run identifier; determining the target run's code file and environment image based on the target run identifier; constructing the target run's runtime environment based on the code file and environment image, and starting a program process within the runtime environment; in response to the target job being a first type of job: determining the data used by the target run based on the target run identifier, running the target job through the program process, and, during the execution of the target job, invoking a database unit to perform corresponding database operations on the used data. This method for reproducing a target run can completely reproduce the state of the target run during its previous execution, achieving reproducibility of projects or jobs involving database operations.

Description

Technical Field

[0001] Embodiments of this disclosure relate to a method for reproducing a target operation, a rerun method, a data system, a method for utilizing operation information, an electronic device, and a storage medium. Background Technology

[0002] In fields such as cloud computing, big data, and artificial intelligence, the reproducibility of data processing tasks is crucial. It affects the judgment of the correctness, credibility, skill, and randomness of data processing in these fields, and provides a baseline for improvement relative to the original technology.

[0003] The reproducibility of a data processing task refers to its ability to be copied or recreated. For example, in machine learning, the reproducibility of a data processing task means recreating the workflow of the task and obtaining the same results as the initial task. If a data processing task is created using the same data and the same workflow but yields different results, it affects the correctness of the entire workflow. The inability to reproduce data processing tasks, especially in machine learning which is heavily relied upon in daily life, makes it impossible to prove that these systems are impartial, reliable, and interpretable, thus challenging the credibility of the data processing task. For some machine learning models, if the model cannot be replicated, it is impossible to determine whether the improvement in model performance is due to scientific skill or randomness. In the process of improving a model, in order to try to deploy a better version of the model, it is necessary to first replicate the model as a baseline; otherwise, it is impossible to compare two fundamentally different models. Summary of the Invention

[0004] This disclosure provides at least one embodiment of a method for reproducing a target operation, applied to a data system, wherein the data system includes a data processing unit and a database unit, wherein the data processing unit is configured to run at least one job in a project, the database unit is configured to perform database operations, and the data processing unit is further configured to, in response to a data usage request for the database generated during the running of the at least one job, invoke the database unit to perform a database operation corresponding to the data usage request; the reproduction method includes: obtaining a target operation identifier, wherein the target operation identifier is used to indicate a target operation for a target job, the target operation is in a completed state, and the target operation identifier is created during the execution of the target operation; determining the code file of the target operation and the environment image of the target operation based on the target operation identifier; constructing the runtime environment of the target operation based on the code file and the environment image, and starting a program process in the runtime environment; in response to the target job being a first type of job: determining the data used by the target operation based on the target operation identifier, wherein the data used by the target operation is stored in the database, and running the target job through the program process, wherein, during the running of the target job, the database unit is invoked to perform corresponding database operations on the used data.

[0005] At least one embodiment of this disclosure provides a rerun method, including: displaying job execution records corresponding to at least some of the completed jobs; obtaining a target execution identifier number for a target job, wherein the target job corresponds to a execution record in the job execution records, the target job is a run for the target job, the target job is selected from the job execution records, and the rerun method described in any embodiment of this disclosure is executed to rerun the target job.

[0006] This disclosure provides at least one embodiment of a method for reproducing a target run, applied to a data system. The data system includes a data processing unit configured to run at least one job in a project, each job corresponding to a job declaration file. A database unit is configured to perform database operations. The reproduction method includes: obtaining a target run identifier, wherein the target run identifier indicates a target run for a target job, the target run being in a completed state, and the target run identifier being created during the execution of the target run; determining run information corresponding to the target run based on the target run identifier, wherein the run information includes job declaration file information and run process parameters specified when submitting the job run request; obtaining a job declaration file corresponding to the job declaration file information, wherein the job declaration file includes at least one type of job configuration parameters, and the job declaration file information indicates the version of the job declaration file; determining configuration information corresponding to the target run based on at least one type of job configuration parameters and run process parameters specified when submitting the job run request; and reproducing the target run based on the configuration information corresponding to the target run.

[0007] This disclosure provides at least one embodiment of a method for utilizing runtime information, applied to a data system. The data system includes a data processing unit configured to run at least one job in a project, each job corresponding to a job declaration file. A database unit is configured to perform database operations. The data processing unit is further configured to, in response to a data usage request for the database generated during the running of the at least one job, invoke the database unit to perform the database operation corresponding to the data usage request; obtain requests received by the data processing unit, wherein the requests received by the data processing unit include job requests; job requests include job execution requests; job execution requests are used to start the execution of a job; the job execution request includes the job to be run, the job declaration file information corresponding to the job to be run, and the runtime parameters specified when submitting the job execution request; generate runtime information corresponding to each job based on the requests received by the data processing unit, the runtime information including the job declaration file information corresponding to each job and the runtime parameters specified when submitting the job execution request; and reproduce the job or generate an audit log based on the runtime information.

[0008] At least one embodiment of this disclosure provides a data system, including a data processing unit and a database unit. The data processing unit is configured to run at least one job in a project, and the database unit is configured to perform database operations. The data processing unit is further configured to, in response to a data usage request for the database generated during the execution of the at least one job, invoke the database unit to perform a database operation corresponding to the data usage request. The data processing unit includes an acquisition module configured to acquire a target execution identifier, wherein the target execution identifier indicates a target execution for a target job, the target execution is in a completed state, and the target execution identifier indicates that the target job was being executed. The system includes a runtime creation module, configured to determine the target runtime's code file and environment image based on the target runtime identifier; a construction module, configured to construct the target runtime's runtime environment based on the code file and environment image, and start the program process within the runtime environment; and a processing module, configured to, in response to the target job being a first type of job: determine the data used by the target runtime based on the target runtime identifier, and run the target job through the program process, wherein the data used by the target runtime is stored in the database, and during the execution of the target job, the database unit is invoked to perform corresponding database operations on the used data.

[0009] At least one embodiment of this disclosure provides an electronic device, including: a data system as described in any embodiment of this disclosure; and the database.

[0010] At least one embodiment of this disclosure provides an electronic device, including: a processor; a memory including one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to implement the method for reproducing the target operation or the method for rerunning the operation as described in any embodiment of this disclosure.

[0011] At least one embodiment of this disclosure provides a storage medium storing non-transitory computer-readable instructions that, when executed by a computer, implement a method for reproducing the target operation or a method for re-running the operation as described in any embodiment of this disclosure. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0013] Figure 1AA schematic flowchart illustrating an image processing method provided for at least one embodiment of this disclosure;

[0014] Figure 1B A schematic block diagram of a data system provided for at least one embodiment of this disclosure;

[0015] Figure 2 A schematic flowchart of step S20 in a method for reproducing a target operation provided in at least one embodiment of this disclosure;

[0016] Figure 3 A schematic flowchart illustrating a method for reproducing the target operation provided in at least one embodiment of this disclosure;

[0017] Figure 4 A schematic flowchart illustrating a rerun method provided for at least one embodiment of this disclosure;

[0018] Figure 5 A schematic block diagram of a data system provided for at least one embodiment of this disclosure;

[0019] Figure 6A A schematic diagram of an electronic device provided for at least one embodiment of this disclosure;

[0020] Figure 6B A schematic diagram of another electronic device provided for at least one embodiment of this disclosure;

[0021] Figure 7 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0023] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and components are omitted.

[0024] A data processing task can be a complete project or a job within a project. For example, a project can consist of at least one job.

[0025] Some data processing tasks, such as those used for artificial intelligence, are reproducible in that users can rerun the task after a period of time, or they can share the task with other users who can rerun it locally or on a cloud platform without any additional setup or configuration, making it convenient for them to use.

[0026] The reproducibility of data processing tasks is crucial for the development process. Typically, users develop, train, and debug locally, resulting in a pre-trained project or job. However, data processing tasks require matching code, runtime environment, and data support—for example, data storage systems (file systems or data lakes) for reading and managing data. This approach makes it difficult to unify code, environment, and data, and also hinders their corresponding management. Therefore, after a data processing task completes, it's difficult to accurately retrieve the code, runtime environment, and data that were in use during its execution. This makes the task difficult to reproduce and prevents it from being shared with other users.

[0027] At least one embodiment of this disclosure provides a method for reproducing a target run, a rerun method, a data system, an electronic device, and a storage medium. The method for reproducing a target run includes: obtaining a target run identifier, wherein the target run identifier is used to indicate a target run for a target job, the target run is in a completed state, and the target run identifier is created during the execution of the target run; determining the code file and environment image of the target run based on the target run identifier; constructing a runtime environment for the target run based on the code file and environment image, and starting a program process in the runtime environment; in response to the target job being a first type of job: determining the data used by the target run based on the target run identifier, wherein the data used by the target run is stored in a database, and running the target job through a program process, wherein during the execution of the target job, a database unit is invoked to perform corresponding database operations on the data used.

[0028] This method for reproducing the target execution can locate the corresponding code file and environment image based on the target execution identifier to build the runtime environment of the target execution in the previous run and start the program process in the runtime environment. Furthermore, it can also find the data used by the target execution based on the target execution identifier. When the target execution needs to call data, it can also find the data used by the target execution during runtime. Even if the code, environment, and data are modified after the target execution ends, it can accurately locate the code, environment, and data corresponding to the target execution and completely reproduce the state of the target execution in the previous run. This achieves the reproducibility of projects or jobs with database operations. Not only can users rerun this data processing task after a period of time, but they can also share this data processing task with other users, who can easily rerun and use this data processing task on the platform, improving development efficiency.

[0029] The method for reproducing the target operation provided in this disclosure can be applied to the data system provided in this disclosure, which can be configured on an electronic device provided in at least one embodiment of this disclosure. The electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a hardware device such as a mobile phone or tablet computer.

[0030] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, but this disclosure is not limited to these specific embodiments.

[0031] Figure 1A A schematic flowchart illustrating a method for reproducing a target operation provided for at least one embodiment of this disclosure.

[0032] like Figure 1A As shown, at least one embodiment of the present disclosure provides an image processing method including steps S10 to S40.

[0033] For example, the method for reproducing the target operation is applied to a data system.

[0034] Figure 1B A schematic block diagram of a data system provided for at least one embodiment of this disclosure. (See diagram below.) Figure 1B As shown, the data system includes a data processing unit and a database unit.

[0035] For example, the data processing unit is configured to run at least one job in the project, and the database unit is configured to perform database operations.

[0036] For example, the data processing unit is also configured to invoke the database unit to perform database operations corresponding to the data usage request in response to a data usage request for the database generated during the execution of at least one job.

[0037] For example, data systems are used to perform data processing tasks. These tasks can be artificial intelligence tasks such as model training and inference, or big data analysis and processing tasks. Performing these data processing tasks typically requires massive amounts of data.

[0038] A project can be a complete data processing task, which includes one or more jobs with execution order and dependencies.

[0039] For example, to facilitate unified management of code, runtime environment, and data, and to fully utilize the capabilities of the database when executing data processing tasks, data processing units and database units that can be called by the data processing units are set up in the data system. This allows the projects or jobs executed by the data processing units in the data system to be more deeply integrated with database operations such as data querying, data updating, and user-defined function calls. The database units are used to complete the efficient storage, management, querying, and data processing of massive amounts of data, thereby improving the processing efficiency of the data processing units in executing projects or jobs.

[0040] Simultaneously, jobs that can be called after execution can be set in the data processing unit, and the callable jobs provided by the data processing unit can be registered as UDFs (user-defined functions) in the database unit, thereby extending the data processing capabilities of the database unit by utilizing the data processing unit.

[0041] For example, the data processing unit can perform distributed computing and elastic computing, while the database unit can perform distributed batch data storage, management, and querying, and support approximate queries of unstructured data representations.

[0042] It should be noted that the data processing tasks in this disclosure (such as target jobs or target projects) have the need to use data in the database. The data processing unit and the database unit in the data system cooperate with each other. The data processing unit can call the database unit to perform database operations and run the data processing task based on the data obtained from the database operations.

[0043] For example, during the process of running a model training job, the data usage requirements for the database can be data query requirements, such as retrieving sample data from the database for training; during the process of running a model inference job, the data usage requirements for the database can include data query requirements and data update requirements, such as retrieving data to be processed from the database for model inference and updating the database with the inference results; during the process of running a job corresponding to an artificial intelligence project (for example, its corresponding data processing task is to search for images similar to input image A from the base image database), the data usage requirements for the database can include data query requirements, approximate query requirements, UDF call requirements, data update requirements, etc.

[0044] For example, in the process of running an artificial intelligence project, sample images are queried from the database (corresponding to data query requirements), a feature extraction model is trained using the sample images, the trained feature extraction model is run as a callable job, declared as a UDF in the database unit, the UDF is called in the database unit to extract features from the base database images stored in the database (corresponding to UDF call requirements), the base database image features are written into the database (corresponding to data update requirements), and then an approximate query of the features of the input image A is performed within the range of the base database image features (corresponding to approximate query requirements).

[0045] In this disclosure, the data in the database includes not only the data stored in the database, but also the data stored in third-party storage (such as unstructured data storage systems including object storage systems and file storage systems), the address storage of data in third-party storage systems, and the data in the database.

[0046] It should be noted that this disclosure does not impose specific restrictions on the specific structure or implementation method of the data processing unit and the database unit.

[0047] In step S10, the target running identifier number is obtained.

[0048] For example, the target run identifier is used to indicate the target run for the target job, the target run is in the completed state, and the target run identifier is created when the target run is executed.

[0049] For example, the target run ID is the identifier of the target job during its execution. The target run ID can be automatically generated when the target job is run. For example, the target job can be run multiple times, and each run can generate a different target run ID. For example, the target run ID can consist of a preset length of characters, such as a random hexadecimal string.

[0050] For example, the target job generates a first target run identifier (runid1) when it runs for the first time, a second target run identifier (runid2) when it runs for the second time, and a third target run identifier (runid3) when it runs for the third time. If all three runs have already ended, and the user wants to reproduce the state of the target job when it runs for the second time, then the target run identifier is the second target run identifier (runid2), and the target run is the second run.

[0051] For example, the already run jobs can be displayed to the user so that the user can select the job that needs to be reproduced, and the target run identifier can be determined based on the user's selection.

[0052] Then, based on the target run identifier, the code files, environment image, data, etc. required to reproduce the target job can be obtained to re-run the target job.

[0053] In step S20, the target runtime code file and the target runtime environment image are determined based on the target runtime identifier.

[0054] In step S30, the target runtime environment is constructed based on the code file and environment image, and the program process is started in the runtime environment.

[0055] For example, multiple runs of a target job might correspond to multiple different versions of code files and environment images. For instance, the first run of the same target job might use only Python code file 1 and environment image 1 (e.g., PyTorch 1.0), while the second run might use code file 2 and environment image 2 (e.g., PyTorch 2.0). Therefore, to reproduce the target run, it's necessary to determine the code files and environment images used by the target run, build the corresponding container based on the environment image, and run the program process within the container to execute the program corresponding to the code file.

[0056] In step S40, in response to the target job being a first type of job: the data used by the target job is determined according to the target job identifier number. For example, the data used by the target job is stored in a database. The target job is run through a program process. For example, during the running of the target job, the database unit is called to perform corresponding database operations on the data used.

[0057] For example, the target assignment may include the first type of assignment.

[0058] For example, a Type 1 job indicates that the target job cannot be invoked; it is a one-time job that ends after execution. When a Type 1 job runs, it processes specified data within an environment that the code file can run in. Even using the same code file and environment image, different specified data can lead to different results. For instance, training a model using the same code file with sample set 1 and sample set 2 may yield different training results. Therefore, to reproduce the target runtime of a Type 1 job, in addition to determining the code file and environment image used, it is also necessary to determine the data used by the target runtime in order to reproduce the results.

[0059] For example, the data used in the first type of job is stored in a database, and the database unit can perform distributed batch data storage, management, and querying, and support approximate queries of unstructured data representations, thereby making full use of the database's capabilities in the first type of job. The database unit can be used to complete the efficient storage, management, querying, and data processing of massive amounts of data, thereby improving the processing efficiency of the data processing unit in executing the project or job.

[0060] Therefore, the target execution reproduction method provided in at least one embodiment of this disclosure can completely reproduce the target's state when it was previously running, realizing the reproducibility of projects or jobs with database operations. Not only can users rerun the data processing task after a period of time, but they can also share the data processing task with other users, who can conveniently rerun and use the data processing task locally or on a cloud platform, thereby improving development efficiency.

[0061] Depending on the type of the target task, the method of reproducing the target task may vary slightly. The following is a detailed description of the reproduction process for different types of target tasks.

[0062] Figure 2 A schematic flowchart of step S20 in a method for reproducing the target operation provided in at least one embodiment of this disclosure.

[0063] like Figure 2 As shown, step S20 may include at least steps S201-S203.

[0064] In step S201, at least one processing information is determined based on the target operation identifier.

[0065] For example, at least one piece of processing information includes a job declaration file corresponding to the target run identifier, or run information corresponding to the target run identifier, or both a job declaration file corresponding to the target run identifier and run information corresponding to the target run identifier.

[0066] For example, the job declaration file is generated when editing the target job and records configuration information; the runtime information records the runtime parameters specified in the job run request when the target job is started through the job run request.

[0067] In step S202, the configuration information corresponding to the target operation is determined based on at least one processing information.

[0068] For example, configuration information includes at least one type of configuration parameter.

[0069] In step S203, the code file and environment image of the target program are determined based on the configuration information corresponding to the target program.

[0070] As mentioned earlier, a target job can correspond to multiple different versions of code files and environment images. The job declaration file can specify default configuration parameters such as the default code version number and environment image version number for the target job. These default configuration parameters can also be modified when submitting a job execution request. For configuration parameters that have been modified during the target job's runtime, the job will run according to the modified parameters. For configuration parameters that have not been modified during the job's runtime, the job will run according to the default configuration parameters.

[0071] For example, job declaration files are generated when editing a target job. Editing includes creating and / or modifying. For instance, when job A is created, job declaration file 1.1 is generated. After two subsequent edits, job declaration files 1.2 and 1.3 are generated respectively. Information about each job and its corresponding version of the job declaration file can be displayed to the user, allowing the user to select the job to run.

[0072] The job declaration file records configuration information when editing the target job. For example, it records the address of the environment image used by the target job, the address of the code file, the version number of the code file, the version number of the environment image, and the address of the data used by the target job. Furthermore, for a project's job declaration file, the execution order and dependencies between one or more jobs included in the project can also be declared; for example, the output of job 1 can be used as the input of job 2. Continuing the previous example, when job A is created, job declaration file 1.1 is generated (for example, the default configuration parameters specified in this job declaration file are code version 1 and environment image version 1). After two edits, job declaration file 1.2 (for example, the default configuration parameters specified in this job declaration file are code version 1 and environment image version 2) and job declaration file 1.3 (for example, the default configuration parameters specified in this job declaration file are code version 3 and environment image version 3) are generated. The system displays information about the job declaration files for each version of Job A and other jobs to the user, allowing the user to select which job they want to run and which version of the job declaration file they want to run.

[0073] For example, the job declaration file can also record some running parameters required for the job to run, such as the number of parallel program processes required for elastic computing, the number of central processing units (CPUs) used to execute the target, and the number of graphics processing units (GPUs) used to execute the target. In other words, how many CPUs and / or GPUs are needed to execute the target.

[0074] For example, for a job used for model training, the job declaration file may also include the number of iterations during model training, such as the iteration number indicating when the model stops training after reaching the required number of iterations; in addition, the job declaration file may also include checkpoints, which indicate how many iterations are needed during model training before saving the relevant parameters of the model training so that the program can roll back.

[0075] It should be noted that, depending on actual needs, the job declaration file may include any combination of configuration information, or may include more or other configuration information. Those skilled in the art can declare corresponding configurations in the job declaration file according to actual needs, and this disclosure does not impose specific limitations in this regard. For example, the job declaration file may be in YAML language format. YAML is a highly readable format used to express data serialization and can easily express data formats such as lists, hash tables, and scalars. Of course, this disclosure is not limited to this, and other language formats may also be used to express the job declaration file.

[0076] For example, the job declaration file corresponding to a job can be stored in a code repository along with the corresponding code file and environment image. The code repository can be a version control system with version control and code storage capabilities, such as Git (a distributed version control system) or SVN (Subversion). For instance, the job declaration file and code file corresponding to a job in the code repository can be periodically downloaded to the local machine, or downloaded when a user needs to run a job, allowing the user to view and select the job to run, and then use the code file and environment image to build the runtime environment and start the program process.

[0077] For example, a mapping relationship between the target run identifier and the target job is established when creating the target run identifier. In some examples, in step S201, the target job can be found through the target run identifier based on this mapping relationship, and then the job declaration file can be downloaded from the code repository. For example, in other examples, the target job can be found through the target run identifier based on this mapping relationship, and the job declaration file corresponding to the target job has been pre-stored locally, so the job declaration file can be found locally.

[0078] For example, the configuration information stored in the job declaration file can be understood as some default configuration information corresponding to the target job, while the runtime information includes the runtime parameters specified by the user through the job run request when the target job is submitted, that is, when the target job is started through the job run request.

[0079] For example, a job execution request can be submitted by a user through a relevant client (in which case the user can list the projects / jobs that can be run, then select the job to be run and submit the job execution request) to a web server that receives the job execution request. For example, the job execution request can be in YAML language format, and this disclosure does not impose any specific restrictions on it.

[0080] When submitting a job execution request, you can specify execution process parameters to modify some or all configuration parameters. If a certain type of execution process parameter is specified in the job execution request, the job will be executed according to the execution process parameter of that type instead of the corresponding type of configuration parameter included in the job declaration file (that is, the execution process parameter overrides the corresponding type of configuration parameter in the job declaration file); if a certain type of execution process parameter is not specified in the job execution request, the job will be executed according to the corresponding type of configuration parameter included in the job declaration file.

[0081] For example, runtime parameters are recorded in the runtime information for use in reproducing the target runtime. Runtime parameters can include some or all of the configuration parameters defined in the job declaration file. For instance, runtime parameters can include the address of the code file, the version number of the code file, configuration parameters related to the data used, and runtime parameters. For example, if a user submits a job run request for job A corresponding to job declaration file 1.2 (the default configuration parameters specified in this job declaration file are code version 1 and environment image version 2), and modifies the code version number to code version 3 when submitting the job run request, then the runtime information will record the data version number as code version 3. This confirms that code version 3 and environment version 2 are actually used when executing the target runtime. Therefore, code version 3 and environment version 2 should also be used when reproducing the target runtime.

[0082] For example, the running process parameters may include some or all of the configuration parameters defined in the job declaration file. In other words, configuration parameters not defined in the job declaration file cannot be specified when submitting the job run request and will not appear in the running information.

[0083] For example, the job declaration file corresponding to the target job includes both modifiable and non-modifiable job configuration parameters.

[0084] For example, modifiable job configuration parameters refer to job configuration parameters that can be modified when submitting a job execution request. These include runtime parameters and parameters transmitted to the running program. Runtime parameters may include, for example, at least one of the following: the number of GPUs, the number of CPUs, the amount of memory, or the number of program processes used by the target execution. Parameters transmitted to the running program may include, for example, data configuration parameters. For modifiable job configuration parameters, the specific values ​​of the configuration parameters defined in the job declaration file can also be referred to as default parameter values.

[0085] For example, unmodifiable job configuration parameters refer to job configuration parameters that cannot be modified when submitting a job execution request, such as the job type and the name of the environment image.

[0086] It should be noted that, in different usage scenarios, modifiable and non-modifiable job configuration parameters can be set, selected, and adjusted as needed, and this disclosure does not impose specific restrictions on them.

[0087] For example, when executing a target application, runtime parameters can be specified by submitting a job run request. These parameters can include modifiable job configuration parameters, such as data configuration parameters and runtime parameters. For instance, if a configuration parameter, such as a code configuration parameter, is not defined in the job declaration file, then the runtime parameters will not include the code configuration parameter, and it cannot be modified when submitting the job run request.

[0088] It should be noted that in this disclosure, configuration parameters have two meanings. The type of configuration parameter indicates the meaning represented by the value of the configuration parameter, such as code version number, environment image version number, number of GPUs, memory size, data version identifier, etc. The value of the configuration parameter can be a specific code version number (e.g., 1), a specific environment image version number (e.g., 2), a specific number of GPUs (e.g., 1), a specific memory size (e.g., 2G), or a data version identifier (e.g., November 3, 2020). In addition, the configuration parameters in this disclosure can come from unmodifiable job configuration parameters and modifiable job configuration parameters in the job declaration file, or from runtime information.

[0089] Of course, this disclosure is not limited to this. The operating parameters may also include other types of configuration parameters, which can be selected and set according to actual needs by those skilled in the art.

[0090] For example, the runtime information may also include job declaration file information, such as the version number of the job declaration file. For instance, the job declaration file information can be specified when submitting a job run request. Therefore, the job declaration file in the processing information is determined based on the job declaration file information; that is, the configuration information is determined based on the job declaration file determined by the job declaration file information. Alternatively, when runtime parameters are specified when submitting a run request, the configuration information is determined based on the job declaration file and runtime parameters determined by the job declaration file information.

[0091] For example, the runtime information can be stored in the storage area of ​​the data processing unit, corresponding to the target runtime identifier, such as in a distributed database (e.g., FoundationDB) within the data processing unit. For instance, in step S201, the runtime information for executing the target runtime can be retrieved from the storage area using the target runtime identifier.

[0092] In step S202, the configuration information corresponding to the target operation is determined based on at least one processing information.

[0093] For example, configuration information includes at least one type of configuration parameter.

[0094] For example, in step S201, based on the target run identifier, the job declaration file (which includes code version 1 and environment image version 2) and the run information (which includes code version 3) corresponding to the target run identifier are determined; in step S202, based on these two types of processing information, the configuration information corresponding to the target run is determined, namely the configuration parameters of the two types: code version 3 and environment image version 2.

[0095] Understandably, if the configuration parameters are not modified when submitting the job run request, or if all configuration parameters are modified, only one type of processing information, either the job declaration file or the runtime information, needs to be obtained. In step S203, the code file and environment image of the target run are determined based on the two types of configuration parameters: code version 3 and environment image version 2 corresponding to the target run. This allows users to modify configuration parameters when submitting a job run request, making job execution more flexible. Furthermore, the user's modifications to the configuration parameters when submitting the job run request are recorded in the runtime information. When reproducing the target run, the configuration parameters used in the target run are determined jointly based on the runtime information and the job declaration file, thus fully ensuring the reproducibility of the target run.

[0096] For example, step S202 may include: for any type of configuration parameter, selecting target processing information from at least one type of processing information, and using the parameters of any type contained in the target processing information as the configuration parameters of any type, wherein the target processing information is the processing information with the highest priority among one or more types of processing information containing parameters of any type, and the priority of the running information is higher than the priority of the job declaration file.

[0097] For example, for any type of configuration parameter, such as code configuration parameters, which could be code version numbers, if the runtime information records code configuration parameters, then the code configuration parameters recorded in the runtime information are selected as the code configuration parameters in the configuration information (at this time, the processing information containing this type of parameter is the runtime information and the job declaration file, and the job declaration file has a higher priority, so the runtime information is the target processing information). If the runtime information does not record code configuration parameters, then the code configuration parameters recorded in the job declaration file (at this time, the processing information containing this type of parameter is only the job declaration file, so the job declaration file is the target processing information) are selected as the code configuration parameters in the configuration information.

[0098] In other words, during the determination of configuration information, parameters set through the job execution request during target execution are prioritized as the parameters for reproducing the target execution. If no corresponding type of configuration parameter is specified during target execution, i.e., no corresponding type of configuration parameter is recorded in the execution information, then the default parameters of the corresponding type recorded in the job declaration file are used as the configuration parameters in the configuration information. Of course, if the execution information contains all types of configuration parameters in the configuration information, then the parameters of the corresponding type in the execution information can be used as the configuration parameters in the configuration information.

[0099] For example, the code files, environment image, data, and runtime parameters used for the target operation can all be reconfigured and recorded in the runtime information. This improves the flexibility of the target job during runtime. At the same time, when reproducing the target operation, the configuration parameters of the target operation can be reproduced based on the runtime information, thereby ensuring the reproducibility of the target job.

[0100] Therefore, in the target operation reproduction method provided in at least one embodiment of this disclosure, not only can the default settings when editing the target job be used for rerunning, but the state of the target operation can also be completely reproduced, and the additional parameters set when the target operation is executed can be accurately identified in order to accurately reproduce the target operation.

[0101] For example, in response to a target job being a first type of job, at least one type of configuration parameter includes code configuration parameters and environment configuration parameters. For example, code configuration parameters include the code version number of a code file or the storage address of a code file. For example, environment configuration parameters include the environment image version number of an environment image or the storage address of an environment image.

[0102] For example, step S203 may include: extracting the code file of the corresponding version according to the code configuration parameters, or extracting the code file stored in the storage address of the code file; extracting the environment image of the corresponding version according to the environment configuration parameters, or extracting the environment image stored in the storage address of the environment image.

[0103] For example, an environment image stores the information needed to build the runtime environment of the target application. This includes building information such as plugins, library files, and environment variables required for the target application to run, allowing the runtime environment to be built based on this information. For instance, an environment image can be a file containing this build information.

[0104] For example, when the code configuration parameters include the code version number of the code file, the code file corresponding to that code version number can be extracted from the code repository. For example, when the code configuration parameters include the storage address of the code file, the code file can be extracted from that storage address.

[0105] For example, when the environment configuration parameters include the environment image version number, the environment image corresponding to that environment image version number can be extracted from the code repository. For example, when the environment image configuration parameters include the storage address of the environment image, the environment image can be extracted from that storage address.

[0106] For example, step S30 may include: parsing the environment image to obtain build information, and building at least one container based on the build information; building the target runtime environment in each of the at least one container based on the code file, and starting the program process in each runtime environment.

[0107] For example, since the startup time of a program process is relatively long, in order to improve the execution efficiency of the target, one or more program processes can be started in advance to reduce the time spent starting the program process when it is needed, thereby improving the running efficiency.

[0108] For example, the job declaration file can include runtime parameters, such as a range of the number of program processes to be started. During actual operation, the number of program processes to be started (e.g., N, where N is a positive integer) can be selected from this range based on the current system load, and N containers can be built. Runtime environments can be built in each of the N containers, and program processes can be started in each of the N runtime environments.

[0109] For example, in response to the target job being a first type of job, at least one type of configuration parameter also includes data configuration parameters, which include the data version identifier (e.g., a timestamp) and data location of the data used by the target run.

[0110] For the first type of job, it is also necessary to determine the data that the target operation is targeting when reproducing its target runtime. Therefore, the configuration information determined based on the processing information also needs to include data configuration parameters.

[0111] Similarly, data configuration parameters can be determined by either the job declaration file corresponding to the target job or the operation information corresponding to the target job, or by both. For example, the data configuration parameters included in the job declaration file are: data location indicates the data in the first column of the data table 'test', and the data version identifier is January 1, 2022, 19:00. The data configuration parameters included in the operation information are: data location indicates the data in the first column of the data table 'test', and the data version identifier is January 2, 2022, 19:00. Therefore, the data configuration parameters determined based on the processing information are: data location indicates the data in the first column of the data table 'test', and the data version identifier is January 2, 2022, 19:00. For example, if the data configuration parameter included in the job declaration file is the data in the first column of the data table 'test', and no data version identifier is specified, and the running information also does not include a data version identifier, then the default data version identifier used when running the job will be used. That is, the data configuration parameter used by the target run is: the data location indicates the data in the first column of the data table 'test', and the data version identifier is the target running time (e.g., March 1, 2022, 19:00). For example, the data version identifier can be the time when the data used by the target run is stored, or it can be in the form of a data version number. This disclosure does not limit the specific form of the data version identifier. For example, if the running information records the running time of the target run, this running time can be used as the data version identifier to determine the corresponding data from the database.

[0112] For example, data location indicates the specific location of the data used by the target operation in the data table of the database unit. For example, data location can indicate which column of which data table is used, etc. This disclosure does not impose specific limitations in this regard.

[0113] For example, in step S40, determining the data used by the target run based on the target run identifier may include: determining the data specified by the data version identifier and data location in the database as the data used by the target run based on the data configuration parameters. For example, after starting the program process, the code indicated by the code file is run, and when the program reaches the point where it indicates that the data used by the target run needs to be extracted from the database, the data used by the target run is determined based on the data configuration parameters, thereby running the target job through the program process, and calling the database unit to perform corresponding database operations on the data during the execution of the target job.

[0114] For example, database units have a data version snapshot function (also known as a snapshot). This function, implemented in the operating system and storage technology, records the system state at a specific point in time. For instance, the mapping between data version identifiers and the corresponding data storage units can be recorded in the data table metadata. That is, the database unit's data table metadata records the data version identifiers saved in the snapshot and the corresponding data storage units. Data table metadata describes the data, indicating its storage location, historical data, resource lookups, file records, etc.

[0115] For example, the metadata of the data table records the data in the first column of the data table 'test', and the data with a data version identifier of January 1, 2022, 19:00, located in data storage units 1, 2, and 3. Corresponding to data storage units 1, 2, and 3 is the data in the second column of the data table 'test', with a data version identifier of January 3, 2022, 19:00 (for example, based on the data from January 1, 2022, 19:00, a new column is inserted to the left of the first column of the data table, making the original first column the second column). For example, determining the data specified by the data version identifier and data location in the database as the data used by the target operation based on data configuration parameters can include: in the database unit, based on the correspondence, determining the data storage unit where the data configuration parameters are located from the database; and determining the data used by the target operation based on the data in the data storage unit.

[0116] Continuing with the previous example, the data location in the data configuration parameters indicates the first column of data in the data table 'test', with a data version identifier of January 1, 2022, 19:00. Based on the correspondence between the data configuration parameters and the records in the data table metadata, the data storage unit corresponding to the data configuration parameters is determined to be data storage unit 1-3. Data storage unit 1-3 may store data from each column of the data table 'test', with the first column serving as the data used by the target program.

[0117] For example, in step S40, running the target job through the program process may include: running the target job through the program process according to the running parameters.

[0118] For example, based on the number of CPUs or GPUs used by the target job specified in the running parameters, allocate the corresponding number of CPUs or GPUs to each program process to run the target job.

[0119] Therefore, through the above steps S20-S40, the first type of job can be reproduced. The code file, environment image and data of the target runtime are obtained through the target runtime identifier, thereby completely reproducing the target runtime.

[0120] It should be noted that the target operation reproduction method provided in at least one embodiment of this disclosure can also be used to reproduce the target project. Its specific process is the same as that of the target job being a first type of job, and will not be repeated here.

[0121] For example, the target job may also include a second type of job, which represents a job that is in a callable state at runtime and belongs to service jobs.

[0122] For example, a second type of job, after execution, can generate a Hypertext Transfer Protocol (HTTP) domain name or port for invocation, and process the data corresponding to the invocation request during invocation. For instance, the invoker of a second type of job can be selected from other jobs running in the data processing unit, the database unit, or a third party. That is, when the target job is a second type of job, it can be invoked by other jobs running in the data processing unit, or by the database unit, or by external applications such as APIs (Application Programming Interfaces) or web pages.

[0123] For example, when the target assignment is of type two, refer to Figure 2 Step S20 may still include at least steps S201-S203.

[0124] The specific execution process of steps S201-S202 is the same as the execution process described above when the target job is a first type of job, and the repetition will not be repeated. For example, in response to a target job being a second type of job, at least one type of configuration parameter includes a first configuration parameter. The first configuration parameter includes information for indicating the code file to be run and the environment image to be run.

[0125] The second type of job is one that is in a callable state after it has run. Upon receiving a job call request, this type of job processes the data corresponding to the request and returns the result to the caller. After a second-type job is submitted for execution, a program process is started to listen for call requests. When a call is detected, more program processes are started to process the data corresponding to the request. A second-type job can be called multiple times by different callers during its execution until it finishes.

[0126] The second type of job, corresponding to the reproduction of the target execution, refers to putting the target job back into a running and callable state, rather than reproducing each call during the target execution. Understandably, during the execution of the target job, it may only be necessary to process the data corresponding to the job call request (which can be obtained through the call port), without needing to retrieve data from the database.

[0127] For example, step S203 may include: determining the target execution code file and the target execution environment image based on the first configuration parameters. For example, the second type of job can be further subdivided into model service jobs, function jobs, and application jobs based on their functions.

[0128] For example, when a model service job is invoked, it uses a trained model to infer the data corresponding to the job call request and returns the inference result.

[0129] For example, a function job, when invoked, takes the data corresponding to the job call request as input and returns the function's output. For instance, a function job can implement a specific function, taking the data corresponding to the job call request as input, performing the appropriate data processing according to the function's functionality, and returning the function's output.

[0130] For example, when an application job is invoked, it processes the data corresponding to the target job based on the data in the job invocation request and returns the data processing result.

[0131] The form of the first configuration parameter varies slightly depending on the type of the second type of job.

[0132] For example, when the target job is a model service job, the first configuration parameter includes a model identifier and / or a model address; the first configuration parameter also includes model framework information. Based on the first configuration parameter, determining the target running code file and the target running environment image may include: obtaining the model file corresponding to the model based on the model identifier and / or model address; obtaining the pre-specified environment image corresponding to the framework information as the target running environment image.

[0133] For example, in some examples, the first configuration parameter may include a model identifier (or model address) and a model framework. The model identifier or model address can be used to locate the model file, and the model framework indicates the framework information of the model, such as the model using the TensorFlow framework or other types of frameworks.

[0134] This disclosure does not impose specific restrictions on the model identifier or the specific form of the model framework, as long as the model framework information and the model storage address can be obtained based on the first configuration parameters.

[0135] For example, obtain the model file corresponding to the model based on the model identifier or model address, and determine the environment image corresponding to the running environment of the model file based on the model framework. For example, when the model framework is framework A (e.g., TensorFlow), pre-edit the environment image corresponding to framework A, such as environment image A, and use environment image A as the target running environment image.

[0136] Thus, the model files and environment image are obtained when the target job is a model service job.

[0137] Similar to the first type of job, the first configuration parameter of the second type of job can be determined by either the job declaration file or the runtime information, or by both, which will not be elaborated here.

[0138] Thus, for model service jobs, when editing the job declaration file, users only need to set the model identifier (or model address) and model framework in the job declaration file to define the model service job. The determination of model files and environment images is completed automatically by the system, which reduces the workload of users in editing the code files corresponding to the model files, allowing them to focus on the model itself.

[0139] For example, when the target job is a function job, the first configuration parameter includes code snippet information and the coding language recorded in the function job's job declaration file. Alternatively, the first configuration parameter may include code snippet information recorded in the function job's job declaration file, in which case the coding language of the code snippet can be obtained by parsing the code snippet corresponding to the code snippet information. In other words, the coding language can be included in the first configuration parameter, or it can be obtained by parsing the code snippet. The code snippet information may be, for example, the storage address of the code snippet, through which the code snippet can be retrieved.

[0140] For example, the assignment declaration file may contain code snippets that implement the functionality of the function assignment. The assignment declaration file may also contain the programming language used to write the code snippets, such as C++, Java, Python, etc.

[0141] For example, determining the target running code file and the target running environment image based on the first configuration parameter may include: determining the function addition code file corresponding to the code writing language and pre-specifying it, and concatenating the code snippet and the function addition code file into the target running code file; obtaining the environment image corresponding to the code writing language and pre-specifying it as the target running environment image based on the code writing language; or, determining the target running environment image based on the environment image included in the first configuration parameter (in this case, the user has declared the environment image to be used in the first configuration parameter).

[0142] For example, different coding languages ​​have different function add-in files, which contain common library files, configurations, etc., for different coding languages.

[0143] For example, functions can be added to code files and code snippets and then concatenated into the target executable code file.

[0144] For example, different coding languages ​​correspond to different environment images. For instance, when the coding language is coding language B (e.g., C++), the environment image corresponding to coding language B is pre-edited, for example, environment image B, and environment image B is used as the target environment image for running.

[0145] This yields the code file and environment image when the target job is a function job.

[0146] Thus, for function assignments, when editing the assignment declaration file, users only need to set the code snippet in the assignment declaration file. The system automatically determines the code file and environment image, which reduces the user's workload, allowing them to focus on the code snippet corresponding to the function itself.

[0147] For example, when the target job is an application job, the first configuration parameters include code configuration parameters and environment configuration parameters. The code configuration parameters include the code version number of the code file or the storage address of the code file, and the environment configuration parameters include the environment image version number of the environment image or the storage address of the environment image.

[0148] For example, determining the target runtime code file and the target runtime environment image based on the first configuration parameter may include: extracting the corresponding version of the code file from the code repository or extracting the code file stored in the code file's storage address based on the code configuration parameter; extracting the corresponding version of the environment image from the code repository or extracting the environment image stored in the environment image's storage address based on the environment configuration parameter, wherein the environment image includes the information required to build the runtime environment.

[0149] In one example, code files and environment images can be retrieved from the code repository and environment image storage.

[0150] At this point, the relevant introductions to code configuration parameters and environment configuration parameters can be found in the configuration information for jobs of type 1; details that are repeated will not be repeated here. Furthermore, the process of determining the code file based on the code configuration parameters and the environment image based on the environment configuration parameters is the same as when the target job is a type 1 job; details that are repeated will not be repeated here.

[0151] This yields the code files and environment image when the target job is an application job.

[0152] Subsequently, when the target job is a second type of job, after obtaining the code file and environment image, step S30 is executed. The specific execution process of step S30 is the same as the execution process described above when the target job is a first type of job, and the repeated parts will not be repeated.

[0153] Figure 3 A schematic flowchart illustrating a method for reproducing the target operation provided in at least one embodiment of this disclosure.

[0154] like Figure 3 As shown, when the target job is a second type of job, after executing steps S10-S30, the target operation reproduction method provided in at least one embodiment of this disclosure further includes step S50.

[0155] For example, step S50 may include: in response to a target job being a second type of job, when a program process is invoked by a job invocation request, performing data processing corresponding to the target job on the data corresponding to the job invocation request based on the program process and returning the data processing result. After a second type of job is submitted for execution, a program process is started to listen for invocation requests. When an invocation is detected, more program processes are started to process the data corresponding to the invocation request. When data processing is elastic computing, the number of program processes started when an invocation is detected can be determined based on the number of invocation requests and the load of the data processing unit. In addition, invocation requests can be distributed to different program processes according to the execution status of each program process to achieve load balancing.

[0156] For example, such as Figure 3 As shown, when the target job is a second type of job, after executing steps S10-S30 and S50, the target operation reproduction method provided in at least one embodiment of this disclosure further includes step S60.

[0157] For example, step S60 may include: in response to the target job being a second type of job, determining at least a portion of the data used by the target job based on the target job identifier, wherein at least a portion of the data is stored in the database.

[0158] For example, during the execution of the target job, the database unit is invoked to perform corresponding database operations on at least some of the data.

[0159] Understandably, during the execution of a target operation corresponding to a type II target job, it may only be necessary to process the data corresponding to the job call request (which can be obtained through the call port), without needing to extract data from the database. In other cases, however, data extraction from the database is required during the execution of a target operation corresponding to a type II target job. For example, if the target job is to query the database's base image library for an image similar to the image corresponding to the call request and return the queried approximate image to the caller, then executing the target job requires determining the base image from the database. In this case, besides receiving and processing the data corresponding to the job call request, the target job will also extract at least some of the data used for the target operation from the database and, during the execution of the target job, call database units to perform corresponding database operations on at least some of the data. Therefore, reproducing the target operation requires determining the data in the database used by the target operation.

[0160] In other words, at this point, all the data used by the target operation comes partly from the database and partly from job call requests.

[0161] For example, at this time, in addition to the first configuration parameter, at least one type of configuration parameter also includes a data configuration parameter, which includes the data version identifier and data location of at least a portion of the data.

[0162] For example, determining at least a portion of the data used by the target run based on the target run identifier may include: determining at least a portion of the data in the database specified by the data version identifier and the data location based on data configuration parameters.

[0163] The process of using data configuration parameters to determine at least some of the data can be referred to the aforementioned process when the target job is a type 1 job; the repetitive parts will not be repeated here.

[0164] For example, in one instance, the target job is an application job that queries an image similar to the query image corresponding to the call request from a database of base images (or, the base images themselves are stored in a third-party storage device, and the storage address of the base images in the third-party storage device is stored in the database). This includes: extracting query features from the image corresponding to the call request using a trained feature extraction model; querying approximate features from the base database features (features extracted from the base images using the feature extraction model); and returning the image corresponding to the approximate features as the result to the caller. When the target runtime corresponding to this target job is rerun, the data used by the target runtime (i.e., the base database data or base database features used by the target runtime) can be determined from the database based on data configuration parameters. After the program process starts, the query image is received through the job call request, and the database unit is invoked to perform corresponding database operations on the data used by the target runtime. At this point, the data used by the target runtime comes not only from the external job call request but also from the database.

[0165] In the target execution reproduction method provided in at least one embodiment of this disclosure, the code file and environment image corresponding to the target execution can be accurately located based on the target execution identifier. Furthermore, when data from the database is used during the execution of the target execution, the portion of data from the database used by the target execution at that time can also be accurately located to accurately recreate the target execution. Therefore, users can conveniently reproduce the target execution after it has finished, and can even share the target execution with other users, who can also conveniently rerun the target execution, significantly improving development efficiency and achieving reproducibility of data processing tasks.

[0166] For at least some of the data processing tasks performed by the data system, auditability should also be considered in order to provide better auditing services.

[0167] For example, auditability refers to the ability to generate audit logs for a data system by recording all requests from data processing units, database units, and code repositories. Audit logs can include records related to the operation and execution of projects or jobs; modifications to datasets and database operations can be recorded for easy review later.

[0168] For example, the method for reproducing the target operation provided in at least one embodiment of this disclosure further includes: recording requests received by the network server; and determining the audit log of the data system based on at least a portion of the recorded requests.

[0169] For example, the web server is configured to receive job requests (in some embodiments, this may also include code repository requests) and send the received job requests to the data processing unit. When the received job request includes a code repository request, the received code repository request or a code repository request generated by the web server in response to the job request can be sent to the code repository. Job requests include job run requests and may also include job invocation requests. Job run requests are used to start the execution of a job of type 1 or type 2, job invocation requests are used to invoke a running job of type 2, and code repository requests are used to manipulate the code files and / or environment images corresponding to the job stored in the code repository.

[0170] For example, the first type of job is a job that cannot be invoked; the second type of job is a job that is in an invoked state at runtime.

[0171] For example, in a data system, a first web-based client (e.g., a job display interface for users to select jobs to start or stop; the first web-based client also has an interface for specifying execution parameters when submitting a job execution request) and / or a first client (e.g., a CLI (Command Line Interface) client, which allows users to select jobs to start or stop and specify execution parameters via command line) is provided for sending and submitting job execution requests and job invocation requests. The first web-based client and the first client can also submit code repository requests. The network server sends the received job execution requests and job invocation requests to the data processing unit for corresponding data processing. Furthermore, the network server also sends received code repository requests or code repository requests generated in response to job requests (e.g., when the network server receives a job execution request for a job, it can automatically download the corresponding job declaration file, code file, environment image, etc.) to the code repository for corresponding processing, such as downloading code files or environment images, or submitting code files or environment images.

[0172] Therefore, whether it is a job request or a code repository request, it will pass through the web server, so monitoring the web server can record all requests related to the data processing unit.

[0173] For example, it can also record all requests received by the database unit.

[0174] For example, in a data system, a second web page (e.g., a web page that can submit SQL commands) and / or a second client (e.g., a client that can send SQL commands) are also provided for sending database commands to the database unit. All requests related to the database unit can be determined by monitoring all requests received by the database unit.

[0175] For example, all requests received by the web server and all requests received by the database unit are associated with user information. For instance, when a user registers, the user's code repository account can be associated with the data processing unit and the database unit, thereby filtering out the requests sent by the target user from all requests received by the database unit and the data processing unit to generate audit logs for the target user.

[0176] For example, at least one embodiment of this disclosure also provides a method for reproducing a target operation, applied to a data system. For example, the data system includes a data processing unit configured to run at least one job in a project, each job corresponding to a job declaration file, and a database unit configured to perform database operations. Further details regarding the data system can be found at the foregoing description and will not be repeated here.

[0177] For example, at least one embodiment of this disclosure provides a reproduction method including: obtaining a target run identifier, wherein the target run identifier is used to indicate a target run for a target job, the target run is in a completed state, and the target run identifier is created during the execution of the target run; determining the run information corresponding to the target run based on the target run identifier, wherein the run information includes job declaration file information and run process parameters specified when submitting the job run request; obtaining a job declaration file corresponding to the job declaration file information, wherein the job declaration file includes at least one type of job configuration parameters, and the job declaration file information is used to indicate the job declaration file in parameter form; determining the configuration information corresponding to the target run based on at least one type of job configuration parameters and run process parameters specified when submitting the job run request; and reproducing the target run based on the configuration information corresponding to the target run.

[0178] The specific process of obtaining the target run identifier and determining the corresponding run information can be found in the relevant descriptions above, and will not be repeated here.

[0179] For example, as mentioned earlier, the job declaration file information can be the version number of the job declaration file. In other words, the configuration information is determined based on the job declaration file identified by the job declaration file information and the runtime parameters specified when submitting the job declaration file information.

[0180] Specifically, for a certain type of configuration parameter, if the runtime parameters record a configuration parameter of that type, then the runtime information is used as the target processing information, and the parameter of the corresponding type recorded in the runtime parameters is used as the configuration parameter of that type in the configuration information. For example, taking data configuration parameters as an example, if the runtime parameters record data configuration parameters, then the data configuration parameters recorded in the runtime parameters are used as the data configuration parameters in the configuration information. Specifically, the specific value of the data configuration parameter in the configuration information is the same as the specific value of the data configuration parameter recorded in the runtime parameters.

[0181] If the configuration parameters of this type are not recorded in the running process parameters, the job declaration file will be used as the target processing information, and the job configuration parameters of the corresponding type recorded in the job declaration file will be used as the configuration parameters of this type in the configuration information.

[0182] In other words, the parameters set by the job execution request during the execution of the target are still preferred as the parameters for reproducing the execution of the target. If no corresponding type of configuration parameter is specified during the execution of the target, that is, no corresponding type of configuration parameter is recorded in the execution information, then the default parameter of the corresponding type recorded in the job declaration file determined by the job declaration file information will be used as the configuration parameter in the configuration information.

[0183] Therefore, in the target operation reproduction method provided in at least one embodiment of this disclosure, not only can the default settings when editing the target job be used for rerunning, but other job declaration files can also be selected as default settings during the running phase, and the state of the target operation can be completely reproduced, which improves the flexibility of the target job operation. At the same time, when reproducing the target operation, the configuration parameters of the target operation can be reproduced according to the running information, thereby ensuring the reproducibility of the target job.

[0184] For specific details on determining the configuration information, please refer to the aforementioned content, which will not be repeated here.

[0185] Furthermore, based on the configuration information corresponding to the target runtime, the code files and environment image of the target runtime can be determined. In some embodiments, the data used by the target runtime can also be determined, thereby enabling the target runtime to be rerun. The specific process is as described above, and repeated details will not be repeated here.

[0186] Understandably, after the target run is completed, the target run results (e.g., if the target run is a training job and the trained model is stored) can be stored, and the results obtained from rerunning the target run can be compared with the target run results to evaluate the accuracy of reproduction.

[0187] For example, at least one embodiment of this disclosure also provides a method for utilizing operational information, which is applied to a data system. For a description of the data system, please refer to the foregoing content, which will not be repeated here.

[0188] For example, at least one embodiment of this disclosure provides a method for utilizing runtime information: obtaining a request received by a data processing unit; generating runtime information corresponding to each run based on the request received by the data processing unit; the runtime information includes job declaration file information corresponding to each run and runtime process parameters specified when submitting the job run request; and reproducing the job or generating an audit log based on the runtime information. The request received by the data processing unit includes a job request; the job request includes a job run request; the job run request is used to start the run of a job; the job run request includes the job to be run, the job declaration file information corresponding to the job to be run, and the runtime process parameters specified when submitting the job run request. For a description of the request received by the data processing unit, the job run request, and the runtime process parameters, please refer to the foregoing content, which will not be repeated here.

[0189] For example, upon receiving a request, the data processing unit can parse the received request, generate and record the corresponding operation information for each operation. The operation information includes the job declaration file information for each operation and the operation process parameters specified when submitting the job operation request, so as to reproduce the job or generate audit logs based on the operation information.

[0190] For details on the specific process of reproducing jobs or generating audit logs based on operational information, please refer to the aforementioned content, which will not be repeated here.

[0191] In this way, operational information can be generated based on the requests received by the data processing unit, which can be used for both operational reproduction and auditing.

[0192] For example, at least one embodiment of this disclosure also provides a rerun method. Figure 4 This is a schematic flowchart illustrating a rerun method provided for at least one embodiment of the present disclosure.

[0193] like Figure 4 As shown, a rerun method provided in at least one embodiment of this disclosure may include steps S70-S90.

[0194] Step S70: Display the job execution records corresponding to at least some of the jobs that have been completed.

[0195] For example, a human-computer interaction interface is provided, which displays the job execution records corresponding to at least some of the completed jobs.

[0196] For example, each job execution record may include a execution identifier. In addition, each job execution record may also include information such as execution time; this disclosure does not impose specific limitations on this.

[0197] Step S80: Obtain the target running identifier number.

[0198] For example, the target run identifier is the target run identifier corresponding to the target run that the user wants to reproduce.

[0199] For example, the target run corresponds to a run record in the job run record. The target run is a run for the target job, and the target run is selected from the job run record.

[0200] For example, users can select the target run identifier corresponding to the target run they want to reproduce from the job run record, click on the run record, and display the corresponding interactive interface so that users can rerun the target run.

[0201] For example, the interactive interface can display the target run's running information, job declaration file, etc., and can also provide corresponding buttons for users to select to start rerunning the target run. This disclosure does not impose specific restrictions on this.

[0202] For example, the system can automatically search for the target run identifier from the job run record. For instance, the user can provide information such as the run time, and the system can automatically search for the matching target run identifier from the job run record.

[0203] This disclosure does not impose specific restrictions on the process of obtaining the target operation identifier.

[0204] Step S90: Based on the target run identifier, execute the reproduction method described in any embodiment of this disclosure to rerun the target run.

[0205] After obtaining the target run identifier, the target run can be rerun by referring to the reproduction method provided in any of the embodiments described above.

[0206] The specific execution process of the reproduction method can be referred to the foregoing content, and will not be repeated here. Furthermore, the rerun method provided in at least one embodiment of this disclosure can achieve the same technical effect as the aforementioned target execution reproduction method, and will not be repeated here.

[0207] At least one embodiment of this disclosure also provides a data system. Figure 5 A schematic block diagram of a data system provided for at least one embodiment of this disclosure.

[0208] like Figure 5 As shown, the data system 500 may include a data processing unit 501 and a database unit 502.

[0209] For example, data processing unit 501 is configured to run at least one job in a project, and database unit 502 is configured to perform database operations.

[0210] For example, the data processing unit 501 is also configured to invoke the database unit to perform database operations corresponding to the data usage request in response to a data usage request for the database generated during the execution of at least one job.

[0211] For a description of the data processing unit 501 and the database unit 502, please refer to the relevant content in the target operation reproduction method provided in at least one embodiment of this disclosure, which will not be repeated here.

[0212] For example, the data processing unit 501 may include an acquisition module 5011, a determination module 5012, a construction module 5013, and a processing module 5014.

[0213] For example, the acquisition module 5011 is configured to acquire the target run identifier. For example, the target run identifier is used to indicate the target run for the target job, the target run is in a completed state, and the target run identifier is created when the target run is executed.

[0214] The module 5012 is configured to determine the target running code file and the target running environment image based on the target running identifier.

[0215] Build module 5013 is configured to build the target runtime environment based on code files and environment image, and start the program process in the runtime environment.

[0216] The processing module 5014 is configured to respond to a target job of type 1: determine the data used by the target job based on the target job identifier, wherein the data used by the target job is stored in the database; and run the target job through the program process, wherein the database unit is called to perform corresponding database operations on the data used during the execution of the target job.

[0217] For example, these modules can be implemented through hardware (e.g., circuit) modules, software modules, or any combination of both. The following embodiments are the same and will not be described again. For example, these units can be implemented through a central processing unit (CPU), a graphics processing unit (GPU), a tensor processor (TPU), a field-programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities, as well as corresponding computer instructions.

[0218] For example, the acquisition module 5011, determination module 5012, construction module 5013, and processing module 5014 may include code and programs stored in memory; the processor may execute the code and programs to implement some or all of the functions of the acquisition module 5011, determination module 5012, construction module 5013, and processing module 5014 as described above. For example, the acquisition module 5011, determination module 5012, construction module 5013, and processing module 5014 may be dedicated hardware devices used to implement some or all of the functions of the acquisition module 5011, determination module 5012, construction module 5013, and processing module 5014 as described above. For example, the acquisition module 5011, determination module 5012, construction module 5013, and processing module 5014 may be a circuit board or a combination of multiple circuit boards used to implement the functions described above. In embodiments of this disclosure, the circuit board or combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processor; and (3) processor-executable firmware stored in memory.

[0219] It should be noted that module 5011 can be used to implement... Figure 1A As shown in step S10, module 5012 is determined to be able to implement Figure 1A As shown in step S20, the construction module 5013 can be used to implement... Figure 1A The processing module 5014 can be used to implement step S30 as shown. Figure 1A The step S40 is shown. Therefore, for a detailed description of the functions that the acquisition module 5011, the determination module 5012, the construction module 5013 and the processing module 5014 can achieve, please refer to the relevant descriptions of steps S10 to S40 in the embodiment of the above-described target operation reproduction method. Repeated descriptions will not be repeated.

[0220] It should be noted that in the embodiments of this disclosure, the data system 500 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited and can be constructed from analog devices, digital chips, or other suitable methods according to circuit principles.

[0221] This data system provides a resource-sharing platform where users can share pre-trained or debugged jobs or projects, such as trained model service jobs and function jobs, thus offering users more flexible resources. Furthermore, any previously run data processing task can be reproduced within the system, facilitating local development and training. In addition, since data processing tasks run within the system, all users can easily access pre-trained or debugged jobs or projects from other users, providing a more convenient sharing platform, improving development efficiency, and saving development costs.

[0222] At least one embodiment of this disclosure also provides an electronic device. Figure 6A This is a schematic diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0223] For example, such as Figure 6A As shown, the electronic device includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604. The processor 601, communication interface 602, and memory 603 communicate with each other via the communication bus 604. The processor 601, communication interface 602, and memory 603 can also communicate with each other via a network connection. This disclosure does not limit the type and function of the network. It should be noted that... Figure 6A The components of the electronic device shown are merely exemplary and not limiting; the electronic device may have other components depending on the actual application requirements.

[0224] For example, memory 603 is used to store computer-readable instructions non-transitory. When processor 601 executes the computer-readable instructions, it implements the method for reproducing or rerunning the target operation according to any of the above embodiments. For specific implementations and explanations of the various steps of the method for reproducing or rerunning the target operation, please refer to the embodiments of the method for reproducing or rerunning the target operation described above, and will not be repeated here.

[0225] For example, other implementations of the method for reproducing or rerunning the target operation by the processor 601 executing computer-readable instructions stored in the memory 603 are the same as those mentioned in the foregoing method embodiment section, and will not be repeated here.

[0226] For example, the communication bus 604 can be a Peripheral Component Interconnect Standard (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not indicate that there is only one bus or one type of bus.

[0227] For example, communication interface 602 is used to enable communication between electronic devices and other devices.

[0228] For example, processor 601 and memory 603 can be located on the server side (or in the cloud).

[0229] For example, processor 601 can control other components in an electronic device to perform desired functions. Processor 601 can be a central processing unit (CPU), network processor (NP), tensor processor (TPU), or graphics processing unit (GPU) with data processing and / or program execution capabilities; it can also be a digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The central processing unit (CPU) can be based on x86 or ARM architectures, etc.

[0230] For example, memory 603 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and processor 601 may execute the computer-readable instructions to implement various functions of the electronic device. Various application programs and various data may also be stored in the storage medium.

[0231] For example, a detailed description of the process of reproducing or rerunning the target operation by an electronic device can be found in the relevant descriptions in the embodiments of the reproducing or rerunning method of the target operation, and repeated descriptions will not be repeated.

[0232] At least one embodiment of this disclosure also provides an electronic device. Figure 6B A schematic diagram of another electronic device provided for at least one embodiment of this disclosure.

[0233] like Figure 6B As shown, the electronic device 600 includes a data system 605. For example, the data system 605 can be the aforementioned data system 500, and will not be repeated here.

[0234] It should be noted that in the embodiments of this disclosure, the electronic device 600 may include more or fewer circuits or units, such as other circuits or units that support the operation of the data system 605, and this disclosure does not impose specific limitations on this.

[0235] For example, electronic device 600 may also include database 606 for storing data used by the target operation; for example, electronic device 600 may also include storage device, such as storage device which can be implemented as Figure 6B The code repository 607 in the document can be used to store code files, environment images, job declaration files, etc. Of course, the storage device can also be implemented in other forms, and this disclosure does not impose any specific restrictions on it.

[0236] The connections between the various circuits or units are unrestricted and can be determined according to actual needs. The specific configuration of each circuit or unit is also unrestricted; it can be constructed using analog devices, digital chips, or other suitable methods, based on circuit principles.

[0237] Figure 7 This is a schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. For example, such as... Figure 7 As shown, the storage medium 700 can be a non-transitory computer-readable storage medium on which one or more computer-readable instructions 701 can be stored non-transitory. For example, when the non-transitory computer-readable instructions 701 are executed by a processor, one or more steps of a reproduction method or rerun method that runs according to the target described above can be performed.

[0238] For example, the storage medium 700 can be used in the aforementioned electronic device; for example, the storage medium 700 can be... Figure 6A The memory 603 in the electronic device shown.

[0239] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard disk for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0240] For example, the description of storage medium 700 can be found in the description of memory in the embodiments of the electronic device, and repeated descriptions will not be repeated here.

[0241] The following points should be noted regarding this disclosure:

[0242] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0243] (2) For clarity, the thickness and dimensions of layers or structures are enlarged in the accompanying drawings used to describe embodiments of the invention. It will be understood that when an element such as a layer, film, region, or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element, or there may be intermediate elements present.

[0244] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0245] The above description is only a specific embodiment of this disclosure, but the protection scope of this disclosure is not limited thereto. The protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A method for reproducing the execution of a target, applied to a data system, wherein, The data system includes a data processing unit and a database unit, wherein the data processing unit is configured to run at least one job in the project, and the database unit is configured to perform database operations. The data processing unit is also configured to, in response to a data usage request for the database generated during the execution of the at least one job, invoke the database unit to perform the database operation corresponding to the data usage request. The reproduction method includes: Obtain the target run identifier, wherein the target run identifier is used to indicate the target run for the target job, the target run is in a completed state, and the target run identifier is created when the target run is executed; Based on the target runtime identifier, determine the target runtime's code file and the target runtime's environment image; The target runtime environment is constructed based on the code file and the environment image, and the program process is started in the runtime environment; In response to the target job being a first type of job: Based on the target operation identifier, the data used by the target operation is determined, wherein the data used by the target operation is stored in the database. The target job is run through the program process, wherein the database unit is invoked to perform corresponding database operations on the data used during the execution of the target job.

2. The reproducing method according to claim 1, wherein, Based on the target runtime identifier, the code file and environment image of the target runtime are determined, including: Based on the target operation identifier, at least one processing information is determined, wherein the at least one processing information includes a job declaration file corresponding to the target operation identifier and / or operation information corresponding to the target operation identifier. The job declaration file is generated when editing the target job, and the operation information records the operation process parameters specified in the job operation request when the target operation is started through the job operation request. Based on the at least one processing information, configuration information is determined, wherein the configuration information includes at least one type of configuration parameter; Based on the configuration information, the target running code file and the target running environment image are determined.

3. The reproduction method according to claim 2, wherein, Based on the at least one processing information, configuration information is determined, including: For any type of configuration parameter, select target processing information from the at least one type of processing information, and use the parameters corresponding to the type of the target processing information as the configuration parameters of the type of the target processing information. The target processing information is the processing information with the highest priority among one or more types of processing information containing parameters corresponding to the type of the target processing information, and the priority of the running information is higher than the priority of the job declaration file.

4. The reproduction method according to claim 2, wherein, In response to the target job being a job of the first type, the configuration parameters of at least one type include code configuration parameters and environment configuration parameters. The code configuration parameters include the code version number of the code file or the storage address of the code file; The environment configuration parameters include the environment image version number or the storage address of the environment image; Based on the configuration information, the target running code file and the target running environment image are determined, including: Based on the code configuration parameters, extract the corresponding version of the code file, or extract the code file stored in the storage address of the code file; Based on the environment configuration parameters, extract the corresponding version of the environment image, or extract the environment image stored in the storage address of the environment image, wherein the environment image includes the information required to build the runtime environment.

5. The reproduction method according to any one of claims 2-4, wherein, The at least one type of configuration parameter includes data configuration parameters. The data configuration parameters include the data version identifier and data location of the data used by the target operation. Based on the target operation identifier, the data used by the target operation is determined, including: Based on the data configuration parameters, the data in the database specified by the data version identifier and the data location is determined as the data used by the target operation.

6. The reproduction method according to claim 5, wherein, The database unit has a data version snapshot function. The data table metadata of the database unit records the version identifier and data location of the data saved by the snapshot and the correspondence between the data and the data storage unit where the data is located. The data corresponding to the data configuration parameters is saved by the snapshot. Based on the data configuration parameters, the data in the database specified by the version identifier and the data location is determined as the data used by the target runtime, including: In the database unit, based on the correspondence, the data storage unit where the data corresponding to the data configuration parameter is located is determined from the database according to the data configuration parameter; Based on the data in the data storage unit, determine the data used by the target operation.

7. The reproduction method according to claim 2 or 3, wherein, In response to the target job being a second type of job, the configuration parameters of the at least one type include the first configuration parameters. The second type of job refers to a job that is in a callable state during runtime. The caller of the second type of job is selected from any one of the following: other jobs running in the data processing unit, the database unit, or a third party. The first configuration parameter includes information indicating the code file to be run by the target and the environment image to be run by the target. Based on the configuration information, the target running code file and the target running environment image are determined, including: Based on the first configuration parameters, the target running code file and the target running environment image are determined.

8. The reproduction method according to claim 7, wherein, The second type of job includes model service jobs, which, when invoked, use a model to perform inference on the data corresponding to the job invocation request and return the inference results. The first configuration parameter includes a model identifier and / or a model address; the first configuration parameter also includes model framework information. Based on the first configuration parameters, determining the target running code file and the target running environment image includes: Based on the model identifier and / or model address, obtain the model file corresponding to the model; Obtain the pre-specified environment image corresponding to the framework information as the environment image for the target to run.

9. The reproduction method according to claim 7, wherein, The second type of job includes function jobs, which, when invoked, take the data corresponding to the job invocation request as input to the function and return the output of the function. The first configuration parameter includes code snippet information recorded in the job declaration file of the function job, or the first configuration parameter includes code snippet information and the code writing language recorded in the job declaration file of the function job. Based on the first configuration parameters, determining the target running code file and the target running environment image includes: Based on the coding language, determine the function addition code file, and concatenate the code fragment corresponding to the code fragment information and the function addition code file into the target execution code file; Based on the code writing language, obtain the pre-specified environment image corresponding to the code writing language as the target running environment image, or determine the target running environment image based on the environment image included in the first configuration parameters; wherein, the code writing language is included in the first configuration parameters or obtained by parsing the code fragment.

10. The reproduction method according to claim 7, wherein, The second type of job includes application jobs, which, when invoked, process the data corresponding to the job invocation request, perform the data processing corresponding to the target job, and return the data processing result. The first configuration parameters include code configuration parameters and environment configuration parameters. The code configuration parameters include the code version number of the code file or the storage address of the code file; The environment configuration parameters include the environment image version number or the storage address of the environment image; Based on the first configuration parameters, determining the target running code file and the target running environment image includes: Based on the code configuration parameters, extract the corresponding version of the code file, or extract the code file stored in the storage address of the code file; Based on the environment configuration parameters, extract the corresponding version of the environment image, or extract the environment image stored in the storage address of the environment image, wherein the environment image includes the information required to build the runtime environment.

11. The reproduction method according to claim 2 or 3, further comprising: In response to the target job being a second type of job, when the program process is invoked by a job invocation request, the program process performs data processing on the data corresponding to the job invocation request and returns the data processing result.

12. The reproduction method according to claim 11, further comprising: In response to the target job being a second type of job, at least a portion of the data used by the target job is determined based on the target job identifier, wherein the at least a portion of the data is stored in the database; Specifically, during the execution of the target job, the database unit is invoked to perform corresponding database operations on at least a portion of the data.

13. The reproduction method according to claim 12, wherein, The at least one type of configuration parameter includes data configuration parameters. The data configuration parameters include the data version identifier and data location of at least a portion of the data. Based on the target run identifier, at least a portion of the data used by the target run is determined, including: Based on the data configuration parameters, the data in the database specified by the data version identifier and the data location is determined as at least a portion of the data.

14. The reproduction method according to claim 2, wherein, The configuration parameters of at least one type include the running parameters of the target runtime, which include at least one of the following: the number of graphics processors used to execute the target runtime, the number of central processing units used to execute the target runtime, memory, and the number of program processes. The process of running the target job through the program process includes: The target job is executed through the program process according to the said operating parameters.

15. The reproduction method according to claim 14, wherein, The program process is then started within the runtime environment based on the code file and the environment image, including: The environment image is parsed to obtain build information, and at least one container is built based on the build information; According to the code file, the target running environment is built in the at least one container, and the program process is started in each running environment.

16. The reproduction method according to claim 1, further comprising: The network server records all requests received by the network server, which is configured to receive job requests and send the received job requests to the data processing unit. The job requests include at least one of a job run request and a job call request. The job run request is used to initiate the execution of a job of type 1 or type 2, and the job call request is used to call a currently running job of type 2. The first type of job is a job that cannot be called; the second type of job is a job that is in a callable state during runtime. The audit logs of the data system are determined based on at least some of the recorded requests.

17. The reproduction method according to claim 16, wherein, All requests received by the network server are associated with user information. The audit logs of the data system are determined based on at least a portion of all recorded requests, including: The audit logs of the data system for the target user are determined based on all recorded requests.

18. The reproduction method according to claim 16 or 17, further comprising: Record all requests received by the database unit; All requests recorded include all requests received by the database unit.

19. The reproduction method according to claim 1, wherein, The at least one job corresponds to a job declaration file. The reproduction method further includes: Based on the target run identifier, determine the run information corresponding to the target run, wherein the run information includes job declaration file information and run process parameters specified when submitting the job run request; Obtain the job declaration file corresponding to the job declaration file information, wherein the job declaration file includes at least one type of job configuration parameters, and the job declaration file information is used to indicate the version of the job declaration file; Based on the job configuration parameters of the at least one type and the running process parameters specified when submitting the job run request, determine the configuration information corresponding to the target run; Based on the configuration information corresponding to the target operation, reproduce the target operation.

20. A rerun method, comprising: Displays the job execution records corresponding to at least partially completed runs; Obtain the target run identifier, wherein the target run corresponds to a run record in the job run record, the target run is a run for the target job, and the target run is selected from the job run record; Based on the target run identifier, execute the reproduction method according to any one of claims 1-19 to rerun the target run.

21. A data system, comprising a data processing unit and a database unit, wherein, The data processing unit is configured to run at least one job in the project, and the database unit is configured to perform database operations. The data processing unit is also configured to, in response to a data usage request for the database generated during the execution of the at least one job, invoke the database unit to perform the database operation corresponding to the data usage request. The data processing unit includes: The acquisition module is configured to acquire a target run identifier, wherein the target run identifier is used to indicate a target run for a target job, the target run is in a completed state, and the target run identifier is created when the target run is executed; The determination module is configured to determine the target running code file and the target running environment image based on the target running identifier. The build module is configured to build the runtime environment for the target based on the code file and the environment image, and start the program process in the runtime environment; The processing module is configured to respond to the target job being a first type of job by: determining the data used by the target job based on the target job identifier, running the target job through the program process, wherein the data used by the target job is stored in the database, and calling the database unit to perform corresponding database operations on the data used during the running of the target job.

22. A method for utilizing operational information, applied to the data system as described in claim 21, wherein, The at least one job corresponds to a job declaration file. The utilization method includes: The data processing unit receives a request, wherein the request received by the data processing unit includes a job request, the job request includes a job execution request, the job execution request is used to start the execution of a job, and the job execution request includes the job to be executed, the job declaration file information corresponding to the job to be executed, and the execution process parameters specified when submitting the job execution request. The data processing unit generates operation information corresponding to each operation based on the request received. The operation information includes job declaration file information corresponding to each operation and operation process parameters specified when submitting the job operation request. The operation information is used to reproduce the job or generate an audit log.

23. An electronic device, comprising: The data system as described in claim 21.

24. An electronic device, comprising: processor; Memory, including one or more computer program modules; The one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to implement the method for reproducing the target operation as described in any one of claims 1-19, or the rerun method as described in claim 20, or the method for utilizing operation information as described in claim 22.

25. A storage medium storing non-transitory computer-readable instructions that, when executed by a computer, implement the method for reproducing the target operation as described in any one of claims 1-19, the re-running method as described in claim 20, or the method for utilizing operation information as described in claim 22.

Images