Operation state management method and device of earthquake processing system

Abstract

The invention relates to the technical field of petroleum seismic exploration software engineering, and particularly discloses an operation state management method and device for a seismic processing system, and the method comprises the steps: submitting an operation to a Yarn platform, and enabling the operation to flow according to the operation state of the Yarn platform; the method comprises the following steps: a Pi platform obtains an operation state of circulation of a Yarn platform, and converts the operation state of circulation of the Yarn platform into an operation state of circulation of the Pi platform; and carrying out business processing on the operation state transferred by the Pi platform through the operation management service of the Pi platform. According to the method, after a job is submitted to Yarn, the job state is transferred according to the state of Yarn, pi obtains the job state of the corresponding job, and the job state is converted; different business processing flows are executed according to different operation states and operation management services, so that correct operation in an operation life cycle is ensured, actual production requirements are met, and the stability of cluster operation and the smoothness of business processing are ensured.

Description

Technical Field

[0001] This invention relates to the field of petroleum seismic exploration software engineering technology, specifically to a method and apparatus for managing the operational status of a seismic processing system. Background Technology

[0002] The π platform earthquake processing system is the industry's first earthquake processing system to utilize Hadoop series big data technologies. It overcomes the data I / O bottleneck of traditional file systems in current earthquake data systems. Thanks to Hadoop series big data technologies, data I / O and job execution speeds are more than ten times faster than under traditional file system frameworks. The π platform processing system's job scheduling is based on Yarn. Yarn job states are not suitable for direct use in the processing system; they need to be transformed to form the π platform's own job states, and corresponding business processing is performed based on these job states.

[0003] Therefore, a job status management system based on Yarn is anticipated. This system would allow jobs to be submitted to Yarn, and their statuses on Yarn to be converted into job statuses in the processing system. Simultaneously, corresponding business processes would be performed based on the different job statuses. Currently, no such technological invention is found in existing commercial earthquake processing software.

[0004] Based on this technical background, the present invention studies a method and apparatus for managing the operational status of an earthquake processing system. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method and apparatus for managing the job status of an earthquake processing system. After a job is submitted to Yarn, the job status flows according to Yarn's status flow. π retrieves the corresponding job status and performs the job status transition. For different job statuses, the job management service executes different business processing flows to ensure correct operation throughout the job's lifecycle. Actual production verification has shown that this method can meet practical requirements, ensuring the stability of cluster job operation and the smoothness of business processing.

[0006] To achieve the above objectives, a first aspect of the present invention provides a method for managing the operational status of an earthquake processing system, comprising:

[0007] Submit the job to the Yarn platform and let the job flow according to the job status on the Yarn platform;

[0008] The job status flowing on the Yarn platform is obtained through the π platform, and the job status flowing on the Yarn platform is converted into the job status flowing on the π platform.

[0009] The job management service of the π platform performs business processing on the job status of the jobs flowing through the π platform.

[0010] A second aspect of the present invention provides an operational status management device for an earthquake processing system, comprising:

[0011] The job submission module is used to submit jobs to the Yarn platform and to process the jobs according to their status on the Yarn platform.

[0012] The state transition module is used to obtain the job status flowing on the Yarn platform through the π platform and convert the job status flowing on the Yarn platform to the job status flowing on the π platform.

[0013] The business processing module is used to process the business status of jobs flowing through the π platform through the job management service of the π platform.

[0014] A third aspect of the present invention provides an electronic device, the electronic device comprising:

[0015] Memory, which stores executable instructions;

[0016] A processor that executes the executable instructions in the memory to implement the operation status management method of the earthquake processing system described in the first aspect.

[0017] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the operation status management method of the earthquake processing system described in the first aspect.

[0018] The beneficial effects of this invention include:

[0019] (1) The operation status management method of the earthquake processing system proposed in this invention, after the operation is submitted to Yarn, the operation status flows according to the Yarn status flow, π obtains the operation status of the corresponding operation and performs the operation status conversion; for different operation statuses, the operation management service executes different business processing flows to ensure the correct operation within the operation life cycle. After actual production verification, this method can meet the actual requirements, ensure the stability of cluster operation and the smoothness of business processing.

[0020] (2) The operation status management method of the earthquake processing system proposed in this invention has completed the design and implementation of the operation status flow of the π platform. When multiple projects and multiple operations are running at the same time, the operation status flow is stable, correct and smooth, and related business processing is completed, such as output of result data and deletion of temporary files.

[0021] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0022] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings.

[0023] Figure 1 This is a flowchart illustrating the operational status management method for the earthquake processing system proposed in this invention.

[0024] Figure 2 This is a schematic diagram of the operation status display on the π platform in a specific embodiment of the operation status management method of the earthquake processing system proposed in this invention.

[0025] Figure 3 This is a schematic diagram of the operation status display on the Yarn platform, which is a specific implementation of the operation status management method for the earthquake processing system proposed in this invention.

[0026] Figure 4 This is a schematic diagram of the operation status flow on the π platform in a specific implementation of the operation status management method of the earthquake processing system proposed in this invention.

[0027] Figure 5 This is a diagram of the operational status on the π platform, representing a specific implementation of the operational status management method for the earthquake processing system proposed in this invention. Detailed Implementation

[0028] Preferred embodiments of the invention will now be described in more detail. While preferred embodiments of the invention are described below, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0029] This invention provides a method for managing the operational status of an earthquake processing system, such as... Figure 1 As shown, it includes:

[0030] Submit the job to the Yarn platform and let the job flow according to the job status on the Yarn platform;

[0031] The job status flowing on the Yarn platform is obtained through the π platform, and the job status flowing on the Yarn platform is converted into the job status flowing on the π platform.

[0032] The job management service on the π platform is used to process the status of jobs circulating on the π platform.

[0033] In this invention, after a job is submitted to Yarn, the job status flows according to the Yarn status flow. π obtains the job status of the corresponding job and performs job status conversion. For different job statuses, the job management service executes different business processing flows to ensure the correct operation throughout the job lifecycle. After actual production verification, this method can meet the actual requirements, ensure the stability of cluster job operation, and the smoothness of business processing.

[0034] According to the present invention, the job status of the Yarn platform includes:

[0035] NEW represents the state when an application is newly created. In this state, the application is assigned a unique Application ID, has not been allocated resources, and has not entered the resource queue.

[0036] NEW_SAVING is the state in which the application is waiting for resources to be saved. This state only exists on clusters where Application history saving is enabled. If no history is saved, this state transition will not occur.

[0037] SUBMITTED indicates that the application has been submitted to the Yarn platform and is waiting in the queue for scheduling resources. In this state, the Yarn platform has only performed preliminary runtime configuration on the application and has not allocated any containers to the application.

[0038] ACCEPTED indicates that the application has passed through the queue and has been allocated the initial minimum container it needs;

[0039] RUNNING indicates that the application is running and has a running container.

[0040] FINISHED indicates that the application has successfully completed and its final state has been saved to the application history of the Yarn platform.

[0041] FAILED indicates that the application has failed to run and its final state has been saved to the application history in the Yarn platform.

[0042] KILLED means that the application has been terminated and its final state has been saved to the Yarn platform application history.

[0043] SUSPENDED indicates that the job is paused.

[0044] According to the present invention, the operating status of the π platform includes:

[0045] NOT_SUBMITTED indicates that the job has been saved to the database but has not yet been submitted to the cluster.

[0046] SUBMITTING represents the state of a MR job slice submitted to the cluster but not yet started running, or the state of a Spark job execution command line submission.

[0047] SUBMITTED indicates that the job slice has been successfully submitted to the cluster but has not yet started running.

[0048] PENDING indicates a state where a job has been submitted but has not yet been allocated computing resources for computation.

[0049] RUNNING indicates that the job is currently running.

[0050] SUCCEEDED means the job has been successfully completed and its final state has been saved to the application history of the π platform.

[0051] FAILED indicates that the job failed to run and its final state has been saved to the application history of the π platform.

[0052] KILLED means the job has been terminated and its final state has been saved to the application history of the π platform.

[0053] FAILING indicates an intermediate state where the job has failed, but is not yet completely in a failed state.

[0054] KILLING indicates that the task is in the killing stage, but has not yet been completely killed.

[0055] SUSPENDED indicates that the job is paused.

[0056] According to the present invention, the correspondence between the job status of the Yarn platform and the job status of the π platform includes:

[0057] NEW, NEW_SAVING, SUBMITTED, and ACCEPTED on the Yarn platform correspond to PENDING on the π platform.

[0058] The RUNNING on the Yarn platform corresponds to the RUNNING on the π platform;

[0059] The FINISHED on the Yarn platform corresponds to the SUCCEEDED on the π platform.

[0060] FAILED on the Yarn platform corresponds to FAILED on the π platform;

[0061] KILLED on the Yarn platform corresponds to KILLED on the π platform;

[0062] The SUSPENDED on the Yarn platform corresponds to the SUSPENDED on the π platform.

[0063] Preferably, the processing services corresponding to the operating status of the π platform include:

[0064] The processing business corresponding to NOT_SUBMITTED is that the job is submitted from the interface, the job information is saved in JSON format, the job-related information is saved in the database, and the job is displayed on the interface before the job submission action is executed.

[0065] The processing business corresponding to SUBMITTING is the job management service of the π platform. It retrieves jobs that have not yet been submitted from the database and starts the job submission process. This process involves submitting the job via command line until the final submission result is returned.

[0066] The processing logic corresponding to SUBMITTED indicates that the job has completed the job submission process.

[0067] However, we have not yet obtained any information about the job from the Yarn platform;

[0068] The processing business corresponding to PENDING is that the job can be found on the Yarn platform, has not yet been allocated computing resources, and has begun computing.

[0069] The processing task corresponding to RUNNING is that computing resources have been allocated to the job, and computation has begun.

[0070] Preferably, the business processing of the job status transferred on the π platform through the job management service of the π platform includes:

[0071] When a job transitions from NOT_SUBMITTED to SUBMITTING status, the job is submitted via command line until the final result of the submission is returned. If the status is Failed, the final status of the job in the database is updated to FAILED.

[0072] When a job changes from SUBMITTING to SUBMITED, update the job's status in the database to SUBMITED and complete the job submission on the job's command line.

[0073] When a job changes from SUBMITED to PENDDING, update the job's status in the database and wait for the Yarn platform to allocate resources and perform computation.

[0074] When a job changes from PENDDING to RUNNING status, update the job's status in the database, create a temporary directory dedicated to the job, and provide a temporary file for the job to write to.

[0075] According to the present invention, the business processing of the job status transferred on the π platform through the job management service of the π platform further includes:

[0076] When the job changes from RUNNING to SUCCEEDED, update the job's status in the database, delete the temporary directory and all temporary files in the directory, write the result data, update the database status of the result data, and make the data visible in the interactive interface.

[0077] When a job changes from RUNNING to FAILING, update the job's status in the database and delete the temporary directory and all temporary files in the directory.

[0078] When a job changes from RUNNING to KILLING status, update the job's status in the database and delete the temporary directory and all temporary files in the directory.

[0079] When a job changes from RUNNING to FAILED status, update the job's status in the database;

[0080] When a job changes from RUNNING to KILLED status, update the job's status in the database.

[0081] In this invention, the design and implementation of the job status transition on the π platform were completed. When multiple projects and multiple jobs are running simultaneously, the job status transition is stable, correct, and smooth, and related business processes such as output of results data and deletion of temporary files are completed.

[0082] The present invention will be described in more detail below through embodiments.

[0083] Example 1:

[0084] This embodiment proposes a method for managing the operational status of an earthquake processing system, specifically including:

[0085] Processing jobs are submitted and run in Yarn, following Yarn's lifecycle management; the Yarn job lifecycle is as follows:

[0086] The state of a newly created application. The application is assigned a unique Application ID, but no resources have been allocated, nor has it entered the resource queue;

[0087] NEW_SAVING: The application is waiting for resources to be saved. This state only exists on clusters where application history saving is enabled. If no history is saved, this state transition will not occur.

[0088] The SUBMITTED application has been submitted to YARN and is waiting in the queue for scheduling resources; in this state, YARN has only performed preliminary runtime configuration of the application, but has not yet allocated any containers to the application.

[0089] ACCEPTED means the application has been processed through the queue and has been allocated the initial and minimum containers it needs.

[0090] RUNNING: The application is running and has a running container.

[0091] FINISHED means the application has completed successfully and its final state has been saved to the YARN application history.

[0092] FAILED: The application has failed to run and its final state has been saved to the YARN application history.

[0093] KILLED means the application has been terminated and its final state has been saved to the YARN application history.

[0094] The corresponding job status in π is as follows:

[0095] NOT_SUBMITTED: The job has been saved to the database but has not yet been submitted to the cluster;

[0096] SUBMITTING: The MR job slicing process, submitted to the cluster, but not yet started running; or the Spark job command-line submission process.

[0097] SUBMITTED: The job slicing was successful and submitted to the cluster, but has not yet started running;

[0098] PENDING: The job has been submitted but has not yet been allocated computing resources for computation; RUNNING: The job is currently running.

[0099] SUCCEEDED: The job has been successfully completed and its final state has been saved to the π application history.

[0100] FAILED: The job failed to run and its final state has been saved to the π application history;

[0101] KILLED: has been terminated, and its final state has been saved to the π application history;

[0102] FAILING: The job has failed in an intermediate state; it is not yet fully in a failure state.

[0103] KILLING: The task is in the killing stage, but has not yet been completely killed.

[0104] In this embodiment, the job status on the π platform is displayed as follows: Figure 2 As shown, the job status on the Yarn platform is displayed as follows: Figure 3 As shown in Table 1, the correspondence between the job status of the Yarn platform and the job status of the π platform is as follows;

[0105] Table 1. Correspondence between job status on the Yarn platform and job status on the π platform.

[0106]

[0107] In this embodiment, the services processed by various job states in π include:

[0108] NOT_SUBMITTED: The job is submitted from the interface. The job information is saved in JSON format. The job-related information is saved in the database. Before the job submission action is executed, the state displayed on the interface is NOT_SUBMITTED.

[0109] SUBMITTING: π's job management service retrieves unsubmitted jobs from the database and begins the job submission process. This process involves submitting the job via command line until the final submission result (failure or success) is returned.

[0110] SUBMITTED: The assignment has been submitted, but the assignment information has not yet been retrieved from Yarn.

[0111] PENDING: The job can be found in Yarn, but computing resources have not yet been allocated. Computation has begun.

[0112] RUNNING: The job has been allocated computing resources and has begun computation.

[0113] In this embodiment, the job status transition on the π platform is as follows: Figure 4 As shown, the job status on the π platform is as follows: Figure 5 As shown, changes in job status are accompanied by the processing of corresponding business processes; the diagram shows the transitions in status throughout the job lifecycle.

[0114] When a job transitions from NOT_SUBMITTED to SUBMITTING status, the job is submitted via command line until the final result (failure or success) is returned. When the status is failure, the final status of the job in the database is updated to FAILED.

[0115] When a job changes from SUBMITTING to SUBMITED, update the job's status in the database to SUBMITED, and complete the job submission on the job's command line; the job has been submitted to Yarn.

[0116] When a job changes from SUBMITED to PENDDING, update the job's status in the database and wait for Yarn to allocate resources and perform computations.

[0117] When a job changes from PENDDING to RUNNING status, the job's status in the database is updated, and the job creates its own temporary directory for writing temporary files.

[0118] When a job changes from RUNNING to SUCCEEDED, the job's status in the database is updated. The job deletes the temporary directory and all temporary files in the directory, writes the output data, and updates the database status of the output data. The data is visible in the interactive interface.

[0119] When a job changes from RUNNING to FAILING, the job's status in the database is updated, and the job deletes the temporary directory and all temporary files in the directory.

[0120] When a job changes from RUNNING to KILLING status, the job's status in the database is updated, and the job deletes the temporary directory and all temporary files in the directory.

[0121] When a job changes from RUNNING to FAILED status, update the job's status in the database;

[0122] When a job changes from RUNNING to KILLED status, update the job's status in the database.

[0123] Example 2:

[0124] This embodiment provides a method for managing the operational status of an earthquake processing system, such as... Figure 1 As shown, it includes:

[0125] Submit the job to the Yarn platform and let the job flow according to the job status on the Yarn platform;

[0126] The job status flowing on the Yarn platform is obtained through the π platform, and the job status flowing on the Yarn platform is converted into the job status flowing on the π platform.

[0127] The job management service of the π platform is used to process the status of jobs circulating on the π platform.

[0128] The job statuses processed on the Yarn platform include:

[0129] NEW represents the state when an application is newly created. In this state, the application is assigned a unique Application ID, has not been allocated resources, and has not entered the resource queue.

[0130] NEW_SAVING is the state in which the application is waiting for resources to be saved. This state only exists on clusters where Application history saving is enabled. If no history is saved, this state transition will not occur.

[0131] SUBMITTED indicates that the application has been submitted to the Yarn platform and is waiting in the queue for scheduling resources. In this state, the Yarn platform has only performed preliminary runtime configuration on the application and has not allocated any containers to the application.

[0132] ACCEPTED indicates that the application has passed through the queue and has been allocated the initial minimum container it needs;

[0133] RUNNING indicates that the application is running and has a running container.

[0134] FINISHED indicates that the application has successfully completed and its final state has been saved to the application history of the Yarn platform.

[0135] FAILED indicates that the application has failed to run and its final state has been saved to the application history in the Yarn platform.

[0136] KILLED means that the application has been terminated and its final state has been saved to the Yarn platform application history.

[0137] SUSPENDED indicates that the job is paused.

[0138] The operational status of the π platform includes:

[0139] NOT_SUBMITTED indicates that the job has been saved to the database but has not yet been submitted to the cluster.

[0140] SUBMITTING represents the state of a MR job slice submitted to the cluster but not yet started running, or the state of a Spark job execution command line submission.

[0141] SUBMITTED indicates that the job slice has been successfully submitted to the cluster but has not yet started running.

[0142] PENDING indicates a state where a job has been submitted but has not yet been allocated computing resources for computation.

[0143] RUNNING indicates that the job is currently running.

[0144] SUCCEEDED means the job has been successfully completed and its final state has been saved to the application history of the π platform.

[0145] FAILED indicates that the job failed to run and its final state has been saved to the application history of the π platform.

[0146] KILLED means the job has been terminated and its final state has been saved to the application history of the π platform.

[0147] FAILING indicates an intermediate state where the job has failed, but is not yet completely in a failed state.

[0148] KILLING indicates that the task is in the killing stage, but has not yet been completely killed.

[0149] SUSPENDED indicates that the job is paused.

[0150] The correspondence between the job status on the Yarn platform and the job status on the π platform includes:

[0151] NEW, NEW_SAVING, SUBMITTED, and ACCEPTED on the Yarn platform correspond to PENDING on the π platform.

[0152] The RUNNING on the Yarn platform corresponds to the RUNNING on the π platform;

[0153] The FINISHED on the Yarn platform corresponds to the SUCCEEDED on the π platform.

[0154] FAILED on the Yarn platform corresponds to FAILED on the π platform;

[0155] KILLED on the Yarn platform corresponds to KILLED on the π platform;

[0156] The SUSPENDED on the Yarn platform corresponds to the SUSPENDED on the π platform.

[0157] The processing tasks corresponding to the job status of the π platform include:

[0158] The processing business corresponding to NOT_SUBMITTED is that the job is submitted from the interface, the job information is saved in JSON format, the job-related information is saved in the database, and the job is displayed on the interface before the job submission action is executed.

[0159] The processing business corresponding to SUBMITTING is the job management service of the π platform. It retrieves jobs that have not yet been submitted from the database and starts the job submission process. This process involves submitting the job via command line until the final submission result is returned.

[0160] The processing logic corresponding to SUBMITTED indicates that the job has completed the job submission process.

[0161] However, we have not yet obtained any information about the job from the Yarn platform;

[0162] The processing business corresponding to PENDING is that the job can be found on the Yarn platform, has not yet been allocated computing resources, and has begun computing.

[0163] The processing task corresponding to RUNNING is that computing resources have been allocated to the job and computation has begun.

[0164] The business processing of job status via the π platform's job management service includes:

[0165] When a job transitions from NOT_SUBMITTED to SUBMITTING status, the job is submitted via command line until the final result of the submission is returned. If the status is Failed, the final status of the job in the database is updated to FAILED.

[0166] When a job changes from SUBMITTING to SUBMITED, update the job's status in the database to SUBMITED and complete the job submission on the job's command line.

[0167] When a job changes from SUBMITED to PENDDING, update the job's status in the database and wait for the Yarn platform to allocate resources and perform computation.

[0168] When a job changes from PENDDING to RUNNING status, update the job's status in the database, create a temporary directory dedicated to the job, and provide a temporary file for the job to write to.

[0169] When the job changes from RUNNING to SUCCEEDED, update the job's status in the database, delete the temporary directory and all temporary files in the directory, write the result data, update the database status of the result data, and make the data visible in the interactive interface.

[0170] When a job changes from RUNNING to FAILING, update the job's status in the database and delete the temporary directory and all temporary files in the directory.

[0171] When a job changes from RUNNING to KILLING status, update the job's status in the database and delete the temporary directory and all temporary files in the directory.

[0172] When a job changes from RUNNING to FAILED status, update the job's status in the database;

[0173] When a job changes from RUNNING to KILLED status, update the job's status in the database.

[0174] Example 3:

[0175] This embodiment provides an operational status management device for an earthquake processing system, including:

[0176] The job submission module is used to submit jobs to the Yarn platform and to manage the job flow according to the job status on the Yarn platform.

[0177] The state transition module is used to obtain the job status flowing on the Yarn platform through the π platform and convert the job status flowing on the Yarn platform to the job status flowing on the π platform.

[0178] The business processing module is used to process the status of jobs circulating on the π platform through the job management service of the π platform.

[0179] The job statuses processed on the Yarn platform include:

[0180] NEW represents the state when an application is newly created. In this state, the application is assigned a unique Application ID, has not been allocated resources, and has not entered the resource queue.

[0181] NEW_SAVING is the state in which the application is waiting for resources to be saved. This state only exists on clusters where Application history saving is enabled. If no history is saved, this state transition will not occur.

[0182] SUBMITTED indicates that the application has been submitted to the Yarn platform and is waiting in the queue for scheduling resources. In this state, the Yarn platform has only performed preliminary runtime configuration on the application and has not allocated any containers to the application.

[0183] ACCEPTED indicates that the application has passed through the queue and has been allocated the initial minimum container it needs;

[0184] RUNNING indicates that the application is running and has a running container.

[0185] FINISHED indicates that the application has successfully completed and its final state has been saved to the application history of the Yarn platform.

[0186] FAILED indicates that the application has failed to run and its final state has been saved to the application history in the Yarn platform.

[0187] KILLED means that the application has been terminated and its final state has been saved to the Yarn platform application history.

[0188] SUSPENDED indicates that the job is paused.

[0189] The operational status of the π platform includes:

[0190] NOT_SUBMITTED indicates that the job has been saved to the database but has not yet been submitted to the cluster.

[0191] SUBMITTING represents the state of a MR job slice submitted to the cluster but not yet started running, or the state of a Spark job execution command line submission.

[0192] SUBMITTED indicates that the job slice has been successfully submitted to the cluster but has not yet started running.

[0193] PENDING indicates a state where a job has been submitted but has not yet been allocated computing resources for computation.

[0194] RUNNING indicates that the job is currently running.

[0195] SUCCEEDED means the job has been successfully completed and its final state has been saved to the application history of the π platform.

[0196] FAILED indicates that the job failed to run and its final state has been saved to the application history of the π platform.

[0197] KILLED means the job has been terminated and its final state has been saved to the application history of the π platform.

[0198] FAILING indicates an intermediate state where the job has failed, but is not yet completely in a failed state.

[0199] KILLING indicates that the task is in the killing stage, but has not yet been completely killed.

[0200] SUSPENDED indicates that the job is paused.

[0201] The correspondence between the job status on the Yarn platform and the job status on the π platform includes:

[0202] NEW, NEW_SAVING, SUBMITTED, and ACCEPTED on the Yarn platform correspond to PENDING on the π platform.

[0203] The RUNNING on the Yarn platform corresponds to the RUNNING on the π platform;

[0204] The FINISHED on the Yarn platform corresponds to the SUCCEEDED on the π platform.

[0205] FAILED on the Yarn platform corresponds to FAILED on the π platform;

[0206] KILLED on the Yarn platform corresponds to KILLED on the π platform;

[0207] The SUSPENDED on the Yarn platform corresponds to the SUSPENDED on the π platform.

[0208] The processing tasks corresponding to the job status of the π platform include:

[0209] The processing business corresponding to NOT_SUBMITTED is that the job is submitted from the interface, the job information is saved in JSON format, the job-related information is saved in the database, and the job is displayed on the interface before the job submission action is executed.

[0210] The processing business corresponding to SUBMITTING is the job management service of the π platform. It retrieves jobs that have not yet been submitted from the database and starts the job submission process. This process involves submitting the job via command line until the final submission result is returned.

[0211] The processing logic corresponding to SUBMITTED indicates that the job has completed the job submission process.

[0212] However, we have not yet obtained any information about the job from the Yarn platform;

[0213] The processing business corresponding to PENDING is that the job can be found on the Yarn platform, has not yet been allocated computing resources, and has begun computing.

[0214] The processing task corresponding to RUNNING is that computing resources have been allocated to the job and computation has begun.

[0215] The business processing of job status via the π platform's job management service includes:

[0216] When a job transitions from NOT_SUBMITTED to SUBMITTING status, the job is submitted via command line until the final result of the submission is returned. If the status is Failed, the final status of the job in the database is updated to FAILED.

[0217] When a job changes from SUBMITTING to SUBMITED, update the job's status in the database to SUBMITED and complete the job submission on the job's command line.

[0218] When a job changes from SUBMITED to PENDDING, update the job's status in the database and wait for the Yarn platform to allocate resources and perform computation.

[0219] When a job changes from PENDDING to RUNNING status, update the job's status in the database, create a temporary directory dedicated to the job, and provide a temporary file for the job to write to.

[0220] When the job changes from RUNNING to SUCCEEDED, update the job's status in the database, delete the temporary directory and all temporary files in the directory, write the result data, update the database status of the result data, and make the data visible in the interactive interface.

[0221] When a job changes from RUNNING to FAILING, update the job's status in the database and delete the temporary directory and all temporary files in the directory.

[0222] When a job changes from RUNNING to KILLING status, update the job's status in the database and delete the temporary directory and all temporary files in the directory.

[0223] When a job changes from RUNNING to FAILED status, update the job's status in the database;

[0224] When a job changes from RUNNING to KILLED status, update the job's status in the database.

[0225] Example 4:

[0226] This invention provides an electronic device including a memory and a processor.

[0227] Memory, which stores executable instructions;

[0228] The processor executes executable instructions in memory to implement the operational status management method of the earthquake processing system.

[0229] This memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0230] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of the invention, the processor is used to execute computer-readable instructions stored in the memory.

[0231] Those skilled in the art should understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this invention.

[0232] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0233] Example 5:

[0234] This invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for managing the operational status of an earthquake processing system.

[0235] A computer-readable storage medium according to embodiments of the present invention stores non-transitory computer-readable instructions. When these non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the methods described in the foregoing embodiments of the present invention are performed.

[0236] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).

[0237] The operation status management method for the earthquake processing system proposed in the embodiments of the present invention involves submitting an operation to Yarn, and then the operation status flow follows the Yarn status flow. π obtains the operation status of the corresponding operation and performs operation status conversion. For different operation statuses, the operation management service executes different business processing flows to ensure the correct operation throughout the operation lifecycle. After actual production verification, this method can meet the actual requirements, ensure the stability of cluster operation and the smoothness of business processing.

[0238] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A method for managing the operational status of an earthquake processing system, characterized in that, include: Submit the job to the Yarn platform and let the job flow according to the job status on the Yarn platform; The job status flowing on the Yarn platform is obtained through the π platform, and the job status flowing on the Yarn platform is converted into the job status flowing on the π platform. The job management service of the π platform performs business processing on the job status of the jobs flowing through the π platform.

2. The method according to claim 1, characterized in that, The job statuses processed on the Yarn platform include: NEW represents the state when an application is newly created. In this state, the application is assigned a unique Application ID, has not been allocated resources, and has not entered the resource queue. NEW_SAVING is the state in which the application is waiting for resources to be saved. This state only exists on clusters where Application history saving is enabled. If no history is saved, this state transition will not occur. SUBMITTED indicates that the application has been submitted to the Yarn platform and is waiting in the queue for scheduling resources. In this state, the Yarn platform has only performed preliminary runtime configuration on the application and has not allocated any containers to the application. ACCEPTED indicates that the application has passed through the queue and has been allocated the initial minimum container it needs; RUNNING indicates that the application is running and has a running container. FINISHED means that the application has been successfully completed and its final state has been saved to the application history of the Yarn platform; FAILED indicates that the application has failed to run and its final state has been saved to the application history of the Yarn platform. KILLED means that the application has been terminated and its final state has been saved to the Yarn platform application history; SUSPENDED indicates that the job is paused.

3. The method according to claim 2, characterized in that, The operational status of the π platform includes: NOT_SUBMITTED indicates that the job has been saved to the database but has not yet been submitted to the cluster. SUBMITTING represents the state of a MR job slice submitted to the cluster but not yet started running, or the state of a Spark job execution command line submission. SUBMITTED indicates that the job slice has been successfully submitted to the cluster but has not yet started running. PENDING indicates a state where a job has been submitted but has not yet been allocated computing resources for computation. RUNNING indicates that the job is currently running. SUCCEEDED means the job has been successfully completed and its final state has been saved to the application history of the π platform. FAILED indicates that the job failed to run and its final state has been saved to the application history of the π platform. KILLED means the job has been terminated and its final state has been saved to the application history of the π platform. FAILING indicates an intermediate state where the job has failed, but is not yet completely in a failed state. KILLING indicates that the task is in the killing stage, but has not yet been completely killed. SUSPENDED indicates that the job is paused.

4. The method according to claim 3, characterized in that, The correspondence between the job status of the Yarn platform and the job status of the π platform includes: The NEW, NEW_SAVING, SUBMITTED, and ACCEPTED of the Yarn platform correspond to the PENDING of the π platform. The RUNNING of the Yarn platform corresponds to the RUNNING of the π platform; The FINISHED of the Yarn platform corresponds to the SUCCEEDED of the π platform. The FAILED of the Yarn platform corresponds to the FAILED of the π platform; The KILLED of the Yarn platform corresponds to the KILLED of the π platform; The SUSPENDED of the Yarn platform corresponds to the SUSPENDED of the π platform.

5. The method according to claim 3, characterized in that, The processing services corresponding to the operational status of the π platform include: The processing business corresponding to NOT_SUBMITTED is that the job is submitted from the interface, the job information is saved in JSON format, the job-related information is saved in the database, and the job is displayed on the interface before the job submission action is executed. The processing business corresponding to SUBMITTING is the job management service of the π platform. It retrieves jobs that have not yet been submitted from the database and starts the job submission process. This process involves submitting the job via command line until the final submission result is returned. The processing business corresponding to SUBMITTED is that the job has completed the job submission action, but has not yet obtained relevant job information from the Yarn platform; The processing business corresponding to PENDING is that the job can be found on the Yarn platform, has not yet been allocated computing resources, and has begun computing. The processing task corresponding to RUNNING is that computing resources have been allocated to the job, and computation has begun.

6. The method according to claim 3, characterized in that, The business processing of the job status transferred through the π platform via the π platform's job management service includes: When a job transitions from NOT_SUBMITTED to SUBMITTING status, the job is submitted via command line until the final result of the submission is returned. If the status is Failed, the final status of the job in the database is updated to FAILED. When a job changes from SUBMITTING to SUBMITED, update the job's status in the database to SUBMITED and complete the job submission on the job's command line. When a job changes from SUBMITED to PENDDING, update the job's status in the database and wait for the Yarn platform to allocate resources and perform computation. When a job changes from PENDDING to RUNNING status, update the job's status in the database, create a temporary directory dedicated to the job, and provide a temporary file for the job to write to.

7. The method according to claim 6, characterized in that, The business processing of the job status transferred through the π platform via the job management service also includes: When the job changes from RUNNING to SUCCEEDED, update the job's status in the database, delete the temporary directory and all temporary files in the directory, write the result data, update the database status of the result data, and make the data visible in the interactive interface. When a job changes from RUNNING to FAILING, update the job's status in the database and delete the temporary directory and all temporary files in the directory. When a job changes from RUNNING to KILLING status, update the job's status in the database and delete the temporary directory and all temporary files in the directory. When a job changes from RUNNING to FAILED status, update the job's status in the database; When a job changes from RUNNING to KILLED status, update the job's status in the database.

8. A device for managing the operational status of an earthquake processing system, characterized in that, include: The job submission module is used to submit jobs to the Yarn platform and to process the jobs according to their status on the Yarn platform. The state transition module is used to obtain the job status flowing on the Yarn platform through the π platform and convert the job status flowing on the Yarn platform to the job status flowing on the π platform. The business processing module is used to process the business status of jobs flowing through the π platform through the job management service of the π platform.

9. An electronic device, characterized in that, The electronic device includes: Memory, which stores executable instructions; A processor that executes the executable instructions in the memory to implement the operation status management method of the earthquake processing system according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the operation status management method of the earthquake processing system according to any one of claims 1-7.

Images