Processing method and device for data processing activity, electronic equipment and storage medium
By acquiring and analyzing information about data jobs and database operations, determining target jobs and performing scheduling intervention, the problem of difficulty in effectively scheduling data jobs and database operations in the prior art is solved, and the efficient and stable operation of the data job system is achieved.
Patent Information
- Application Number
- CN202411729375.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2025-04-25
AI Technical Summary
In the digital warehouse scenario, it is difficult for the prior art to effectively schedule and intervene in data operations and corresponding data operations on the database, resulting in the operational efficiency of the data operation system being affected.
By obtaining the job information of the data job and the status information of the database data operation, the target job matching the data operation is determined, and the scheduling intervention process is performed based on the information of the target job.
It realizes efficient scheduling intervention on data operations and data operations performed by data operations, ensuring the efficient and stable operation of the data operation system.
Smart Images
Figure CN119883531A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for processing data processing activities. Background Art
[0002] In the data warehouse scenario, the data processing activities of the data operation system can be divided according to the resource facilities where they are located, and can include: data operations running on the execution machine and data operations performed on the database. The scheduling system calls up the data job on an execution machine controlled by the scheduling system based on the job configuration and job logic dependencies. When the data job program is running, it calls the database client program on the execution machine to connect to the database to perform data operations (such as executing SQL statements or database commands). If the execution is successful, the data job is completed. In order to make full use of database resources, in terms of architecture, data business processing of multiple application systems is usually arranged to share the same data warehouse. This poses two challenges to the operation and maintenance of the data warehouse: 1. Each system business field has its own timeliness requirements. 2. The processing performance of the data job has not been fully optimized.
[0003] When it is necessary to urgently complete certain tasks, or when individual SQL statements take too long to run, occupy resources, or block key tasks, operation and maintenance technicians are required to perform scheduling intervention, that is, stop and suspend inefficient and low-priority tasks, free up resources, and prioritize key tasks. The problem is: on the one hand, the job scheduling system generally does not record the running status of the current data operation of the data job on the database, and lacks the intervention function of "database execution of data operations". On the other hand, during manual intervention, although the database provides the running status information of the data operation for diagnosis, the number of data jobs is huge, and it is difficult to locate the data job to which a certain data operation belongs, and it is impossible to determine whether the corresponding job belongs to the business scope that should be intervened.
[0004] In the existing related technologies, when multiple business fields are involved and a large number of data operations are carried out in parallel, it is impossible to effectively schedule and intervene in the data operations and the corresponding data operations on the database, which affects the operating efficiency of the data operation system. Summary of the invention
[0005] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.
[0006] To this end, the first purpose of the present disclosure is to propose a method for processing data processing activities, which effectively schedules and intervenes in data operations and data operations performed by data operations, so as to enable the data operation system to run efficiently and stably.
[0007] The second objective of the present disclosure is to provide a processing device for data processing activities.
[0008] A third objective of the present disclosure is to provide an electronic device.
[0009] A fourth objective of the present disclosure is to provide a computer-readable storage medium.
[0010] A fifth object of the present disclosure is to provide a computer program product.
[0011] To achieve the above-mentioned purpose, the first aspect of the present disclosure proposes a method for processing data processing activities, including: obtaining job information of a candidate data job, wherein the data job runs on an execution machine, and the data job calls the service of a database to perform data operations; obtaining status information of the data operations in operation on the database; determining a target job that matches the data operation from multiple candidate data jobs based on the job information and status information; and processing the target job and the data operations performed by the target job based on the job information and / or status information of the target job.
[0012] To achieve the above-mentioned purpose, the second aspect of the present disclosure proposes a processing device for data processing activities, including: a first acquisition module, used to obtain job information of a candidate data job, wherein the data job runs on an execution machine, and the data job calls the service of a database to perform data operations; a second acquisition module, used to obtain status information of data operations in operation on the database; a determination module, used to determine a target job that matches the data operation from multiple candidate data jobs based on the job information and status information; and a processing module, used to process the target job and the data operations performed by the target job based on the job information and / or status information of the target job.
[0013] To achieve the above-mentioned purpose, the third aspect embodiment of the present disclosure proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement a processing method for data processing activities proposed in the first aspect embodiment of the present disclosure.
[0014] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present disclosure proposes a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by the processor, they are used to implement a processing method for data processing activities proposed in the first aspect embodiment of the present disclosure.
[0015] To achieve the above-mentioned purpose, the fifth aspect of the present disclosure proposes a computer program product, including a computer program, which, when executed by a processor, implements a method for processing data processing activities proposed in the first aspect of the present disclosure.
[0016] The present disclosure provides a processing method, device, electronic device, non-transient computer-readable storage medium storing computer instructions, and computer program product for data processing activities, by obtaining the job information of a candidate data job, wherein the data job runs on an execution machine, and the data job calls the service of a database to perform a data operation; obtaining the status information of the data operation in operation on the database; determining the target job matching the data operation from multiple candidate data jobs according to the job information and status information; and processing the target job and the data operation performed by the target job according to the job information and / or status information of the target job. Since it is not only possible to determine the target job matching the data operation, but also to support the processing of the target job and the data operation performed by the target job, that is, it is possible to locate and trace the target job to which the data operation belongs according to the running data operation, and then support the completion of the scheduling and intervention processing of the target job and the data operation, thereby realizing efficient scheduling and intervention of the data job and the data operation performed by the data job, thereby ensuring the efficient and stable operation of the data operation system.
[0017] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0019] Figure 1 is a schematic diagram of an execution machine controlled by a scheduling system in an embodiment of the present disclosure;
[0020] Figure 2 A flowchart of a method for processing data processing activities provided by an embodiment of the present disclosure;
[0021] Figure 3 is a schematic diagram of a process tree in an embodiment of the present disclosure;
[0022] Figure 4 A flowchart of another method for processing data processing activities provided by an embodiment of the present disclosure;
[0023] Figure 5 is a schematic diagram of the structure of a processing device for data processing activities in an embodiment of the present disclosure;
[0024] Figure 6 A schematic diagram of the structure of a processing device for data processing activities provided by an embodiment of the present disclosure;
[0025] Figure 7It is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0026] The embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and are not to be construed as limitations on the present disclosure. The technical methods of the present disclosure are applied to related fields, and the technical application environments and scenarios described in the present disclosure (such as "data warehouse", "data operation system", "scheduling system") are also intended to be used to explain the present disclosure, and are not to be construed as limitations on the present disclosure.
[0027] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. The acquisition, transmission, storage, use and processing of data in the technical solution of this disclosure are in compliance with the relevant provisions of national laws and regulations.
[0028] It should be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.
[0029] The general technical terms involved or may be involved in this disclosure are listed below, and can be understood by referring to the technical standards in the relevant fields:
[0030] SQL: Structured Query Language.
[0031] IP: Internet Protocol.
[0032] IP address: Network address, which is used by networked devices to connect and access each other on the network.
[0033] The embodiments of the present disclosure can be specifically applied in a scheduling system, such as Figure 1 As shown, Figure 1Schematic diagram of the execution machine controlled by the scheduling system in the embodiment of the present disclosure. Among them, the database system, as a resource facility for carrying data processing activities, provides database client programs for respective products. The database client program submits the SQL statements and commands transmitted by the job program to the database system for execution through the execution machine port and returns the execution result. The configuration of a data job (a data job can also be called a data processing job) usually includes a job name and a corresponding job execution command, and the command contains the job program file path and command parameters. The scheduling system executes the command on the execution machine, which calls up the data job, and thus generates a process corresponding to the command on the execution machine. In order to make full use of database resources, the scheduling system will call up multiple data jobs in parallel, and thus there are multiple corresponding data job processes. In the related art, since the job scheduling system lacks the intervention function of "database execution data operation", other intervention methods such as manual methods are not efficient enough to cope with the huge number of jobs, and it is difficult to timely locate the data jobs and business areas to which the full database data operation belongs. In the case of involving multiple business areas and a large number of data jobs in parallel, it is impossible to effectively perform batch scheduling intervention on data jobs and corresponding data operations on the database, thereby affecting the operation of the data operation system.
[0034] In order to solve the above-mentioned problems, the embodiments of the present disclosure provide a method for processing data processing activities, which can effectively schedule and intervene in data operations and data operations performed by data operations, thereby ensuring efficient and stable operation of the data operation system.
[0035] The method for processing data processing activities in the embodiments of the present disclosure may include: data operations and corresponding data operations, which are not limited.
[0036] Figure 2 A flowchart of a method for processing data processing activities provided in an embodiment of the present disclosure.
[0037] This embodiment takes the example that the processing method for data processing activities is configured in the processing device for data processing activities. In this embodiment, the processing method for data processing activities can be configured in the processing device for data processing activities. The processing device for data processing activities can be set in a server or in an electronic device, and there is no limitation to this.
[0038] This embodiment takes the case where the method for processing data processing activities is configured in an electronic device as an example, wherein the electronic device is a hardware device with a signal processing function, such as a computer, a smart phone, a tablet computer, a personal digital assistant, an e-book, an integrated circuit, etc.
[0039] It should be noted that the execution subject of the embodiment of the present disclosure may be, for example, a central processing unit (CPU) in a server or electronic device in terms of hardware, or may be, for example, a related background service in a server or electronic device in terms of software, without limitation.
[0040] like Figure 2 As shown, the method for processing data processing activities includes:
[0041] S201: Obtain job information of a candidate data job, wherein the data job runs on an execution machine and calls a service of a database to perform a data operation.
[0042] The execution machine refers to the device, component, etc. used to execute data jobs. The scheduling system executes the job execution command on the execution machine to call up the data job, and thus generates a process corresponding to the command on the execution machine. The process running on the execution machine can be called the process of the data job. The process of the data job can call the database service to perform data operations during operation. The objects, scope of influence, specific operation methods and contents of the data operation are the specific implementation of the business program logic of the data job on the database data.
[0043] The job information is used to describe the situation of the data job that is a candidate.
[0044] Optionally, in some embodiments, the content of the job information includes at least one of the following: execution machine address, job name, job command line, job process number, job running time, and execution machine port number. Thus, the situation of the candidate data job can be accurately and effectively characterized, supporting the accuracy of the implementation of intervention scheduling.
[0045] Optionally, in some embodiments, when obtaining the job information of a candidate data job, it can be filtered on demand: that is, the job information of a candidate data job running on any execution machine among multiple execution machines can be obtained; that is, the job information of any job among multiple candidate data jobs can be obtained; that is, any or all of the job information can be obtained; the above job information of the candidate data job obtained through on-demand filtering can be used to trace the target job from which the data operation originated, without limitation.
[0046] Optionally, in some embodiments, obtaining the job information of the data jobs that are candidates and are running on each execution machine may be achieved by executing some system commands; or obtaining the job information of the data jobs that are candidates and are running on each execution machine may be achieved by executing some program scripts; or obtaining the job information of the data jobs that are candidates and are running on each execution machine may be achieved in any other possible manner without limitation.
[0047] Optionally, in some embodiments, the data items in the collected job information content may include the IP address of the execution machine (an optional example of the execution machine address), the job name, the job command line, the job process number, the job running time, and the local port number opened by the job process on the execution machine to call the database client (i.e., the execution machine port number). Each data item in the job information may be obtained from the job scheduling system (requiring scheduling system support), or obtained from the execution machine operating system. For example, each data item of the job information is obtained from the execution machine operating system. Specifically, for example, when the execution machine operating system is a Linux operating system and the database client program is a psql program (where psql refers to the client program that comes with the PostgreSQL database), the process number of the psql program and the port number opened by the process (i.e., the execution machine port number) can be obtained based on command method one, i.e., through the lsof command; the psql process number and the process tree in which it is located can be obtained based on command method two, i.e., through the ps command (as shown below) Figure 3 As shown, Figure 3 It is a process tree diagram in the embodiment of the present disclosure), and obtains the process number of the job process on the same tree and its running time, and parses the job name from the command line that initiates the job process, for example, it can be solved by the job program name contained in the command line, or it can be solved by the job name contained in the command line parameters, without limitation.
[0048] S202: Acquire status information of data operations in operation on the database.
[0049] Optionally, in some examples, the data operation includes at least one of the following: logging in and / or connecting to a database, executing an SQL statement, and executing a database command, without limitation.
[0050] The status information is used to describe the status of data operations.
[0051] Optionally, in some embodiments, the content of the status information includes at least one of the following: server process number, client address, client port number, and running time of the data operation. Thus, the data operation situation can be accurately and effectively characterized, supporting the accuracy of the implementation of the intervention scheduling.
[0052] After obtaining the job information of the candidate data job running on the execution machine, the state information of the data operation in operation on the database can also be obtained. For example, the data operation in operation refers to the data operation in operation, such as: the SQL statement in operation.
[0053] Optionally, in some embodiments, in the process of acquiring the status information of the data operation in the running state on the database, it can be implemented by using the running state collection unit, which is not limited.
[0054] Optionally, in some embodiments, a specific data operation is taken as an example of executing an SQL statement: the status information of the SQL statement running on the database can be obtained, and the data items in the content of the status information may include: the DB server process number (PID for short) that executes the SQL statement, the client IP (an optional example of the client address), the client port number, and the SQL running time (an optional example of the running time of the data operation).
[0055] Optionally, in some embodiments, when the database system is PostgreSQL (which may also include its derived database products), each data item in the above status information content may be obtained through the database system table pg_stat_activity, which is not limited.
[0056] Optionally, in some embodiments, obtaining status information of data operations in a running state on the database may be achieved by executing some system commands; or obtaining status information of data operations in a running state on the database may be achieved by executing some program scripts; or obtaining status information of data operations in a running state on the database may be achieved in any other possible manner, without limitation.
[0057] S203: Determine a target job matching the data operation from a plurality of candidate data jobs according to the job information and the status information.
[0058] After obtaining the job information of the candidate data jobs running on the execution machine and obtaining the status information of the data operations running on the database, a target job matching the data operation can be determined from multiple candidate data jobs based on the job information and status information.
[0059] The candidate data job that matches the data operation can be called the target job. The candidate data job meets the match, which means that: the data operation belongs to the candidate data job and is the specific implementation of the business program logic of the job on the database data; the data operation originates from the candidate data job and is triggered by the command and parameters of the data job.
[0060] Optionally, in some embodiments, in the process of determining a target job that matches a data operation from multiple candidate data jobs based on job information and status information, it may be determined whether some data items in the job information are the same as some data items in the status information. If they are the same, the candidate data job is determined to be the target job that matches the data operation; or the determination of the target job that matches the data operation from multiple candidate data jobs may be implemented based on artificial intelligence; or the determination of the target job that matches the data operation from multiple candidate data jobs may be implemented by executing some program scripts, and there is no limitation on this.
[0061] S204: Process the target job and the data operation performed by the target job according to the job information and / or status information of the target job.
[0062] Optionally, in some embodiments, it is possible to determine whether an intervention opportunity is met based on the job information of the target job. If it is determined that the intervention opportunity is met, the target job and the data operations performed by the target job are processed.
[0063] Optionally, in some embodiments, whether the intervention opportunity is met may be determined based on the status information. If it is determined that the intervention opportunity is met, the target job and the data operation performed by the target job are processed.
[0064] Optionally, in some embodiments, it is possible to determine whether the intervention opportunity is met based on the job information and status information of the target job. If it is determined that the intervention opportunity is met, the target job and the data operations performed by the target job are processed.
[0065] Optionally, in some embodiments, the above-mentioned intervention timing may be a custom configuration, and may be specifically personalized according to scheduling intervention requirements and scheduling intervention granularity, and there is no limitation on this.
[0066] Optionally, in some embodiments, processing the target job and data operation includes at least one of the following: stopping the running data operation; exiting the process of the target job; suspending the target job. Thus, it supports improving the flexibility of scheduling intervention and is effectively applicable to personalized scheduling scenarios.
[0067] In this embodiment, by obtaining the job information of the data job as a candidate, wherein the data job runs on the execution machine, and the data job calls the service of the database to perform the data operation; obtaining the status information of the data operation in the running state on the database; determining the target job matching the data operation from multiple data jobs as candidates according to the job information and status information; and processing the target job and the data operation performed by the target job according to the job information and / or status information of the target job. Since it is not only possible to determine the target job matching the data operation, but also to support the processing of the target job and the data operation performed by the target job, that is: it is possible to locate and trace the target job to which the data operation belongs according to the running data operation, and then schedule and intervene the target job and the data operation; thereby achieving efficient scheduling and intervention of the data job and the data operation performed by the data job, thereby ensuring the effective operation of the data operation system.
[0068] Figure 4 A flowchart of another method for processing data processing activities provided in an embodiment of the present disclosure.
[0069] like Figure 4 As shown, the method for processing data processing activities includes:
[0070] S401: Obtain job information of a candidate data job, wherein the data job runs on an execution machine and calls a service of a database to perform a data operation.
[0071] S402: Acquire status information of data operations in operation on the database.
[0072] For the description of S401 - S402 , please refer to the above embodiment, which will not be repeated here.
[0073] S403: Determine the job information that meets the preset matching rule with the state information among the multiple job information as the target job information, and determine the data job to which the target job information belongs as the target job.
[0074] The preset matching rule may be pre-set as needed. For example, the preset matching rule specifies which data items in the job information are the same as which data items in the status information, which means that the corresponding job information and the status information meet the preset matching rule, or the preset matching rule may be any other possible rule, which is not limited.
[0075] After determining the job information that meets the preset matching rule with the status information from the multiple job information, the job information that meets the preset matching rule can be determined as the target job information, and the data job to which the target job information belongs can be determined as the target job.
[0076] Optionally, in some embodiments, a job matching unit may be used to determine job information that meets a preset matching rule with the status information from multiple job information.
[0077] Optionally, in some embodiments, a specific data operation is taken as an example of executing an SQL statement. The above-obtained information (job information and status information) can be matched. If it meets the preset matching rules, the set of the two data items (for example, the set of the target job information and status information of the target job) can be used as a "set of matching data items", which at least includes the identifier of the SQL statement and the identifier of the corresponding job. For example, when the database system is PostgreSQL (including its derived database products), the matching is based on the conditions of "execution machine IP = client IP" and "execution machine port number = client port number"; if the conditions are met, the set of the two data items is used as a "set of matching data items", in which the SQL statement and the job name are in correspondence, that is, the job name corresponding to the SQL statement is obtained.
[0078] S404: Process the target job and the data operation performed by the target job according to the job information and / or status information of the target job.
[0079] Optionally, in some embodiments, it is possible to determine whether an intervention opportunity is met based on the job information of the target job. If it is determined that the intervention opportunity is met, the target job and the data operations performed by the target job are processed.
[0080] Optionally, in some embodiments, whether the intervention opportunity is met may be determined based on the status information. If it is determined that the intervention opportunity is met, the target job and the data operation performed by the target job are processed.
[0081] Optionally, in some embodiments, it is possible to determine whether the intervention opportunity is met based on the job information and status information of the target job. If it is determined that the intervention opportunity is met, the target job and the data operations performed by the target job are processed.
[0082] Optionally, in some embodiments, the above-mentioned intervention timing may be a custom configuration, and may be specifically personalized according to scheduling intervention requirements and scheduling intervention granularity, and there is no limitation on this.
[0083] Optionally, in some embodiments, in the process of processing the target job and the data operation performed by the target job according to the job information and / or status information of the target job, it is possible to determine whether the target job meets the first preset condition based on at least one item of the job information of the target job, and obtain a first result; and / or determine whether the data operation meets the second preset condition based on at least one item of the status information, and obtain a second result, and when the first result is that the target job meets the first preset condition and / or the second result is that the data operation meets the second preset condition, the target job and the data operation performed by the target job are processed. In this way, the accuracy of the intervention timing determination can be effectively improved, and accurate scheduling intervention can be supported.
[0084] That is to say, the above-mentioned intervention timing can be described using the first preset condition and / or the second preset condition.
[0085] Optionally, in some embodiments, the first preset condition includes at least one of the following: resources occupied by the target job are greater than a resource threshold; the business domain to which the target job belongs meets the preset business domain; and the importance of the target job is less than a threshold.
[0086] Optionally, in some embodiments, the second preset condition includes at least one of the following: the running time of the data operation executed by the target job reaches a threshold time.
[0087] When the importance of the above-mentioned target job is less than the threshold, it can also be determined whether there are other data jobs with relatively higher importance as candidates waiting to be processed. If so, it is determined that the intervention timing is met, thereby improving the accuracy of the intervention timing determination and supporting accurate scheduling intervention.
[0088] Optionally, in some embodiments, in the process of processing the target job and the data operation executed by the target job according to the job information and / or status information of the target job, the target job and the data operation executed by the target job may be processed when the second result is that the running time of the data operation executed by the target job reaches the threshold time. In other words, if the running time of the data operation executed by the target job reaches the threshold time, it indicates that there may be an abnormality in the execution of the data operation, and thus, the target job and the data operation executed by the target job can be processed in a timely manner.
[0089] Optionally, in some embodiments, in the process of processing the target job and the data operations performed by the target job according to the job information and / or status information of the target job, it can be determined that the target job has occupied resources according to the target job information, and when the first result is that the resources occupied by the target job are greater than the resource threshold, the target job and the data operations performed by the target job are processed. In other words, if the target job has occupied more resources, it means that the remaining available resources of the scheduling system are less, and if the target job continues to run, it may affect the efficient execution of other data jobs, so that the target job and the data operations performed by the target job can be processed in a timely manner.
[0090] Optionally, in some embodiments, in the process of processing the target job and the data operations performed by the target job according to the job information and / or status information of the target job, the business domain to which the target job belongs may be determined according to the job information of the target job, and when the first result is that the business domain to which the target job belongs meets the preset business domain, the target job and the data operations performed by the target job are processed. The target business domain refers to a business domain that requires scheduling intervention, and the target business domain can be configured as needed. In other words, if the business domain to which the target job belongs is the target business domain, it means that scheduling intervention is required, and thus the target job and data operations can be processed in a timely manner.
[0091] Optionally, in some embodiments, in the process of processing the target job and the data operation performed by the target job according to the job information and / or status information of the target job, the importance of the target job can be determined according to the job information of the target job, and when the first result is that the importance is less than the threshold, the target job and the data operation performed by the target job are processed. When the importance of the above-mentioned target job is less than the threshold, it can also be determined whether there are other data jobs with relatively high importance waiting to be processed. If so, it is determined that the intervention opportunity is met, thereby improving the accuracy of the intervention opportunity determination and supporting the realization of accurate scheduling intervention.
[0092] Optionally, in some embodiments, the above-mentioned processing of the target job and the data operation performed by the target job includes at least one of the following: stopping the running data operation; exiting the process of the target job; suspending the target job, without limitation.
[0093] For example, a specific data operation is used as an example to execute a SQL statement. The above-mentioned set of matching data items (all or part such as: job name, job running time, SQL running time) are substituted into the preset maintenance strategy to decide whether to intervene. For example, in the operating environment of some embodiments, the preset maintenance strategy includes a threshold screening condition of the operating state (such as the running time, etc.), so that the SQL statements and their jobs that may or should be intervened can be screened out according to the operating state. In the operating environment of some embodiments, the preset maintenance strategy includes the job and the currently available resource environment (such as CPU / memory / IO / network bandwidth / cluster / executor) and the screening condition of the threshold, so that the SQL statements and their jobs that may or should be intervened can be screened out according to the resource environment and the threshold. In the operating environment of some embodiments, the preset maintenance strategy includes the job and the business field to which it belongs, so that it can be decided whether to intervene according to the business field to which it belongs. In the operating environment of some embodiments, the preset maintenance strategy includes the job and its importance, so that it can be decided whether to intervene according to the importance. In the operating environment of some embodiments, the preset maintenance strategy can be customized and updated, so that the maintenance strategy can be implemented on demand.
[0094] Optionally, in some embodiments, an execution intervention unit may be used: based on the job information and / or status information of the target job, the target job and the data operation executed by the target job are processed. Taking the specific data operation as the execution of SQL statements as an example, the intervention processing is exemplified: the execution intervention unit intervenes in the data operation on the database based on certain received data items (such as PID). The intervention includes but is not limited to stopping the SQL statement from running: for example, in some embodiments, when the database system is PostgreSQL (including its derived database products), stopping the SQL statement from running is completed by calling the database system function pg_terminate_backend (PID); after the SQL statement stops running, the corresponding target job fails and exits. The execution intervention unit intervenes in the job through the scheduling system interface based on certain received data items (such as job name), and the intervention includes but is not limited to: suspending the job to avoid the job from being improperly called up; for example, in some embodiments, when this unit is implemented in the environment of the Linux operating system, the intervention action is completed by sending a json structure data packet containing "intervention specified job instruction" to the scheduling system through the curl command.
[0095] In this embodiment, not only the target job matching the data operation is determined from multiple candidate data jobs, but also the target job and the data operation executed by the target job are supported for processing: by executing the intervention unit, intervention means are provided for the data operation as the consequence and the data operation as the cause in the data processing activity; thus, the data job and the data operation executed by the data job can be effectively scheduled for intervention, thereby ensuring the performance of the data operation system. Moreover, by configuring the judgment conditions of the preset intervention timing as needed, and then judging whether the intervention timing is met based on the relevant information of the data job or data operation in the data processing activity, it can support the on-demand screening of the target activities to be intervened in multiple dimensions such as the running time, resource occupancy, business field, and importance, thereby improving the flexibility and pertinence of the execution intervention, and effectively adapting to the personalized scheduling intervention scenario.
[0096] like Figure 5 As shown, Figure 5 It is a structural diagram of a processing device for data processing activities in an embodiment of the present disclosure, including: a job status collection unit, a SQL running status collection unit, a job matching unit, and an execution intervention unit.
[0097] The method provided in the embodiments of the present disclosure can quickly and accurately locate which job the problematic SQL statement belongs to when the data operation is to execute an SQL statement. Through the combination of the functions of the various units in the aforementioned embodiment device, intervention actions can be efficiently performed, including: stopping the execution of SQL statements in an automatic background manner and suspending the associated jobs, which can save labor costs and improve efficiency. When the data operation system is running at high load, by quickly, efficiently and accurately executing the "intervention in SQL statements and corresponding jobs" operation, it can be ensured that key jobs have sufficient resources.
[0098] Figure 6 A schematic diagram of the structure of a processing device for data processing activities provided by an embodiment of the present disclosure.
[0099] like Figure 6 As shown, the processing device 60 for the data processing activity includes:
[0100] The first acquisition module 601 is used to acquire job information of a candidate data job, wherein the data job runs on an execution machine and calls a database service to perform a data operation.
[0101] The second acquisition module 602 is used to acquire status information of data operations in operation on the database.
[0102] The determination module 603 is used to determine a target job matching the data operation from a plurality of candidate data jobs according to the job information and the state information.
[0103] The processing module 604 is used to process the target job and the data operation performed by the target job according to the job information and / or status information of the target job.
[0104] It should be noted that the aforementioned explanation of the method for processing data processing activities is also applicable to the processing device for data processing activities in this embodiment, and will not be repeated here.
[0105] In this embodiment, by obtaining the job information of the data job as a candidate, wherein the data job runs on the execution machine, and the data job calls the service of the database to perform the data operation; obtaining the status information of the data operation in the running state on the database; determining the target job matching the data operation from multiple data jobs as candidates according to the job information and status information; and processing the target job and the data operation performed by the target job according to the job information and / or status information of the target job. Since it is not only possible to determine the target job matching the data operation, but also to support the processing of the target job and the data operation performed by the target job, that is: it is possible to locate and trace the target job to which the data operation belongs according to the running data operation, and then schedule and intervene the target job and the data operation; thereby achieving efficient scheduling and intervention of the data job and the data operation performed by the data job, thereby ensuring the effective operation of the data operation system.
[0106] As an example, Figure 7 is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of the present disclosure, such as Figure 7 As shown, the electronic device 700 may further include:
[0107] The memory 710 and the processor 720, and the bus 730 connecting different components (including the memory 710 and the processor 720), the memory 710 stores a computer program, and when the processor 720 executes the program, the processing method for data processing activities of the embodiment of the present disclosure is implemented.
[0108] Bus 730 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.
[0109] The electronic device 700 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 700, including volatile and non-volatile media, removable and non-removable media.
[0110] The memory 710 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 740 and / or cache memory 750. The electronic device 700 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 760 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 730 via one or more data medium interfaces. The memory 710 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present disclosure.
[0111] A program / utility 780 having a set (at least one) of program modules 770 may be stored, for example, in the memory 710, such program modules 770 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 770 generally perform the functions and / or methods of the embodiments described in the present disclosure.
[0112] The electronic device 700 may also communicate with one or more external devices 790 (e.g., keyboard, pointing device, display 791, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 792. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 793. As shown, the network adapter 793 communicates with other modules of the electronic device 700 through a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0113] The processor 720 executes various functional applications and data processing by running the programs stored in the memory 710 .
[0114] It should be noted that the implementation process and technical principles of the electronic device of this embodiment refer to the aforementioned explanation of the processing method for data processing activities in the embodiment of the present disclosure, and will not be repeated here.
[0115] In order to implement the above embodiments, the present disclosure also proposes an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.
[0116] In order to implement the above embodiments, the present disclosure further proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.
[0117] In order to implement the above embodiments, the present disclosure also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.
[0118] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this disclosure shall comply with the relevant laws and regulations and shall not violate public order and good morals.
[0119] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.
[0120] The present disclosure anticipates providing implementation schemes for users to selectively block the use or access of personal information data. That is, the present disclosure anticipates providing hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, risks can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.
[0121] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.
[0122] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0123] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.
[0124] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0125] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0126] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0127] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0128] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.
Claims
1. A method for processing data processing activities, characterized in that: The following steps are involved: Obtaining job information of a data job as a candidate, wherein the data job runs on an execution machine and the data job calls a service of a database to perform a data operation; Acquire status information of data operations in operation on the database; Determining a target job matching the data operation from a plurality of candidate data jobs according to the job information and the state information; and The target job and the data operation performed by the target job are processed according to the job information and / or the status information of the target job.
2. The method according to claim 1, characterized in that The content of the job information includes at least one of the following: execution machine address, job name, job command line, job process number, job running time, and execution machine port number.
3. The method according to claim 1, characterized in that The content of the status information includes at least one of the following: a server process number, a client address, a client port number, and a running time of a data operation.
4. The method according to claim 1, characterized in that: The step of determining a target job matching the data operation from a plurality of candidate data jobs according to the job information and the state information includes: The job information that meets a preset matching rule with the state information among the plurality of job information is determined as the target job information, and the data job to which the target job information belongs is determined as the target job.
5. The method according to claim 1, characterized in that The processing of the target job and the data operation performed by the target job according to the job information and / or the status information of the target job includes: Determine whether the target job meets a first preset condition based on at least one item of the job information of the target job, and obtain a first result; and / or determine whether the data operation meets a second preset condition based on at least one item of the status information, and obtain a second result; When the first result is that the target job meets a first preset condition and / or the second result is that the data operation meets a second preset condition, the target job and the data operation performed by the target job are processed.
6. The method according to claim 5, characterized in that The first preset condition includes at least one of the following: The target job has occupied more resources than the resource threshold; The business domain to which the target operation belongs is consistent with the preset business domain; The importance level of the target job is less than a threshold level.
7. The method according to claim 5, characterized in that The second preset condition includes: The running time of the data operation executed by the target job reaches a threshold time.
8. The method according to claim 5, characterized in that The determining, based on at least one item of the job information of the target job, whether the target job meets a first preset condition to obtain a first result includes at least one of the following: Determine, according to the job information of the target job, the resources occupied by the target job, and judge whether the resources occupied by the target job are greater than a resource threshold, to obtain the first result; Determine the business domain to which the target job belongs according to the job information of the target job, and judge whether the business domain to which the target job belongs conforms to a preset business domain, to obtain the first result; The importance of the target job is determined according to the job information of the target job, and it is determined whether the importance of the target job is less than a threshold, so as to obtain the first result.
9. The method according to claim 5, characterized in that The determining whether the data operation meets a second preset condition based on at least one item of content in the state information to obtain a second result includes: Based on at least one item of content in the status information, it is determined whether the running time of the data operation executed by the target job reaches a threshold time, and the second result is obtained.
10. The method according to any one of claims 1 to 9, characterized in that: The processing of the target job and the data operation performed by the target job includes at least one of the following: Stop running the data operation; Exit the process of the target job; The target job is suspended.
11. The method according to any one of claims 1 to 9, characterized in that: The data operation includes at least one of the following: Log in and / or connect to the database; Execute SQL statements; Execute database commands.
12. A processing device for data processing activities, characterized in that: include: A first acquisition module is used to acquire job information of a data job as a candidate, wherein the data job runs on an execution machine and calls a service of a database to perform a data operation; A second acquisition module is used to acquire status information of data operations in operation on the database; a determination module, configured to determine a target job matching the data operation from a plurality of candidate data jobs according to the job information and the state information; and The processing module is used to process the target job and the data operation performed by the target job according to the job information and / or the status information of the target job.
13. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 11.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 11 when executed by a processor.
15. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 11 when being executed by a processor.