Data processing method and device, computer device and storage medium

Abstract

The application relates to a data processing method and device, computer equipment and a storage medium. It relates to the field of financial technology or other related fields. The method comprises the following steps: acquiring a plurality of job tasks corresponding to a target application program in a financial business system, concurrently running the plurality of job tasks in a same target container of a function calculation platform included in a distributed batch platform; when detecting feature information corresponding to the concurrently run job tasks, intercepting the feature information through a distributed batch tool kit configured by the distributed batch platform; in the case that job monitoring information of the target application program meets a preset condition, returning the intercepted feature information to the function calculation platform to instruct the function calculation platform to clear the target container; the preset condition is that the plurality of job tasks are all processed completely. By using the method, the plurality of jobs of the batch function application can be concurrently run in the same container of the function calculation platform, and the normal processing of all jobs in the container is ensured.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a data processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] When implementing distributed batch programs on a serverless function computing platform based on a distributed batch framework, the requirement for multiple jobs to run concurrently in the same function container in financial business is problematic. Since each job returns a return message to the function computing platform after completion, the serverless function computing platform destroys the function container when it receives the first return message from the batch jobs. This results in other jobs still running on the function container being unable to complete their tasks in a timely manner, posing a risk of financial business processing failure. Summary of the Invention

[0003] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, storage medium, and computer program product that can solve the above-mentioned technical problems.

[0004] Firstly, this application provides a data processing method applied to a distributed batch platform, the method comprising:

[0005] Multiple job tasks corresponding to a target application in a financial business system are obtained, and the multiple job tasks are run concurrently in the same target container of the function computing platform included in the distributed batch platform; the target container runs the target function of the target application.

[0006] When the feature information corresponding to the concurrently running job task is detected, the feature information is intercepted through the distributed batch toolkit configured by the distributed batch platform; the feature information is generated after the job task is processed and is to be sent to the function computing platform.

[0007] If the job monitoring information of the target application meets the preset conditions, the intercepted feature information is returned to the function computing platform to instruct the function computing platform to clear the target container; the preset conditions are that all the multiple job tasks have been completed.

[0008] In one embodiment, before the step of intercepting the feature information through the distributed batch toolkit configured by the distributed batch platform when the feature information corresponding to the concurrently running job task is detected, the method further includes:

[0009] By adding monitoring threads, the original batch toolkit configured in the distributed batch platform is adjusted to obtain the adjusted batch toolkit, which is then used as the distributed batch toolkit.

[0010] The monitoring thread is used to enable the distributed batch toolkit to intercept the feature information and obtain the job monitoring information of the target application; the job monitoring information includes any one or more of the following:

[0011] Job running status information, job log information, container usage status information, and database monitoring information.

[0012] In one embodiment, when feature information corresponding to the concurrently running job task is detected, intercepting the feature information through the distributed batch toolkit configured by the distributed batch platform includes:

[0013] When the characteristic information corresponding to the concurrently running job task is detected, a simulated receiver is used to intercept the characteristic information corresponding to the job task; wherein, the simulated receiver is generated by the distributed batch toolkit based on the program information corresponding to the receiver used to receive the characteristic information.

[0014] In one embodiment, when the job monitoring information is the job running status information, before the step of returning the intercepted feature information to the function computing platform when the job monitoring information of the target application meets preset conditions, the method further includes:

[0015] Obtain a preset list of job identifiers; the list of job identifiers is used to store the identifiers corresponding to multiple job tasks to be run concurrently when the target application starts.

[0016] For each intercepted feature information, the identifier of the job task corresponding to the feature information is deleted from the job identifier list to obtain the latest job identifier list, which is used as the job running status information;

[0017] If the job identifier is not found in the job identifier list, then the preset condition is confirmed to be met.

[0018] In one embodiment, when the job monitoring information is the job log information, before the step of returning the intercepted feature information to the function computing platform when the job monitoring information of the target application meets preset conditions, the method further includes:

[0019] During the concurrent execution of the multiple job tasks in the target container, job log information is acquired; the job log information is used to store the job start log and job end log corresponding to each of the multiple job tasks.

[0020] If it is detected that all of the multiple job tasks have corresponding job start logs and job end logs, then it is confirmed that the preset conditions are met.

[0021] In one embodiment, when the job monitoring information is the container usage status information, before the step of returning the intercepted feature information to the function computing platform when the job monitoring information of the target application meets preset conditions, the method further includes:

[0022] When the target application has finished starting but no job has been executed, obtain the initial state information of the target container;

[0023] During the concurrent execution of the multiple job tasks in the target container, container usage status information is obtained;

[0024] If the difference between the container usage status information and the initial status information is less than a preset error threshold, then the preset condition is confirmed to be met.

[0025] In one embodiment, when the job monitoring information is the database monitoring information, before the step of returning the intercepted feature information to the function computing platform when the job monitoring information of the target application meets preset conditions, the method further includes:

[0026] Obtain key database information of the target application;

[0027] During the concurrent execution of the multiple job tasks in the target container, if the database monitoring information is detected to match the database key information, then the preset conditions are confirmed to be met.

[0028] Secondly, this application also provides a data processing apparatus for use in a distributed batch platform, the apparatus comprising:

[0029] A multi-job concurrent execution module is used to obtain multiple job tasks corresponding to a target application in a financial business system, and to concurrently run the multiple job tasks in the same target container of the function computing platform included in the distributed batch platform; the target container runs the target function of the target application.

[0030] The feature information interception module is used to intercept the feature information corresponding to the concurrently running job task through the distributed batch toolkit configured by the distributed batch platform when the feature information is detected; the feature information is generated after the job task is processed and is to be sent to the function computing platform.

[0031] The feature information return module is used to return the intercepted feature information to the function computing platform when the job monitoring information of the target application meets the preset conditions, so as to instruct the function computing platform to clear the target container; the preset conditions are that all the multiple job tasks have been completed.

[0032] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the data processing method described above.

[0033] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the data processing method described above.

[0034] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the data processing method described above.

[0035] The aforementioned data processing method, apparatus, computer equipment, storage medium, and computer program product acquire multiple job tasks corresponding to a target application in a financial business system. These job tasks are then run concurrently within the same target container of a function computing platform included in a distributed batch platform. The target container runs the target function of the target application. Upon detecting feature information corresponding to the concurrently running job tasks, the feature information is intercepted using a distributed batch toolkit configured on the distributed batch platform. This feature information is generated after the job tasks are completed and is to be sent to the function computing platform. Furthermore, if the job monitoring information of the target application meets preset conditions, the intercepted feature information is returned to the function computing platform to instruct it to clear the target container. These preset conditions are that all job tasks have been completed. This enables multiple jobs of batch functional applications to run concurrently within the same container of the function computing platform, ensuring that all jobs within the container are processed normally and improving the efficiency of financial business processing. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating a data processing method in one embodiment;

[0037] Figure 2 This is a schematic diagram of a distributed batch platform architecture in one embodiment;

[0038] Figure 3 This is a flowchart illustrating one step in monitoring the running status of a job in one embodiment;

[0039] Figure 4 This is a flowchart illustrating another data processing method in one embodiment;

[0040] Figure 5 This is a structural block diagram of a data processing device in one embodiment;

[0041] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties; correspondingly, this application also provides a corresponding user authorization entry point for users to choose to authorize or refuse.

[0044] In one embodiment, such as Figure 1 As shown, a data processing method is provided that can be applied to a distributed batch processing platform. In this embodiment, the method includes the following steps:

[0045] Step 101: Obtain multiple job tasks corresponding to the target application in the financial business system, and run the multiple job tasks concurrently in the same target container of the function computing platform included in the distributed batch platform; the target container runs the target function of the target application.

[0046] Among them, the distributed batch platform can be a serverless batch platform. Serverless is a cloud computing model, namely the cloud-native development model, which allows developers to build and run applications without managing servers. It provides a micro-architecture through serverless computing, so that end users do not need to deploy, configure or manage server services. The server services required for code execution can all be provided by the cloud platform.

[0047] As an example, the target application can be a batch functional application, which is an application based on a distributed batch framework and whose actual executor runs on a function platform. That is, the executor of the target application (such as the target container) can run on a function computing platform.

[0048] In practical applications, in order to enable those skilled in the art to better understand the steps in this embodiment, the following is combined with... Figure 2 The embodiments of this application are illustrated by way of example, but it should be understood that the embodiments of this application are not limited thereto.

[0049] like Figure 2 The diagram illustrates the architecture and interaction flow of a Serverless batch platform. This platform can include a Function Compute platform, Zookeeper, a ServerlessDBF management module, a batch controller, a batch database, a batch console, and an executor running on the Function Compute platform. It is configured with a Serverless-based distributed batch SDK, i.e., a cloud-native function batch software development kit, which can be a customized SDK for batch function applications or a modified version of the original distributed batch SDK. ZooKeeper is a distributed application coordination service software that provides functions including configuration maintenance, domain name service, distributed synchronization, and group services.

[0050] The batch controller is responsible for job allocation and job execution process control in the distributed batch platform. The executor running on the function computing platform is responsible for receiving job allocation instructions from the batch controller in the distributed batch platform. After receiving the instructions from the batch controller, it can interpret the instructions and then execute the corresponding instruction operations, such as job preprocessing, job start and job stop, based on the job instance, instruction type and other information in the instructions. ZooKeeper can play the role of communication between the executor and the controller in the distributed batch platform.

[0051] In one alternative embodiment, such as Figure 2 In the function registration process, the application can import the function configuration file and layered parameter file into the front end of the ServerlessDBF management module. Then, the ServerlessDBF management module can register the function information with Zookeeper based on the imported information and call the function computing platform interface to create functions. At the same time, the batch controller can listen for the registration information.

[0052] In one example, such as Figure 2In the process of starting a batch job and triggering a function, once the batch job execution time is reached, the batch controller can send job execution information to Zookeeper and change the job status in the database to "started". At the same time, the ServerlessDBF management module can listen for this execution information and, carrying the information sent by the controller, trigger the function computing platform interface to start the function program instance and execute the job. That is, multiple job tasks are run concurrently in the same target container of the function computing platform included in the distributed batch platform.

[0053] Step 102: When the feature information corresponding to the concurrently running job task is detected, the feature information is intercepted through the distributed batch toolkit configured by the distributed batch platform.

[0054] As an example, the feature information can be information generated after the job task is completed and to be sent to the Function Compute platform, such as return information returned to the Function Compute platform.

[0055] In practical implementation, by adding a monitoring thread, the original batch toolkit (such as the Serverless-based distributed batch SDK) of the distributed batch platform configuration can be adjusted to obtain the adjusted batch toolkit, which can then be used as the distributed batch toolkit (such as the modified Serverless-based distributed batch SDK). This monitoring thread can be used to enable the distributed batch toolkit to intercept feature information and obtain job monitoring information of the target application. This job monitoring information may include, but is not limited to, job running status information, job log information, container usage status information, and database monitoring information.

[0056] In one example, a simulated receiver can be generated based on the program information corresponding to the receiver used to receive feature information using a distributed batch toolkit. This simulated receiver can then be used to intercept the feature information corresponding to concurrently running job tasks when the feature information corresponding to the job tasks is detected.

[0057] For example, such as Figure 2 In the job completion step, when the function executor finishes the job, it can send an execution completion message to Zookeeper. At the same time, the batch controller can listen for the job completion message and change the job status in the batch database to "completed". Since the executor can return a return message to the function computing platform after sending a message to Zookeeper, the returned return message (i.e., characteristic information) can be intercepted.

[0058] For example, for each concurrently running job task, the executor returns a return message after each job task is completed. When multiple job tasks are sent to a function container, there will be multiple return messages. Since the function computing platform can destroy the corresponding function container after receiving the first return message, other unfinished job tasks will not be able to complete. By intercepting the returned return messages, the time when the function computing platform destroys the container can be controlled, so that the container can be destroyed after all tasks are completed in the case of multiple tasks. This overcomes the problem that multiple jobs of batch function applications cannot run concurrently in the same container of the function computing platform.

[0059] In another example, a serverless-based distributed batch SDK was modified to add a monitoring thread. This thread is capable of intercepting return messages and monitoring metrics such as job execution, job logs, container CPU and memory usage, and database performance. This allows batch functional applications to select the metrics they need to monitor, achieving the goal of running multiple jobs concurrently within the same container on the functional platform.

[0060] Step 103: If the job monitoring information of the target application meets the preset conditions, the intercepted feature information is returned to the function computing platform to instruct the function computing platform to clear the target container; the preset conditions are that all the multiple job tasks have been completed.

[0061] In practical applications, after intercepting feature information, since it is necessary to return the intercepted feature information to the Function Compute platform to instruct the platform to properly destroy the container, the feature information can be resent at an appropriate time, causing the Function Compute platform to delete the container. Specifically, this can be achieved by obtaining job monitoring information from the target application. Once it is determined from this job monitoring information that multiple job tasks have been completed, the intercepted feature information can be returned to the Function Compute platform.

[0062] For example, job monitoring information can be obtained by monitoring job operation status, job logs, container CPU and memory usage, database and other indicators, and then the time to resend the return information can be determined based on the job monitoring information.

[0063] In one example, each batch functional application behaves differently. Corresponding monitoring metrics can be set for each application. For instance, application A prints an end-of-run log upon completion, so this log can be configured as a monitoring metric for application A. Similarly, application B experiences a significant drop in CPU utilization after execution, so CPU utilization can be configured as a monitoring metric for application B. By modifying the Serverless-based distributed batch SDK, distributed batch programs can determine appropriate task completion indicators, such as logs and CPU utilization, based on the concurrent execution of multiple jobs. The modified SDK can then send a return message to the function computing platform when all or some metrics have passed verification, enabling multiple jobs to run concurrently within the same container on the Serverless batch platform.

[0064] In yet another example, such as Figure 2 In the function deregistration process, the application can enter a command in the front end of the ServerlessDBF management module to delete a specified function. After receiving the command, the ServerlessDBF management module can delete the registration information of the corresponding function in Zookeeper and call the function computing platform interface to delete the function, thus completing the function deregistration.

[0065] Compared to traditional methods, the technical solution in this embodiment modifies the Serverless-based distributed batch SDK by adding a monitoring thread. This monitoring thread can intercept the return information of job tasks within the application and monitor job execution status, job logs, container CPU and memory usage, database, and other methods to ensure that the SDK only returns return information to the Serverless function computing platform after all job tasks are completed. This enables distributed batch programs to run jobs concurrently in the same container of the Serverless function computing platform using multiple threads.

[0066] In the above data processing method, multiple job tasks corresponding to the target application in the financial business system are obtained. These job tasks are then run concurrently in the same target container of the function computing platform included in the distributed batch platform. When feature information corresponding to the concurrently running job tasks is detected, the feature information is intercepted by the distributed batch toolkit configured by the distributed batch platform. Then, if the job monitoring information of the target application meets the preset conditions, the intercepted feature information is returned to the function computing platform to instruct the function computing platform to clear the target container. This enables multiple jobs of batch functional applications to run concurrently in the same container of the function computing platform, ensuring that all jobs in the container are processed normally and improving the efficiency of financial business processing.

[0067] In one embodiment, before the step of intercepting the feature information through the distributed batch toolkit configured by the distributed batch platform when the feature information corresponding to the concurrently running job task is detected, the following steps may also be included:

[0068] By adding monitoring threads, the original batch toolkit configured in the distributed batch platform is adjusted to obtain the adjusted batch toolkit, which is then used as the distributed batch toolkit.

[0069] The monitoring thread can be used to enable the distributed batch toolkit to intercept characteristic information, such as return information, and to obtain job monitoring information of the target application. This job monitoring information may include any one or more of the following: job running status information, job log information, container usage status information (such as container CPU usage and container memory usage), and database monitoring information.

[0070] In practical applications, using the modified Serverless-based distributed batch SDK, applications can configure whether to enable certain functions in the application configuration file, such as monitoring job running status 1, monitoring job logs 2, monitoring container CPU usage 3, monitoring container memory usage 4, and monitoring the database 5. Each function can be enabled or disabled by the application.

[0071] In one example, with the return information interception function enabled, it is necessary to select and enable any one or more of the monitoring functions 1-5 mentioned above, so as to achieve the goal of multiple jobs running concurrently in the same container on the Serverless batch platform.

[0072] In another example, since the accuracy of monitoring job running status is higher than that of monitoring job logs, container CPU usage, and container memory usage, enabling the function 1 for monitoring job running status information can improve monitoring accuracy.

[0073] In this embodiment, by adding monitoring threads and adjusting the original batch toolkit configured in the distributed batch platform, an adjusted batch toolkit is obtained. As a distributed batch toolkit, it can modify the Serverless-based distributed batch SDK to enable it to intercept return information and monitor job execution, job logs, container CPU and memory usage, database and other metrics.

[0074] In one embodiment, intercepting the feature information corresponding to the concurrently running job task through the distributed batch toolkit configured by the distributed batch platform when the feature information is detected may include the following steps:

[0075] When the characteristic information corresponding to the concurrently running job task is detected, a simulated receiver is used to intercept the characteristic information corresponding to the job task; wherein, the simulated receiver is generated by the distributed batch toolkit based on the program information corresponding to the receiver used to receive the characteristic information.

[0076] In practical applications, for the return information interception function: by using a modified serverless-based distributed batch SDK (i.e., distributed batch toolkit), return information (i.e., feature information) can be intercepted by creating a fake receiver (i.e., a simulated receiver).

[0077] For example, when a job task is completed, the program code can return a return message to the upper-level program. The distributed batch toolkit can mimic the code of the upper-level program to generate a pseudo-receiver code with a similar name but different internal logic. This allows the program's return message to be returned to the pseudo-receiver code, thus intercepting the return message. After determining that all job tasks have been completed, a return message can be sent to the Serverless Function Computing Platform. This function can be enabled or disabled by the application.

[0078] In this embodiment, by using a receiver-like approach to intercept the feature information corresponding to concurrently running job tasks when the feature information corresponding to the concurrently running job tasks is detected, it is possible to avoid the situation where multiple jobs of batch functional applications cannot run concurrently in the same container of the function computing platform.

[0079] In one embodiment, such as Figure 3 As shown, when the job monitoring information is job running status information, before the step of returning the intercepted feature information to the function computing platform when the job monitoring information of the target application meets the preset conditions, the following steps may also be included:

[0080] Step 301: Obtain a preset job identifier list; the job identifier list is used to store the identifiers corresponding to multiple job tasks to be run concurrently when the target application starts.

[0081] Step 302: For each intercepted feature information, delete the identifier of the job task corresponding to the feature information from the job identifier list to obtain the latest job identifier list, which serves as the job running status information;

[0082] Step 303: If the job identifier is not found in the job identifier list, then the preset condition is confirmed to be met.

[0083] In one example, regarding the function of monitoring job execution status: the modified Serverless-based distributed batch SDK can add a job field (i.e., a job identifier list). When the application starts, this job field can store the names of all job tasks to be run (i.e., the identifiers corresponding to the job tasks). When the modified SDK intercepts a return message, it can remove the job task name corresponding to the intercepted return message from the job field. Thus, when the monitoring shows that there is no content in the job field (i.e., the job identifier list does not contain the identifier of the job task), it can be confirmed that all job tasks have been completed (i.e., the preset conditions are met).

[0084] In this embodiment, a preset list of job identifiers is obtained, and then for each intercepted feature information, the identifier of the job task corresponding to the feature information is deleted from the list of job identifiers to obtain the latest list of job identifiers, which serves as the job running status information. If it is detected that there is no job task identifier in the list of job identifiers, it is confirmed that the preset conditions are met. By monitoring the job running status, it can be determined whether all job tasks have been completed, thus improving the accuracy of monitoring.

[0085] In one embodiment, when the job monitoring information is job log information, before the step of returning the intercepted feature information to the function computing platform when the job monitoring information of the target application meets preset conditions, the following steps may be included:

[0086] During the concurrent execution of the multiple job tasks in the target container, job log information is acquired; the job log information is used to store the job start log and job end log corresponding to each of the multiple job tasks; if it is detected that all of the multiple job tasks have corresponding job start logs and job end logs, then it is confirmed that the preset condition is met.

[0087] In the specific implementation, regarding the function of monitoring job logs: the modified Serverless-based distributed batch SDK can print a log line with the job name before and after each job task starts and ends. It can scan all application logs (i.e., job log information) according to a preset time (e.g., scan once every 5 seconds) until it is confirmed that each job task has corresponding two log lines, and it can be confirmed that all job tasks have been completed (i.e., the preset conditions are met).

[0088] In this embodiment, by acquiring job log information during the concurrent execution of multiple job tasks in the target container, if it is detected that multiple job tasks have corresponding job start logs and job end logs, it is confirmed that the preset conditions are met, and it is possible to determine whether all job tasks have been completed based on the monitoring job logs.

[0089] In one embodiment, when the job monitoring information is container usage status information, before the step of returning the intercepted feature information to the function computing platform when the job monitoring information of the target application meets preset conditions, the following steps may be included:

[0090] When the target application has finished starting but no jobs are being executed, the initial state information of the target container is obtained; during the concurrent running of the multiple job tasks in the target container, the container usage state information is obtained; if the difference between the container usage state information and the initial state information is less than a preset error threshold, then the preset condition is confirmed to be met.

[0091] In one example, regarding the function of monitoring container CPU utilization: the modified Serverless-based distributed batch SDK can call the underlying system instructions to obtain the CPU utilization of the current container (i.e., initial status information) when the application has started but has not executed any jobs. For example, the recorded data is an integer C. Then, the container CPU utilization (i.e., container usage status information) can be obtained according to a preset time (e.g., once every 3 seconds). When the difference between the CPU utilization and the value of C is detected to be less than the preset error value (i.e., preset error threshold) of the application for 3 consecutive times, such as when the error value can be 5%, it can be confirmed that all job tasks have been completed (i.e., the preset conditions are met).

[0092] In another example, regarding the function of monitoring container memory usage: the modified Serverless-based distributed batch SDK can call the underlying system instructions to obtain the memory usage (i.e., initial status information) of the current container when the application has started but has not executed any jobs. For example, the recorded data is an integer M. Then, the memory usage (i.e., container usage status information) can be obtained according to a preset time (obtained once every 3 seconds). When the difference between the memory usage and the value of M is detected to be less than the preset error value (i.e., preset error threshold) of the application for 3 consecutive times, such as when the error value can be 5%, it can be confirmed that all job tasks have been completed (i.e., the preset conditions are met).

[0093] In this embodiment, the initial state information of the target container is obtained when the target application has started but no job has been executed. Then, the container usage state information is obtained during the concurrent execution of multiple job tasks in the target container. If the difference between the container usage state information and the initial state information is less than a preset error threshold, it is confirmed that the preset condition is met. Based on the monitoring of the container usage state, it is possible to determine whether all job tasks have been completed.

[0094] In one embodiment, when the job monitoring information is the database monitoring information, before the step of returning the intercepted feature information to the function computing platform when the job monitoring information of the target application meets preset conditions, the following steps may be included:

[0095] Obtain key database information of the target application; during the concurrent execution of the multiple job tasks in the target container, if the database monitoring information is detected to match the key database information, then it is confirmed that the preset conditions are met.

[0096] In practical applications, for the function of monitoring the database, since the application can write data to the database after running, the modified serverless distributed batch SDK can check the database according to the database address information, operation statements, keywords, etc. configured by the application, according to the preset time (such as every 3 seconds). If the result is confirmed to match the keyword, it can be confirmed that all job tasks have been completed (i.e., the preset conditions are met).

[0097] In this embodiment, by obtaining key database information of the target application, and then during the concurrent execution of multiple job tasks in the target container, if the database monitoring information is detected to match the key database information, it is confirmed that the preset conditions are met, and it is possible to determine whether all job tasks have been completed based on the monitoring database.

[0098] In one embodiment, such as Figure 4 The diagram illustrates another data processing method. In this embodiment, the method includes the following steps:

[0099] In step 401, by adding monitoring threads, the original batch toolkit configured in the distributed batch platform is adjusted to obtain an adjusted batch toolkit, which serves as the distributed batch toolkit. In step 402, multiple job tasks corresponding to the target application in the financial business system are obtained, and these job tasks are run concurrently within the same target container of the function computing platform included in the distributed batch platform. In step 403, when feature information corresponding to concurrently running job tasks is detected, a receiver-like approach is used to intercept this feature information. In step 404, a preset job identifier list is obtained; this list stores the identifiers corresponding to the multiple job tasks to be run concurrently when the target application starts. In step 405, for each intercepted feature information, the identifier of the job task corresponding to the feature information is deleted from the job identifier list to obtain the latest job identifier list, which serves as the job running status information. In step 406, when the job monitoring information is job running status information, if no job task identifier is detected in the job identifier list, a preset condition is confirmed to be met; the preset condition is that multiple job tasks have been completed. In step 407, if the job monitoring information of the target application meets preset conditions, the intercepted feature information is returned to the Function Compute platform to instruct the Function Compute platform to clear the target container. It should be noted that the specific limitations of the above steps can be found in the specific limitations of a data processing method described above, and will not be repeated here.

[0100] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0101] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0102] In one embodiment, such as Figure 5 As shown, a data processing apparatus is provided, comprising:

[0103] The multi-job concurrent execution module 501 is used to obtain multiple job tasks corresponding to the target application in the financial business system, and to concurrently run the multiple job tasks in the same target container of the function computing platform included in the distributed batch platform; the target container runs the target function of the target application.

[0104] The feature information interception module 502 is used to intercept the feature information corresponding to the concurrently running job task through the distributed batch toolkit configured by the distributed batch platform when the feature information is detected; the feature information is generated after the job task is processed and is to be sent to the function computing platform.

[0105] The feature information return module 503 is used to return the intercepted feature information to the function computing platform when the job monitoring information of the target application meets the preset conditions, so as to instruct the function computing platform to clear the target container; the preset conditions are that all the multiple job tasks have been completed.

[0106] In one embodiment, the apparatus further includes:

[0107] The monitoring thread addition module is used to adjust the original batch toolkit configured in the distributed batch platform by adding monitoring threads, so as to obtain the adjusted batch toolkit as the distributed batch toolkit.

[0108] The monitoring thread is used to enable the distributed batch toolkit to intercept the feature information and obtain the job monitoring information of the target application; the job monitoring information includes any one or more of the following: job running status information, job log information, container usage status information, and database monitoring information.

[0109] In one embodiment, the feature information interception module 502 includes:

[0110] The simulated receiver interception submodule is used to intercept the characteristic information corresponding to the concurrently running job task when the characteristic information corresponding to the job task is detected; wherein, the simulated receiver is generated by the distributed batch toolkit based on the program information corresponding to the receiver used to receive the characteristic information.

[0111] In one embodiment, when the job monitoring information is the job running status information, the device further includes:

[0112] The job identifier list acquisition module is used to acquire a preset job identifier list; the job identifier list is used to store the identifiers corresponding to multiple job tasks to be run concurrently when the target application starts.

[0113] The identifier deletion module is used to delete the identifier of the job task corresponding to each intercepted feature information from the job identifier list to obtain the latest job identifier list, which serves as the job running status information.

[0114] The first condition confirmation module is used to confirm that the preset condition is met if the job identifier does not exist in the job identifier list.

[0115] In one embodiment, when the job monitoring information is the job log information, the device further includes:

[0116] The job log information acquisition module is used to acquire job log information during the concurrent execution of the multiple job tasks in the target container; the job log information is used to store the job start log and job end log corresponding to each of the multiple job tasks;

[0117] The second condition confirmation module is used to confirm that the preset condition is met if it is detected that all of the multiple job tasks have corresponding job start logs and job end logs.

[0118] In one embodiment, when the job monitoring information is the container usage status information, the device further includes:

[0119] The container initial information acquisition module is used to acquire the initial state information of the target container when the target application has finished starting but has not executed any jobs;

[0120] The container usage status information acquisition module is used to acquire container usage status information during the concurrent execution of the multiple job tasks in the target container;

[0121] The third condition confirmation module is used to confirm that the preset condition is met if the difference between the container usage status information and the initial status information is less than a preset error threshold.

[0122] In one embodiment, when the job monitoring information is the database monitoring information, the apparatus further includes:

[0123] A database key information acquisition module is used to acquire key database information of the target application.

[0124] The fourth condition confirmation module is used to confirm that the preset condition is met if the database monitoring information matches the database key information during the concurrent execution of the multiple job tasks in the target container.

[0125] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0126] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data for data processing. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0127] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0128] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0129] Multiple job tasks corresponding to a target application in a financial business system are obtained, and the multiple job tasks are run concurrently in the same target container of the function computing platform included in the distributed batch platform; the target container runs the target function of the target application.

[0130] When the feature information corresponding to the concurrently running job task is detected, the feature information is intercepted through the distributed batch toolkit configured by the distributed batch platform; the feature information is generated after the job task is processed and is to be sent to the function computing platform.

[0131] If the job monitoring information of the target application meets the preset conditions, the intercepted feature information is returned to the function computing platform to instruct the function computing platform to clear the target container; the preset conditions are that all the multiple job tasks have been completed.

[0132] In one embodiment, the processor, when executing a computer program, also implements the steps of the data processing method described in the other embodiments above.

[0133] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0134] Multiple job tasks corresponding to a target application in a financial business system are obtained, and the multiple job tasks are run concurrently in the same target container of the function computing platform included in the distributed batch platform; the target container runs the target function of the target application.

[0135] When the feature information corresponding to the concurrently running job task is detected, the feature information is intercepted through the distributed batch toolkit configured by the distributed batch platform; the feature information is generated after the job task is processed and is to be sent to the function computing platform.

[0136] If the job monitoring information of the target application meets the preset conditions, the intercepted feature information is returned to the function computing platform to instruct the function computing platform to clear the target container; the preset conditions are that all the multiple job tasks have been completed.

[0137] In one embodiment, when the computer program is executed by a processor, it also implements the steps of the data processing method described in the other embodiments above.

[0138] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0139] Multiple job tasks corresponding to a target application in a financial business system are obtained, and the multiple job tasks are run concurrently in the same target container of the function computing platform included in the distributed batch platform; the target container runs the target function of the target application.

[0140] When the feature information corresponding to the concurrently running job task is detected, the feature information is intercepted through the distributed batch toolkit configured by the distributed batch platform; the feature information is generated after the job task is processed and is to be sent to the function computing platform.

[0141] If the job monitoring information of the target application meets the preset conditions, the intercepted feature information is returned to the function computing platform to instruct the function computing platform to clear the target container; the preset conditions are that all the multiple job tasks have been completed.

[0142] In one embodiment, when the computer program is executed by a processor, it also implements the steps of the data processing method described in the other embodiments above.

[0143] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, Applied to a distributed batch platform, the method includes: Multiple job tasks corresponding to a target application in a financial business system are obtained, and the multiple job tasks are run concurrently in the same target container of the function computing platform included in the distributed batch platform; the target container runs the target function of the target application. When the feature information corresponding to the concurrently running job task is detected, a simulated receiver is used to intercept the feature information corresponding to the job task; wherein, the simulated receiver is a distributed batch toolkit configured through the distributed batch platform, generated based on the program information corresponding to the receiver used to receive the feature information; the feature information is generated after the job task is processed and is to be sent to the function computing platform; If the job monitoring information of the target application meets the preset conditions, the intercepted feature information is returned to the function computing platform to instruct the function computing platform to clear the target container; the job monitoring information of the target application meeting the preset conditions indicates that multiple concurrent job tasks in the target container have been completed.

2. The method according to claim 1, characterized in that, Before the step of intercepting the feature information corresponding to the concurrently running job task through the distributed batch toolkit configured by the distributed batch platform when the feature information is detected, the method further includes: By adding monitoring threads, the original batch toolkit configured in the distributed batch platform is adjusted to obtain the adjusted batch toolkit, which is then used as the distributed batch toolkit. The monitoring thread is used to enable the distributed batch toolkit to intercept the feature information and obtain the job monitoring information of the target application; the job monitoring information includes any one or more of the following: Job running status information, job log information, container usage status information, and database monitoring information.

3. The method according to claim 2, characterized in that, Before the step of returning the intercepted feature information to the function computing platform when the job monitoring information is the job running status information and the job monitoring information of the target application meets preset conditions, the method further includes: Obtain a preset list of job identifiers; the list of job identifiers is used to store the identifiers corresponding to multiple job tasks to be run concurrently when the target application starts. For each intercepted feature information, the identifier of the job task corresponding to the feature information is deleted from the job identifier list to obtain the latest job identifier list, which is used as the job running status information; If the job identifier is not found in the job identifier list, then the preset condition is confirmed to be met.

4. The method according to claim 2, characterized in that, Before the step of returning the intercepted feature information to the function computing platform when the job monitoring information of the target application meets preset conditions, the method further includes: During the concurrent execution of the multiple job tasks in the target container, job log information is acquired; the job log information is used to store the job start log and job end log corresponding to each of the multiple job tasks. If it is detected that all of the multiple job tasks have corresponding job start logs and job end logs, then it is confirmed that the preset conditions are met.

5. The method according to claim 2, characterized in that, Before the step of returning the intercepted feature information to the function computing platform when the job monitoring information of the target application meets preset conditions, and when the job monitoring information is the container usage status information, the method further includes: When the target application has finished starting but no job has been executed, obtain the initial state information of the target container; During the concurrent execution of the multiple job tasks in the target container, container usage status information is obtained; If the difference between the container usage status information and the initial status information is less than a preset error threshold, then the preset condition is confirmed to be met.

6. The method according to claim 2, characterized in that, Before the step of returning the intercepted feature information to the function computing platform when the job monitoring information of the target application meets preset conditions, the method further includes: Obtain key database information of the target application; During the concurrent execution of the multiple job tasks in the target container, if the database monitoring information is detected to match the database key information, then the preset conditions are confirmed to be met.

7. A data processing apparatus, characterized in that, The device, applied to a distributed batch platform, includes: A multi-job concurrent execution module is used to obtain multiple job tasks corresponding to a target application in a financial business system, and to concurrently run the multiple job tasks in the same target container of the function computing platform included in the distributed batch platform; the target container runs the target function of the target application. The feature information interception module is used to intercept the feature information corresponding to the concurrently running job task by using a simulated receiver when the feature information corresponding to the concurrently running job task is detected; wherein, the simulated receiver is a distributed batch toolkit configured through the distributed batch platform, generated based on the program information corresponding to the receiver used to receive the feature information; the feature information is the information generated after the job task is processed and is to be sent to the function computing platform; The feature information return module is used to return the intercepted feature information to the function computing platform when the job monitoring information of the target application meets the preset conditions, so as to instruct the function computing platform to clear the target container; the job monitoring information of the target application meeting the preset conditions indicates that multiple concurrent job tasks in the target container have been completed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images