Data Processing Method, System, Electronic Device and Storage Medium Based on Cloud Platform

Abstract

The present application provides a data processing method, system, electronic device and storage medium based on a cloud platform, which are applied to the technical field of cloud computing processing. The data processing method includes: obtaining task processing requests submitted by a number of target users through a distributed system, where the task processing requests are requests for processing scientific computing tasks; determining whether the number of the obtained task processing requests reaches a preset expansion threshold, and if so, generating an expansion request for the workload; performing an expansion process on the computing nodes according to the expansion request; redeploying the workload based on the expanded computing nodes to execute the scientific computing tasks based on the redeployed workload. By architecting and designing a scientific computing platform for drug research and development on the cloud, the high-performance computing requirements for scientific computing in drug research and development can be met, and the computing resource costs can be significantly reduced.

Description

Technical Field

[0001] This application relates to the technical field of cloud computing architectures, and specifically relates to a data processing method, system, electronic device, and storage medium based on a cloud platform. Background Art

[0002] The scientific calculations involved in drug research and development have the following characteristics: First, a large amount of computing power is required; second, different computing tasks often require different computing resources, such as CPUs (central processing units) and GPUs (graphics processing units); third, the amount of data involved and generated in the calculations is large; fourth, the computing requirements are very unfixed. Unlike Internet business calculations that are continuous online real-time calculations, the computing requirements in drug research and development often depend on the usage requirements of scientists, such as the computing requirements of many scientists at different times.

[0003] Currently, using single-machine computing can no longer meet the computing power requirements, or even if it does, a computing task may take weeks to months to run, seriously affecting work efficiency. In addition, although a large computer room can be built independently for scientific calculations in drug research and development, the fixed costs are too high, and the computing resources are fixed while the computing requirements are very unfixed, so a great deal of waste is also caused during idle periods.

[0004] Therefore, a new data processing solution is needed for the scientific calculations in drug research and development. Summary of the Invention

[0005] In view of this, the embodiments of this specification provide a data processing method, system, electronic device, and storage medium based on a cloud platform, which can meet the high-performance computing requirements of scientific calculations in drug research and development and can significantly reduce the computing resource costs.

[0006] The embodiments of this specification provide the following technical solutions:

[0007] The embodiments of this specification provide a data processing method based on a cloud platform, including:

[0008] Obtain task processing requests submitted by a number of target users through a distributed system, where the task processing requests are requests for processing scientific calculation tasks;

[0009] Determine whether the number of the obtained task processing requests reaches a preset expansion threshold. If so, generate an expansion request for the workload;

[0010] Perform an expansion process on the computing nodes according to the expansion request;

[0011] Redeploy the workload based on the computing nodes after the expansion process, and execute the scientific computing task based on the redeployed workload.

[0012] An embodiment of this specification also provides a data processing system based on a cloud platform, including: a distributed task scheduling system, a cloud elastic scaling system, and a task execution system. The distributed task scheduling system includes a task module, the cloud elastic scaling system includes a workload scaling module and a computing node scaling module, and the task execution system includes a number of computing nodes.

[0013] The task module is used to submit a task processing request for a scientific computing task by a target user, and determine whether the number of obtained task processing requests reaches a preset expansion threshold. If so, generate an expansion request for the workload.

[0014] The workload scaling module is used to trigger the computing node scaling module to expand the computing nodes according to the expansion request.

[0015] The computing node scaling module is used to perform an expansion process on the computing nodes according to the expansion request.

[0016] The computing nodes after the expansion process redeploy the workload, and execute the scientific computing task based on the redeployed workload.

[0017] An embodiment of this specification also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute: the data processing method described in any one of the embodiments in this specification.

[0018] An embodiment of this specification also provides a computer storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they execute: the data processing method described in any one of the embodiments in this specification.

[0019] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above technical solutions adopted in the embodiments of this specification at least include:

[0020] Based on the massive infrastructure provided by cloud computing, by architecting and designing a scientific computing platform for drug research and development on the cloud, it is equivalent to having a massive amount of computing infrastructure. Thus, there is no need to build a computer room on one's own, nor to maintain machine equipment by oneself. Moreover, machine resources of different types and quantities, storage resources of different types and sizes, etc. can all be allocated on demand. Additionally, by supporting large-scale elastic scaling of heterogeneous computing nodes in this platform, the computing nodes can be dynamically scaled according to computing requirements. When there is a computing demand, the most suitable machine type can be selected according to the computing demand, and the most appropriate number of machines can be expanded. When there is no computing demand, all idle platform computing nodes can be released, which can not only greatly improve resource utilization and save costs, but also meet the high-performance computing requirements of drug research and development, achieving the purpose of cost reduction and efficiency improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for use in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0022] Figure 1 It is a schematic structural diagram of an elastic scaling architecture for scientific data computing based on a cloud platform in the present application;

[0023] Figure 2 It is a flowchart of a data processing method based on a cloud platform in the present application;

[0024] Figure 3 It is a flowchart of a data processing method based on a cloud platform in the present application;

[0025] Figure 4 It is a schematic structural diagram of a data processing system based on a cloud platform in the present application;

[0026] Figure 5 It is a schematic structural diagram of a distributed task scheduling system in a data processing system based on a cloud platform in the present application;

[0027] Figure 6 It is a schematic structural diagram of an elastic scaling system in a data processing system based on a cloud platform in the present application;

[0028] Figure 7 It is a schematic structural diagram of a data processing system based on a cloud platform in the present application;

[0029] Figure 8 It is a schematic structural diagram of an electronic device for data processing based on a cloud platform in the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0031] The following specific examples illustrate the implementation manners of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of them. The present application can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.

[0032] It should be noted that the following description relates to various aspects of embodiments within the scope of the appended claims. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspects described herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using other structures and / or functionality in addition to one or more of the aspects described herein.

[0033] It should also be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application schematically. The diagrams only show the components related to the present application, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0034] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the examples can be practiced without these specific details.

[0035] Currently, a large amount of scientific data calculation is required in drug research and development. If the traditional single-machine calculation mode is adopted, the computing power basically cannot meet the requirements. If a large computer room is invested in construction, not only is the fixed cost input large, but also due to the non-fixed calculation requirements, a large amount of computing resources will be idle.

[0036] In view of this, through in-depth research and improvement exploration of scientific data calculation in drug research and development, and application exploration of existing cloud computing platforms, the embodiments of this specification propose a new data processing solution: as Figure 1 shown, in the elastic scaling processing architecture for scientific data calculation based on the cloud platform, scientists in drug research and development (such as User A, User B, etc.) can submit scientific computing tasks based on distributed task scheduling, and then determine cloud elastic scaling measures in real time based on these tasks. Finally, based on the cloud elastic scaling measures, the cloud computing resources (i.e., computing nodes) are scaled and adjusted in real time.

[0037] It should be noted that cloud computing resources can be the infrastructure in the cloud platform for massive data calculation. Deployed based on the cloud platform, it is equivalent to having a massive amount of computing infrastructure, and these infrastructures can be built and provided by cloud service providers, without the need for the application party to build its own computer room, nor for the application party to maintain. Moreover, the application party can allocate different types and quantities of machine resources, different types and sizes of storage resources, and other computing resources as needed.

[0038] Therefore, based on the massive infrastructure that cloud computing can provide, and designing and deploying a drug research and development scientific computing platform in the cloud platform that can scale and adjust computing resources (such as heterogeneous computing nodes), the computing nodes can be dynamically scaled on demand according to the computing requirements. That is, when there is a computing requirement, the most suitable machine type is selected according to the computing requirement, and the most suitable number of machines is expanded. When there is no computing requirement, the idle computing nodes can also be released. This can not only meet the high-performance computing requirements of scientific computing tasks in drug research and development, but also without excessive investment in costs, and can greatly improve resource utilization through scaling and adjustment, achieving the purpose of cost reduction and efficiency improvement.

[0039] The following describes the technical solutions provided by the embodiments of this application in conjunction with the accompanying drawings.

[0040] As Figure 2 shown, the embodiments of this specification provide a data processing method based on the cloud platform, which may include:

[0041] Step S202: Obtain task processing requests submitted by several target users through a distributed system, where the task processing requests are requests for processing scientific computing tasks.

[0042] In implementation, the target user can be a user who needs to perform scientific computing tasks, such as a scientist. At this time, the distributed system is used to obtain the task processing requests corresponding to the scientist's submitted computing requirements.

[0043] Step S204: Determine whether the number of the obtained task processing requests reaches a preset expansion threshold. If so, execute step S206.

[0044] By counting the number of tasks, it is possible to effectively determine whether the current computing resources are appropriate. When it is determined that the resources may not meet the computing requirements, more resources can be allocated in a timely manner through expansion to achieve on-demand resource allocation.

[0045] It should be noted that the expansion threshold can be preset, adjusted, etc. according to the actual needs of the deployed application, and no limitation is made here. In addition, when it is determined that no expansion processing is required, it can be processed according to a preset process, such as waiting until the quantity is sufficient and then performing expansion processing, which is not limited here.

[0046] Step S206: Generate an expansion request for the workload.

[0047] It should be noted that a workload refers to configurable resources that can run on cloud resources, such as specific application programs, services, functions, or specific workloads, etc. Therefore, virtual machines, databases, containers (Docker), Hadoop nodes, application programs, etc. can all be referred to as workloads on the cloud.

[0048] By generating an expansion request for the workload, the computing resources required for expansion can be allocated by the expansion request to meet the computing requirements.

[0049] Step S208: Perform expansion processing on the computing nodes according to the expansion request.

[0050] In implementation, the computing nodes can refer to virtual hosts provided by cloud service providers. Therefore, expansion processing can be performed on the computing nodes according to the expansion needs of the workload.

[0051] In some implementation manners, based on the workload required by the computing task, after summarizing the demand interpretation of the computing nodes through the cloud platform, the elastic scaling API interface provided by the cloud service provider can be called to dynamically and real-time adjust the types and quantities of resources required for computing, such as platform machine types and quantities, so as to achieve the expansion processing of the computing nodes by allocating resources.

[0052] Step S210: Redeploy the workload based on the computing nodes after the expansion processing to execute the scientific computing task based on the redeployed workload.

[0053] After the expansion processing of the computing nodes is completed, the workload can be deployed and run by starting the computing nodes to implement the execution processing of the scientific computing task.

[0054] Through the above steps S202 to S210, based on the massive computing resources provided by the cloud platform, these massive computing resources can be allocated on demand in real time according to the computing requirements of users to achieve the expansion of the computing nodes. Thus, the required workload can be redeployed based on the expanded computing nodes, and the scientific computing task can be executed by the workload.

[0055] In some embodiments, the task processing request may be circulated and processed in the cloud platform based on the event stream.

[0056] Specifically, when obtaining the task processing requests submitted by several target users, the data processing method may further include: generating a task event corresponding to the task processing request.

[0057] After adopting the event stream, the judgment of capacity expansion can be achieved through event stream processing. Specifically, when determining whether the number of obtained task processing requests reaches a preset capacity expansion threshold, it can be determined whether the number of generated task events reaches the preset capacity expansion threshold, so as to achieve fast event stream processing based on the event trigger of the task event.

[0058] In some embodiments, the triggering processing of the event stream can be achieved by listening to the task event, that is, determining whether the number of generated task events reaches the preset capacity expansion threshold, and the generated task events can be listened to determine whether the number of generated task events reaches the preset capacity expansion threshold.

[0059] In some embodiments, a corresponding work message can be formed based on the event stream, and then the scaling processing of the workload can be achieved based on the push of the work message.

[0060] Specifically, after generating the task event, a first work message corresponding to the task event can be generated based on the task event; then, when determining whether the number of generated task events reaches the preset capacity expansion threshold, the following judgment can be made based on the pushed work message: determining whether the number of generated first work messages reaches the preset capacity expansion threshold.

[0061] In some embodiments, after obtaining the task processing request, the task processing request is circulated and processed in the cloud platform based on the work message.

[0062] Specifically, when obtaining the task processing request, the data processing method may further include: generating a second work message corresponding to the task processing request; then, when determining whether the number of obtained task processing requests reaches the preset capacity expansion threshold, the following judgment processing can be performed based on the work message: determining whether the number of generated second work messages reaches the preset capacity expansion threshold.

[0063] In some embodiments, during the execution of the scientific computing task by the workload, the task status can be dynamically adjusted according to the actual execution result, or the task can be split into multiple subtasks.

[0064] During implementation, according to the execution result of the scientific computing task executed during operation of the workload, the task status of the scientific computing task is changed in real time, or the scientific computing task is created into multiple subtasks, and then the changed task status or the task corresponding to the subtask is calculated based on the computing nodes, which can not only make full use of the computing power of the computing nodes, but also improve the computing performance of the scientific computing task.

[0065] In some embodiments, the change of the task status and the creation of subtasks can generate more events, and this cycle continues until no new subtasks are created and the task status no longer changes, such as when the task has been completed or failed, which can improve the processing efficiency of the task.

[0066] By splitting a computing task and scheduling it to a large number of machines for processing, trading computing power for time, the waiting time of users for scientific computing tasks is greatly reduced, and the results can be obtained faster, improving work efficiency and the R & D iteration speed.

[0067] In some embodiments, the workload can be implemented as a set of meta-definition information, and the required workload can be quickly deployed and implemented through the definition information.

[0068] Specifically, the workload is set as a set of metadata, and the metadata may include the address of the Docker container image to be run, as well as the computing resource requirements for running the workload.

[0069] During implementation, the metadata may include but are not limited to the following fields: image_url (container image address), node_type (machine type for running this container), cpu_requirement (CPU requirement), gpu_requirement (GPU requirement), memory_requirement (memory requirement), storage_requirement (storage requirement), etc. It should be noted that the fields in the metadata can be set according to the actual situation of the deployed application and are not limited here.

[0070] In some embodiments, the computing nodes can be elastically scaled according to the running situation of the workload.

[0071] Specifically, it can be determined whether the target workload has finished running, and when it is determined that the target workload has finished running, the computing nodes deploying the target workload are put into an idle state.

[0072] In some embodiments, when it is determined that the computing nodes enter the idle state, the idle computing resources can be recycled. Specifically, after it is determined that the computing nodes deploying the target workload enter the idle state, the computing nodes deploying the target workload are shut down and recycled.

[0073] By dynamically allocating computing nodes in real time, the most suitable machine type can be selected according to the computing requirements, and the most suitable number of machines can be expanded when there is a computing demand; when there is no computing demand, all idle platform computing nodes can be released, which can greatly improve resource utilization and save costs.

[0074] In some embodiments, the task requests of users can be obtained based on a database. During implementation, when obtaining the task processing requests submitted by several target users through a distributed system, the following operations can be performed: when several target users submit a task processing request to a distributed database, the task processing request can be obtained based on the data changes in the distributed database.

[0075] During implementation, when a user stores task information in a distributed task database, at this time, based on data changes in the task database (such as data addition, status change, etc.), a data change capture program (Change Data Capture, CDC) can be activated, and the data change capture program can convert the data change into a corresponding processing process.

[0076] It should be noted that the method of data change capture can be set and adjusted according to the needs of deployed applications. For example, CDC based on timestamps, CDC based on triggers, CDC based on snapshots, and CDC based on logs, etc., are not limited here.

[0077] In some embodiments, the scientific computing tasks submitted by users can be processed based on a database, an event stream, and a message stream together.

[0078] During implementation, an asynchronous event-driven mode can be adopted for linkage work, such as Figure 3 as shown, the process example is as follows:

[0079] Step1. Obtain the scientific computing task (or task processing request) submitted by the user;

[0080] Step2. Store the task information of the scientific computing task in the task database;

[0081] Step3. The change in the task database (data addition or status change) activates the data change capture program (Change Data Capture, CDC);

[0082] Step4. The data change capture program converts the data change into an event and writes the event into the event stream;

[0083] Step5. Listen for a preset event and trigger event stream processing;

[0084] Step 6: The event stream processing interprets the event, generates the corresponding work message, and pushes the work message to the message queue in the message module;

[0085] Step 7: When the workload is running, it consumes messages from the message queue and executes the task work according to the message content;

[0086] Step 8: According to the execution result, by changing the task status or creating more subtasks, the change of the task status or the creation of subtasks (entering Step 1) leads to the generation of more events. This cycle continues until no new subtasks are created and the task status no longer changes (e.g., the task has been completed or failed).

[0087] Based on the efficient stream processing of databases, event streams, message streams, etc., it can meet the computing performance of tasks and improve the task processing efficiency at the same time.

[0088] Based on the same inventive concept, the embodiments of this specification provide a data processing system corresponding to the data processing method described in any one of the foregoing embodiments, wherein the data processing system is a data processing system deployed and constructed based on the massive computing resources provided by the cloud platform.

[0089] As Figure 4 shown, a data processing system based on the cloud platform provided by the embodiments of this specification may include a distributed task scheduling system 10, a cloud elastic scaling system 30, and a task execution system 50. Among them, the distributed task scheduling system 10 obtains the task processing request submitted by the user,

[0090] In implementation, the task execution system 50 may be the massive infrastructure provided by the cloud platform (such as configurable computing nodes), which is not limited here.

[0091] In implementation, the distributed task scheduling system 10 may act on the application layer of the cloud platform. As Figure 5 shown, a task module 101 is configured in the distributed task scheduling system 10. The task module 101 is used to interact with the user, facilitating the user to submit a scientific computing task (or put forward a task processing request) through the task module 101. Specifically, the task module 101 can be used for the target user to submit a task processing request for a scientific computing task and determine whether the number of obtained task processing requests reaches the preset expansion threshold. If so, it generates an expansion request for the workload.

[0092] In implementation, the elastic scaling system 30 acts on the infrastructure layer of the cloud platform. As Figure 6As shown, it may include two sub-modules, namely the workload autoscaler 301 and the node autoscaler 303. Among them, the workload autoscaler 301 is used to trigger the node autoscaler 303 to expand the computing nodes according to the expansion request provided by the task module 101, and the node autoscaler 303 is used to perform expansion processing on the computing nodes according to the expansion request. Therefore, the execution of the elastic scaling system 30 is as follows: First, the workload autoscaler scales the workload; then, the workload scaling triggers the activity of the node autoscaler; next, the node autoscaler calls the API interface provided by the cloud service provider to scale the machines, and the cloud service provider scales the computing nodes. After the scaled computing nodes are started, the workload is deployed and run to complete the corresponding computing tasks.

[0093] It should be noted that workload scaling can refer to expanding or shrinking the workload. For example, expanding a workload from 10 to 100, that is, shortening the processing time by increasing computing resources; computing nodes can refer to virtual hosts provided by cloud service providers; node scaling can refer to starting or shutting down virtual hosts of corresponding types through the API interface provided by the cloud service provider according to the machine requirements of the workload.

[0094] Based on the above example, the workflow of the data processing system is as follows: The task module 101 obtains the task processing request submitted by the user, and when it is determined that expansion processing is required, it sends a request for expansion processing to the workload autoscaler 301. The workload autoscaler 301 determines the scaling requirements corresponding to the workload according to the expansion request, and then triggers the node autoscaler 303 through the scaling requirements of the workload. The node autoscaler 303 requests the cloud service provider interface (such as the API interface) to scale the computing nodes. After the computing nodes are scaled, the workload is redeployed on the expanded computing nodes and the workload is run, and the scientific computing task is executed based on the workload redeployed on the expanded computing nodes, that is, the computing task corresponding to the task processing request is completed.

[0095] In some embodiments, the cloud elastic scaling system is a system deployed at the infrastructure layer. In order to develop and deploy computing task processing programs more quickly, a distributed task scheduling system (such as the aforementioned task module 101) can be encapsulated based on the cloud elastic scaling system, that is, the task scheduling system is encapsulated in the cloud elastic scaling system.

[0096] In some embodiments, such as Figure 5As shown, the task module 101 further includes a distributed database and a data change capture unit, where the data change capture unit can be a functional unit encapsulated with a data change capture program (Change Data Capture, CDC); the distributed database is used for task processing requests submitted by several target users for scientific computing tasks; the data change capture unit obtains the task processing request based on the data changes in the distributed database.

[0097] Specifically, the core components of the task module 101 may include: a database, and a database data change capture program (Change Data Capture).

[0098] After the task is submitted, it is stored in the database, and its schema definition includes but is not limited to the following core fields: job_id (unique task identifier), job_type (task type), job_status (task status), job_data (task custom information, such as input / output paths), etc. The database data change capture program can be implemented by listening to and interpreting database change events. For example, through the Binlog logs provided by MySQL, database changes can be captured in real time.

[0099] In some embodiments, as Figure 5 shown, the distributed task scheduling system 10 may further include an event module 103, where the event module 103 is used to listen for push events (i.e., task events) generated by the task module 101, and determine whether the number of task events reaches a preset expansion threshold. If so, it triggers the workload scaling module to expand the computing nodes, where the task event is an event corresponding to a task processing request and can be generated by the task module 101.

[0100] As Figure 5 shown, the core components of the event module may include an event stream and event stream processing. It should be noted that well-known stream processing middleware such as Kafka can be selected for the implementation of the event stream and event stream processing, which is not limited here.

[0101] In one example, the data change capture unit can be used to push events to the event module 103.

[0102] In some embodiments, as Figure 5 shown, the distributed task scheduling system 10 further includes a work message module 105. The work message module is used to listen for changes in the length of the message queue to determine whether to trigger the workload scaling module to expand the computing nodes, where the message queue is used to store work information corresponding to task processing requests.

[0103] As Figure 5As shown, the core components of the work message module may include a message queue and a workload. It should be noted that the message queue may be implemented using Amazon Simple Queue Service, and the workload may be implemented based on the Pod of the container orchestration system Kubernetes, which are not limited here.

[0104] In some implementations, the distributed task scheduling system is further used to change the task state corresponding to the scientific computing task or create the scientific computing task into multiple subtasks based on the execution result of the scientific computing task executed by the workload.

[0105] like Figure 5 As shown, the workload component in the work message module 105 can be used to change the task status or create subtasks to further scale the workload, wherein the change of task status and the creation of subtasks result in the generation of more data streams (such as events), and further processing of the data streams will not be described in detail.

[0106] In some implementations, the computing node scaling module 303 is further configured to detect whether the target computing node enters an idle state, and / or shut down and recycle the target computing node in an idle state.

[0107] In some embodiments, Figure 5 As shown, the distributed task scheduling system 10 may include three major sub-modules, namely: a task module 101 , an event module 103 and a work message module 105 .

[0108] During implementation, the three sub-modules can work in conjunction with each other in an asynchronous event-driven mode. The workflow is as follows:

[0109] 1. Users submit tasks through the task module;

[0110] 2. The task module stores the task information in the task database;

[0111] 3. Task database changes activate the data change capture program (Change Data Capture);

[0112] 4. The data change capture program converts data changes into events and writes the events into the event stream in the event module;

[0113] 5. Event stream preset event processor listens to the preset specific event and is triggered, that is, event stream processing;

[0114] 6. Event stream processing interprets processing events, generates specific work messages, and pushes the work messages to the message queue in the message module;

[0115] 7. When the workload is running, it consumes messages from the message queue, executes task work according to the message content, and changes the task status through the task module according to the execution result, or creates more subtasks;

[0116] 8. The change of the task status or the creation of subtasks (returning to the aforementioned step 1) will in turn cause more events to be generated, and so on in a loop until no new subtasks are created and the task status no longer changes (such as the task has been completed or failed).

[0117] In some embodiments, as Figure 7 shown, in the overall architecture schematic diagram of the data processing system deployed on the cloud platform, the cloud elastic scaling system and the distributed task scheduling system interact through the work message module, and the work process is as follows:

[0118] 1. The workload scaling module monitors the change in the length of the message queue;

[0119] 2. The workload scaling module dynamically scales and adjusts the number of workloads according to the change in the length of the message queue;

[0120] 3. The computing node scaling module monitors the number of workloads;

[0121] 4. The computing node scaling module dynamically scales and adjusts the computing nodes according to the number of workloads;

[0122] 5. The computing nodes deploy and run the workloads.

[0123] Based on the same inventive concept, the embodiments of this specification provide an electronic device corresponding to the data processing method described in any one of the foregoing embodiments.

[0124] Figure 8 This invention also provides a schematic structural diagram of an electronic device for data processing. The structure of the electronic device 500 is shown in the figure to be used to implement the data processing solution provided by this invention. Here, the electronic device 500 is only an example and should not limit the functions and usage scope of the embodiments of this invention.

[0125] As Figure 8 shown, in the electronic device 500, it may include: at least one processor 510; and a memory 520 communicatively connected to the at least one processor; wherein, the memory 520 stores instructions executable by the at least one processor 510, and the instructions are executed by the at least one processor 510 so that the at least one processor 510 can execute: the data processing method provided in any one of the embodiments in this specification.

[0126] It should be noted that the electronic device 500 may be presented in the form of a general computing device. For example, it may be a server device.

[0127] In implementation, the components of the electronic device 500 may include but are not limited to: at least one of the above-mentioned processors 510, at least one of the above-mentioned memories 520, and a bus 530 connecting different system components (including the memory 520 and the processor 510), where the bus 530 may include a data bus, an address bus, and a control bus.

[0128] In implementation, the memory 520 may include volatile memory, such as random access memory (RAM) 5201 and / or cache memory 5202, and may further include read-only memory (ROM) 5203.

[0129] The memory 520 may also include a program tool 5205 having a set (at least one) of program modules 5204. Such program modules 5204 include but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0130] The processor 510 executes various functional applications and data processing by running computer programs stored in the memory 520.

[0131] The electronic device 500 may also communicate with one or more external devices 540 (such as a keyboard, a pointing device, etc.). Such communication may be carried out through an input / output (I / O) interface 550. And, the electronic device 500 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 560. The network adapter 560 communicates with other modules in the electronic device 500 through the bus 530. It should be understood that although not shown in the figure, other hardware and / or software modules may be used in combination with the electronic device 500, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0132] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more of the above-mentioned units / modules may be embodied in one unit / module. Conversely, the features and functions of one unit / module described above may be further divided and embodied by multiple units / modules.

[0133] Based on the same inventive concept, an embodiment of this specification provides a computer storage medium storing computer-executable instructions, which, when executed by a processor, perform the data processing method provided in any one of the embodiments in this specification.

[0134] It should be noted that the computer storage medium may include, but is not limited to: portable disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories, optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0135] In a possible implementation, the present application may also provide a form of implementing data processing as a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute some steps in the method described in any of the foregoing embodiments.

[0136] Among them, the program code for executing the present application can be written in any combination of one or more programming languages, and the program code can be executed entirely on the user device, partially on the user device, executed as an independent software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0137] Each embodiment in this specification is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the product embodiments described later, since they correspond to the methods, the description is relatively simple, and the relevant parts can be referred to the description of the system embodiments.

[0138] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data processing method based on a cloud platform, characterized in that For the non-simultaneous computing requirements of several target users, including: Obtain task processing requests submitted by several target users through a distributed system, where the task processing requests are requests for processing scientific computing tasks; Determine whether the number of the obtained task processing requests reaches a preset expansion threshold. If so, generate an expansion request for the workload; Perform an expansion process on the computing nodes according to the expansion request; Redeploy the workload based on the expanded computing nodes to execute the scientific computing task based on the redeployed workload; Among them, according to the execution result of executing the scientific computing task by the workload, the task status of the scientific computing task is changed in real time, or the scientific computing task is created into multiple subtasks; By splitting a computing task and scheduling it to a large number of machines for processing, trading computing power for time to reduce the waiting time for processing the scientific computing task.

2. The data processing method based on a cloud platform according to claim 1, wherein When obtaining task processing requests submitted by several target users, the data processing method further includes: generating a task event corresponding to the task processing request; Determining whether the number of the obtained task processing requests reaches a preset expansion threshold includes: determining whether the number of the generated task events reaches a preset expansion threshold.

3. The data processing method based on a cloud platform according to claim 2, wherein Determining whether the number of the generated task events reaches a preset expansion threshold includes: listening to the generated task events to determine whether the number of the generated task events reaches a preset expansion threshold.

4. The data processing method based on a cloud platform according to claim 2, wherein The data processing method further includes: generating a first work message corresponding to the task event; Determining whether the number of the generated task events reaches a preset expansion threshold includes: determining whether the number of the generated first work messages reaches a preset expansion threshold.

5. The data processing method based on a cloud platform according to claim 1, wherein When obtaining the task processing request, the data processing method further includes: generating a second work message corresponding to the task processing request; Determining whether the number of the obtained task processing requests reaches a preset expansion threshold includes: determining whether the number of the generated second work messages reaches a preset expansion threshold.

6. The data processing method based on a cloud platform according to claim 1, wherein The data processing method further includes: setting the workload as a set of metadata, where the metadata includes the address of the Docker container image to be run and the computing resource requirements for running the workload.

7. The data processing method based on a cloud platform according to claim 1, wherein The data processing method further includes: determining whether the target workload has finished running. If so, put the computing node deploying the target workload into an idle state.

8. The data processing method based on a cloud platform according to claim 7, wherein After determining that the computing node deploying the target workload enters the idle state, the data processing method further includes: shutting down and recycling the computing node deploying the target workload.

9. The data processing method based on a cloud platform according to claim 1, wherein Obtaining task processing requests submitted by several target users through a distributed system includes: when several target users submit task processing requests to a distributed database, obtaining the task processing requests based on the data changes in the distributed database.

10. A data processing system based on a cloud platform, characterized in that, For the non-simultaneous computing requirements of several target users, including: a distributed task scheduling system, a cloud elastic scaling system, and a task execution system. The distributed task scheduling system includes a task module. The cloud elastic scaling system includes a workload scaling module and a computing node scaling module. The task execution system includes several computing nodes; The task module is used to submit a task processing request for a scientific computing task by a target user, and determine whether the number of obtained task processing requests reaches a preset expansion threshold. If so, it generates an expansion request for the workload; The workload scaling module is used to trigger the computing node scaling module to expand the computing nodes according to the expansion request; The computing node scaling module is used to perform an expansion process on the computing nodes according to the expansion request; The computing nodes after the expansion process redeploy the workload to execute the scientific computing task based on the redeployed workload: Wherein, according to the execution result of executing the scientific computing task by the workload, the task status of the scientific computing task is changed in real time, or the scientific computing task is created into multiple subtasks; By splitting a computing task and scheduling it to a large number of machines for processing, trading computing power for time to reduce the waiting time for the scientific computing task.

11. The data processing system based on a cloud platform according to claim 10, wherein The distributed task scheduling system further includes an event module. The event module is used to monitor task events and determine whether the number of task events reaches a preset expansion threshold. If so, it triggers the workload scaling module to expand the computing nodes, where the task events are events corresponding to task processing requests.

12. The data processing system based on a cloud platform according to claim 10, wherein The distributed task scheduling system further includes a work message module. The work message module is used to monitor the change in the length of the message queue to determine whether to trigger the workload scaling module to expand the computing nodes, where the message queue is used to store work information corresponding to task processing requests.

13. The data processing system based on a cloud platform according to claim 10, wherein The computing node scaling module is further used to detect whether a target computing node enters an idle state, and / or shut down and recycle the target computing node in the idle state.

14. The data processing system based on a cloud platform according to claim 10, wherein The task module further includes a distributed database and a data change capture unit; The distributed database is used to submit task processing requests for scientific computing tasks by several target users; The data change capture unit obtains the task processing request based on the data change of the distributed database.

15. An electronic device, characterized in that, Including: At least one processor; And a memory communicatively connected to the at least one processor. Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute: the data processing method according to any one of claims 1-9.

16. A computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they execute the data processing method according to any one of claims 1-9.

Images