Data processing method and device based on server-free computing, equipment and medium
By introducing serverless computing data processing methods in traditional server-free computing clusters, the problems of high deployment and operation and maintenance thresholds and complex expansion and capacity in traditional methods are solved, and higher computing cluster elasticity and user experience are achieved.
Patent Information
- Application Number
- CN202411755715.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-02
- Publication Date
- 2025-04-04
AI Technical Summary
Traditional server computing clusters have high deployment and operation and maintenance thresholds when processing data, and the expansion and scaling process is complex and time-consuming, resulting in insufficient elasticity of computing clusters.
The data processing method based on serverless computing is adopted. By receiving user target instructions, analyzing and executing resource allocation and scheduling operations, a computing power pool is built, and data processing tasks are executed after the construction is completed, which reduces the threshold for resource deployment and operation and maintenance and improves the elasticity of the computing cluster.
It realizes that data processing tasks can be executed without manually configuring server resources, lowers the operation and maintenance threshold, and improves the elasticity and user experience of the computing cluster.
Smart Images
Figure CN119759548A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, device, equipment and medium based on serverless computing. Background Art
[0002] At present, users use traditional server-based computing clusters for data processing, such as data cleaning. When users use traditional server-based computing clusters for data processing, they need to configure the server accordingly based on the resources required for data processing. After the server is successfully configured, the relevant configuration information of the server is sent to the computing cluster. The computing cluster adds the server to the cluster queue based on the relevant configuration information of the server, and then performs tasks related to data processing.
[0003] However, since users are required to configure the server according to the resources required for data processing and are required to be familiar with a variety of big data components in order to complete the corresponding configuration and operation and maintenance work, there are problems with high deployment and operation and maintenance thresholds; at the same time, resources need to be expanded or reduced during business peak or off-peak periods, and the existing expansion and reduction methods require users to do it manually. Since the expansion and reduction process is complex and time-consuming, it leads to insufficient elasticity of the computing cluster. Summary of the invention
[0004] In order to solve the above technical problems, the present disclosure provides a data processing method, device, equipment and medium based on serverless computing.
[0005] A first aspect of an embodiment of the present disclosure provides a data processing method based on serverless computing, including:
[0006] Receiving a target instruction sent by a user, the target instruction is an instruction for executing a data processing task, and the target instruction includes resource configuration information required for executing the data processing task;
[0007] Perform resource allocation and scheduling operations based on resource configuration information to build a computing power pool;
[0008] After the computing power pool is built, a resource scheduling success message is sent to the user, and the data processing task corresponding to the target instruction is executed to obtain the data processing result corresponding to the data processing task.
[0009] A second aspect of an embodiment of the present disclosure provides a data processing device based on serverless computing, including:
[0010] An instruction receiving module is used to receive a target instruction sent by a user, where the target instruction is an instruction for executing a data processing task, and the target instruction includes resource configuration information required for executing the data processing task;
[0011] The computing power pool construction module is used to perform resource allocation and scheduling operations based on resource configuration information to build a computing power pool;
[0012] The data processing module is used to send a resource scheduling success message to the user after the computing power pool is built, and execute the data processing task corresponding to the target instruction to obtain the data processing result corresponding to the data processing task.
[0013] A third aspect of the embodiments of the present disclosure provides an electronic device, including:
[0014] processor;
[0015] A memory for storing executable instructions;
[0016] Among them, the processor is used to read executable instructions from the memory and execute the executable instructions to implement the data processing method based on serverless computing provided by the first aspect above.
[0017] A fourth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the data processing method based on serverless computing provided by the first aspect above.
[0018] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0019] The data processing method, device, equipment and medium based on serverless computing provided by the embodiments of the present disclosure can receive a target instruction sent by a user, wherein the target instruction is an instruction for executing a data processing task, and the target instruction includes resource configuration information required for executing the data processing task. After receiving the target instruction, the target instruction is parsed to obtain the resource configuration information, and resource allocation and scheduling operations are performed based on the resource configuration information to build a computing power pool. Then, after the computing power pool is built, a resource scheduling success message is sent to the user, and the data processing task corresponding to the target instruction is executed to obtain the data processing result corresponding to the data processing task. Thus, after receiving the target instruction sent by the user, resource allocation and scheduling are performed to build a computing power pool, and after building the computing power pool, the data processing task is executed, that is, the user can perform resource allocation and scheduling by sending instructions, without manually configuring the server according to the resources required for the data processing task, which reduces the threshold for resource deployment and operation and maintenance, and at the same time, improves the elasticity of the computing cluster. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0022] Figure 1 is a flow chart of a data processing method based on serverless computing provided by an embodiment of the present disclosure;
[0023] Figure 2 is a schematic diagram of an application scenario provided by an embodiment of the present disclosure;
[0024] Figure 3 is a flow chart of another data processing method based on serverless computing provided by an embodiment of the present disclosure;
[0025] Figure 4 is a structural schematic diagram of a data processing device based on serverless computing provided by an embodiment of the present disclosure;
[0026] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0027] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0028] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0029] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0030] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0031] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0032] Normally, the existing traditional server computing cluster is used for data processing. Since users need to configure the server according to the resources required for data processing, they need to be familiar with a variety of big data components to complete the corresponding configuration and operation and maintenance work, which has the problem of high deployment and operation and maintenance thresholds; at the same time, resources need to be expanded or reduced during business peak or off-peak periods, and the existing expansion and reduction methods require users to do it manually. Since the expansion and reduction process is complex and time-consuming, it leads to the problem of insufficient elasticity of the computing cluster. In response to this problem, the embodiments of the present disclosure provide a data processing method based on serverless computing, which is introduced in conjunction with specific embodiments below.
[0033] Figure 1 The flowchart of a data processing method based on serverless computing provided by an embodiment of the present disclosure is shown in FIG. 1 . The method can be executed by a data processing device based on serverless computing. The data processing device based on serverless computing can be implemented in software and / or hardware. The data processing device based on serverless computing can be configured in an electronic device, such as a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc. In addition, the method can be applied to Figure 2 The application scenario shown includes a user end 10, a server end 20 and a data storage system 30. It can be understood that the data processing method based on serverless computing provided in the embodiment of the present disclosure can also be applied in other scenarios.
[0034] like Figure 2As shown, the user end 10 is used for the user to send a request or instruction for executing a data processing task to the server end 20, wherein the request or instruction includes resource configuration information required for executing the data processing task.
[0035] The client end 10 includes an application Client and a program interface SDK. The client end 10 sends a script in the form of Spark-submit to the server end 20 based on the application Client or the program interface SDK.
[0036] The server 20 is used to process the data processing request or instruction sent by the user, including the resource allocation and scheduling operations required to execute the data processing task and the data processing operation corresponding to the data processing task.
[0037] The server 20 includes Web UI, Spark operator, Volcano / yunikom and Kubernetes cluster.
[0038] The Web UI is a console interface used to display resource status such as resource distribution status and resource usage status.
[0039] Sparkoperator and Volcano / yunikom are components used for resource allocation and deployment. The server can use either Sparkoperator or Volcano / yunikom to allocate and deploy resources according to specific usage scenarios or usage requirements.
[0040] Kubernetes cluster is specifically used to manage resources.
[0041] The data storage system 30 is used to store data, including the storage of data to be processed and the storage of processed data.
[0042] The data storage system 30 includes Lceberg, Hudi, Delta, Hadoop HDFS and KS3.
[0043] Among them, Lceberg, Hudi, Delta, and Hadoop HDFS are components used for data storage respectively. Different components are selected for data storage based on different application scenarios, usage requirements, and data types.
[0044] KS3 is a massive, low-cost, highly secure, and highly reliable distributed cloud storage service provided by Kingsoft Cloud. Users can easily store and retrieve various data files such as pictures, audio, video, and text.
[0045] Combine the following Figure 2The application scenario shown is Figure 1 The data processing method based on serverless computing is introduced as shown in the figure. For example, Figure 2 The server 20 in can execute this method.
[0046] like Figure 1 As shown, the data processing method based on serverless computing provided in this embodiment includes the following steps.
[0047] S110, receiving a target instruction sent by a user, where the target instruction is an instruction for executing a data processing task, and the target instruction includes resource configuration information required for executing the data processing task.
[0048] In the embodiment of the present disclosure, the target instruction may be sent by the user end to the electronic device (such as the server end 20) in the form of a script.
[0049] Resource configuration information can be understood as the hardware resources (such as CPU resources, number of cores, memory resources, etc.) and software resources (such as operating system, language processing program, etc.) required to perform data processing tasks.
[0050] Specifically, the electronic device can receive the target instruction sent by the user in real time based on a preset interface.
[0051] S120. Execute resource allocation and scheduling operations based on resource configuration information to build a computing power pool.
[0052] In the embodiments of the present disclosure, the computing power pool can be understood as a resource pool containing many resources, which integrates a large number of computing resources such as CPU, GPU, etc. to form a shared, scalable, and elastically allocated computing resource pool. This resource is encapsulated and managed so that it can be dynamically allocated to different users or perform different data processing tasks as needed.
[0053] The computing power pool may be composed of a variety of computing resources, wherein the computing power pool may include bare metal servers and / or cloud hosts.
[0054] Specifically, after receiving the target instruction, the electronic device can parse the target instruction to obtain the resource configuration information corresponding to the target instruction. The type and quantity of computing resources are determined according to the resource configuration information, and the computing power pool is constructed based on the type and quantity of computing resources with bare metal servers and / or cloud hosts as the base and K8s resource orchestration capabilities.
[0055] Among them, the specific implementation method of building a computing power pool based on K8s resource orchestration capabilities is similar to the specific implementation method of the existing K8s execution computing power pool construction, which will not be repeated here.
[0056] S130. After the computing power pool is built, a resource scheduling success message is sent to the user, and the data processing task corresponding to the target instruction is executed to obtain the data processing result corresponding to the data processing task.
[0057] In the embodiments of the present disclosure, the data processing tasks may include tasks for performing data cleaning operations, or tasks for performing other computational processing on the data, which are not limited herein.
[0058] Specifically, after building a computing power pool, the electronic device can deploy the applications that need to be run to the computing power pool, test the applications deployed in the computing power pool to ensure that the applications can run normally and meet performance requirements, and send a resource scheduling success message to the user if the test is successful; if the test fails, adjust and optimize the computing power pool until the test is successful, and then send a resource scheduling success message to the user; thereafter, perform task allocation based on the computing resources in the computing power pool, and execute data processing tasks corresponding to the target instructions to obtain data processing results.
[0059] In the disclosed embodiment, it is possible to receive a target instruction sent by a user, wherein the target instruction is an instruction for executing a data processing task, and the target instruction includes resource configuration information required for executing the data processing task. After receiving the target instruction, the target instruction is parsed to obtain the resource configuration information, and resource allocation and scheduling operations are performed based on the resource configuration information to build a computing power pool. Then, after the computing power pool is built, a resource scheduling success message is sent to the user, and the data processing task corresponding to the target instruction is executed to obtain the data processing result corresponding to the data processing task. Thus, after receiving the target instruction sent by the user, resource allocation and scheduling are performed to build a computing power pool, and after building the computing power pool, the data processing task is executed, that is, the user can perform resource allocation and scheduling by sending instructions, without manually configuring the server according to the resources required for the data processing task, which reduces the threshold for resource deployment and operation and maintenance, and at the same time, improves the elasticity of the computing cluster.
[0060] On the basis of the above-mentioned embodiments of the present disclosure, when executing data processing, a product form based on serverless computing is used, which shields the underlying basic components and provides simple interface, script and console usage. Users only need to understand the usage of open source computing systems (such as Spark) to develop big data business, thus achieving a low usage threshold. At the same time, data processing tasks only need to be managed through the interface, without the need for server configuration and cluster configuration, and without the need for operation and maintenance operations such as expansion and reduction, thus achieving zero operation and maintenance. Users do not need to pay for management and control resources, nor do they need to pay for computing resources that are idle during off-peak periods, thus reducing resource usage costs and improving user experience.
[0061] In an embodiment of the present disclosure, before executing a data processing task corresponding to a target instruction, the data processing method based on serverless computing may further include: acquiring the to-be-processed data required to execute the data processing task from a preset data storage system based on a target interface.
[0062] In the embodiment of the present disclosure, the target interface may be a pre-set interface for interacting with the data storage system.
[0063] Data storage systems can support different storage types of data.
[0064] In some embodiments of the present disclosure, before executing a data processing task corresponding to a target instruction, the electronic device determines data identification information required for the data processing task based on the target instruction, and extracts data to be processed from a preset data storage system according to the data identification information based on a target interface.
[0065] In some other embodiments of the present disclosure, before executing the data processing task corresponding to the target instruction, the electronic device determines the data type and / or data attribute information required for the data processing task based on the target instruction, obtains data matching the data type and / or data attribute information from the target website based on the data type and / or data attribute information, and determines it as data to be processed.
[0066] The target instruction also includes a calculation task script, wherein the calculation task script may include information such as calculation commands, equations or formulas required to perform the data processing task.
[0067] Furthermore, executing the data processing task corresponding to the target instruction may specifically include: executing the data processing operation corresponding to the data processing task based on the computing task script and the data to be processed, and obtaining the data processing result.
[0068] Specifically, after acquiring the data to be processed required to execute the target instruction, the electronic device parses the computing task script to obtain the parsing result, performs a data processing operation based on the parsing result and the data to be processed, and then obtains the data processing result.
[0069] In the disclosed embodiment, the server side that performs data processing tasks can be separated from the data storage system, and the data to be processed can be extracted from the data storage system through an interface, thereby avoiding the waste of resources and reduced utilization caused by the existing server computing cluster that integrates computing and storage, that is, the strong coupling of storage and computing, and improving the convenience of capacity expansion. At the same time, the computing task script is carried in the target instruction, and then after the computing power pool is built, the data processing task can be directly executed based on the computing task script, which improves the execution efficiency of the data processing task.
[0070] In an embodiment of the present disclosure, after sending a resource scheduling success message to the user, the data processing method based on serverless computing may also include: receiving a task query script command sent by the user; executing the task query script command, obtaining the task execution status of the target task corresponding to the task query script command, and feeding back the task execution status to the user.
[0071] In some embodiments of the present disclosure, when a user needs to view the execution status of a task, he or she can use a preset interface such as Figure 2 The application Client and program interface SDK shown in the figure send task query script commands in the form of scripts. The electronic device can receive the task query script commands sent by the user in real time, and after receiving the task query script commands, parse the task query script commands, determine the target task to be queried, and obtain the task execution status of the target task, and then feedback the task execution status to the user.
[0072] In other embodiments of the present disclosure, when a user needs to view the execution status of a task, the user may access an electronic device (such as a server) based on a Web UI address to obtain the task execution status.
[0073] In the disclosed embodiment, the user can obtain the task execution status in the form of a script through a preset interface, and can also access and obtain the task execution status through a Web UI address, thereby improving the convenience of the user to view the task execution status and thereby improving the user experience.
[0074] In the embodiment of the present disclosure, the data processing operation includes a data cleaning operation.
[0075] Executing the data processing task corresponding to the target instruction may specifically include: executing a data cleaning operation on the to-be-processed data corresponding to the data processing task to obtain the cleaned target data.
[0076] Among them, the specific implementation method of performing the data cleaning operation is similar to the existing implementation method of performing the data cleaning operation, and will not be repeated here.
[0077] Furthermore, after obtaining the target data after cleaning, the data processing method based on serverless computing may also include: feeding back the target data to the user so that the user constructs a training data set based on the target data, performs training operations of the target machine learning model based on the training data set, and stores the target data in a preset data storage system.
[0078] Among them, the target machine learning model can be a generative machine learning model, etc., which is not limited here.
[0079] In the embodiments of the present disclosure, data cleaning operations can be performed on data based on serverless computing to obtain cleaned data, thereby improving the efficiency of data cleaning, providing support for subsequent users to construct training data sets based on the cleaned target data, and thus improving the convenience of model training.
[0080] Figure 3 is a flowchart of another data processing method based on serverless computing provided by an embodiment of the present disclosure. Figure 3 As shown, the data processing method may specifically include the following steps:
[0081] S310, receiving a target instruction sent by a user, where the target instruction is an instruction for executing a data processing task, and the target instruction includes resource configuration information required for executing the data processing task.
[0082] S320: Execute resource allocation and scheduling operations based on resource configuration information to build a computing power pool.
[0083] S330: After the computing power pool is built, a resource scheduling success message is sent to the user, and the unprocessed data required to execute the data processing task is obtained from the preset data storage system based on the target interface.
[0084] S340, executing the data processing operation corresponding to the data processing task based on the computing task script contained in the target instruction and the data to be processed, to obtain the data processing result.
[0085] S350: Feedback the data processing result to the user, and store the data processing result in a preset data storage system.
[0086] It should be noted that the specific implementation of steps S310-S350 is similar to the implementation of the relevant steps in the above-mentioned embodiment of the present disclosure, and will not be repeated here.
[0087] In the embodiments of the present disclosure, users can perform resource allocation and scheduling by sending instructions, without having to manually configure the server according to the resources required for the data processing task, thereby lowering the resource deployment and operation and maintenance thresholds, and at the same time improving the elasticity of the computing cluster. In addition, computing and storage are separated, avoiding the waste of resources and reduced utilization caused by the existing server computing cluster where computing and storage are integrated, i.e., storage and computing are strongly coupled, thereby improving the convenience of capacity expansion.
[0088] Figure 4 It is a structural diagram of a data processing device based on serverless computing provided in an embodiment of the present disclosure.
[0089] In the embodiments of the present disclosure, the data processing device based on serverless computing can be set in an electronic device and is understood as a part of the functional modules in the above electronic device. Specifically, the electronic device can be a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc., which is not limited here.
[0090] like Figure 4 As shown, the data processing device 400 based on serverless computing may include an instruction receiving module 410 , a computing power pool building module 420 and a data processing module 430 .
[0091] The instruction receiving module 410 may be used to receive a target instruction sent by a user. The target instruction is an instruction for executing a data processing task. The target instruction includes resource configuration information required for executing the data processing task.
[0092] The computing power pool construction module 420 can be used to perform resource allocation and scheduling operations based on resource configuration information to build a computing power pool.
[0093] The data processing module 430 can be used to send a resource scheduling success message to the user after the computing power pool is built, and execute the data processing task corresponding to the target instruction to obtain the data processing result corresponding to the data processing task.
[0094] In the disclosed embodiment, it is possible to receive a target instruction sent by a user, wherein the target instruction is an instruction for executing a data processing task, and the target instruction includes resource configuration information required for executing the data processing task. After receiving the target instruction, the target instruction is parsed to obtain the resource configuration information, and resource allocation and scheduling operations are performed based on the resource configuration information to build a computing power pool. Then, after the computing power pool is built, a resource scheduling success message is sent to the user, and the data processing task corresponding to the target instruction is executed to obtain the data processing result corresponding to the data processing task. Thus, after receiving the target instruction sent by the user, resource allocation and scheduling are performed to build a computing power pool, and after building the computing power pool, the data processing task is executed, that is, the user can perform resource allocation and scheduling by sending instructions, without manually configuring the server according to the resources required for the data processing task, which reduces the threshold for resource deployment and operation and maintenance, and at the same time, improves the elasticity of the computing cluster.
[0095] In some embodiments of the present disclosure, the computing power pool is composed of a variety of computing resources, including bare metal servers and / or cloud hosts.
[0096] In some embodiments of the present disclosure, the data processing device 400 based on serverless computing may further include a data acquisition module.
[0097] The data acquisition module can be used to acquire the to-be-processed data required for executing the data processing task from a preset data storage system based on a target interface before executing the data processing task corresponding to the target instruction.
[0098] In some embodiments of the present disclosure, the target instruction also includes a computing task script.
[0099] The data processing module 430 may be specifically used to execute data processing operations corresponding to the data processing task based on the computing task script and the data to be processed, and obtain a data processing result.
[0100] In some embodiments of the present disclosure, the data processing device 400 based on serverless computing may further include a status query module.
[0101] The status query module can be used to receive the task query script command sent by the user after sending a resource scheduling success message to the user;
[0102] Execute the task query script command, obtain the task execution status of the target task corresponding to the task query script command, and feedback the task execution status to the user.
[0103] In some embodiments of the present disclosure, the data processing module 430 may be specifically used to perform a data cleaning operation on the to-be-processed data corresponding to the data processing task, and obtain cleaned target data.
[0104] In some embodiments of the present disclosure, the data processing device 400 based on serverless computing may further include a data storage module.
[0105] The data storage module can be used to feed back the target data to the user after executing the data processing task corresponding to the target instruction and obtaining the data processing result corresponding to the data processing task, so that the user can build a training data set based on the target data, perform training operations of the target machine learning model based on the training data set, and store the target data in a preset data storage system.
[0106] It should be noted that Figure 4 The data processing device 400 based on serverless computing shown can execute each step in the above method embodiment and realize each process and effect in the above method embodiment, which will not be elaborated here.
[0107] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure.
[0108] In the disclosed embodiment, Figure 5 The electronic device shown may be a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc., which is not limited here.
[0109] like Figure 5 As shown, the electronic device may include a processor 510 and a memory 520 storing computer program instructions.
[0110] Specifically, the processor 510 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present disclosure.
[0111] The memory 520 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 520 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 520 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 520 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 520 is a non-volatile solid-state memory. In a particular embodiment, the memory 520 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.
[0112] The processor 510 reads and executes the computer program instructions stored in the memory 520 to perform the steps of the data processing method based on serverless computing provided in the embodiment of the present disclosure.
[0113] In one example, the electronic device may further include a transceiver 530 and a bus 540. Figure 5 As shown, the processor 510, the memory 520 and the transceiver 530 are connected via a bus 540 and communicate with each other.
[0114] The bus 540 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a Memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 540 may include one or more buses.
[0115] The embodiments of the present disclosure also provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the data processing method based on serverless computing provided by the embodiments of the present disclosure.
[0116] The above-mentioned storage medium may, for example, include a memory 520 of computer program instructions, and the above-mentioned instructions may be executed by the processor 510 of the electronic device to complete the data processing method based on serverless computing provided by the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (CompactDiscROM, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.
[0117] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data processing method based on serverless computing, characterized in that: include: Receiving a target instruction sent by a user, wherein the target instruction is an instruction for executing a data processing task, and the target instruction includes resource configuration information required for executing the data processing task; Perform resource allocation and scheduling operations based on the resource configuration information to build a computing power pool; After the computing power pool is built, a resource scheduling success message is sent to the user, and the data processing task corresponding to the target instruction is executed to obtain the data processing result corresponding to the data processing task.
2. The method according to claim 1, characterized in that The computing power pool is composed of a variety of computing resources, including bare metal servers and / or cloud hosts.
3. The method according to claim 1, characterized in that Before executing the data processing task corresponding to the target instruction, the method further includes: The data to be processed required for executing the data processing task is obtained from a preset data storage system based on the target interface.
4. The method according to claim 3, characterized in that The target instruction also includes a calculation task script; The executing of the data processing task corresponding to the target instruction includes: The data processing operation corresponding to the data processing task is performed based on the computing task script and the data to be processed to obtain the data processing result.
5. The method according to claim 1, characterized in that After sending the resource scheduling success message to the user, the method further includes: Receiving a task query script command sent by the user; Execute the task query script command, obtain the task execution status of the target task corresponding to the task query script command, and feed back the task execution status to the user.
6. The method according to claim 1, characterized in that The executing of the data processing task corresponding to the target instruction includes: Execute a data cleaning operation on the data to be processed corresponding to the data processing task to obtain cleaned target data; After executing the data processing task corresponding to the target instruction and obtaining the data processing result corresponding to the data processing task, the method further includes: The target data is fed back to the user so that the user constructs a training data set based on the target data, performs a training operation of a target machine learning model based on the training data set, and stores the target data in a preset data storage system.
7. A data processing device based on serverless computing, characterized in that: include: An instruction receiving module, used to receive a target instruction sent by a user, wherein the target instruction is an instruction for executing a data processing task, and the target instruction includes resource configuration information required for executing the data processing task; A computing power pool construction module, used to perform resource allocation and scheduling operations based on the resource configuration information to construct a computing power pool; The data processing module is used to send a resource scheduling success message to the user after the computing power pool is built, and execute the data processing task corresponding to the target instruction to obtain the data processing result corresponding to the data processing task.
8. The device according to claim 7, characterized in that The computing power pool is composed of a variety of computing resources, including bare metal servers and / or cloud hosts.
9. An electronic device, characterized in that: include: processor; A memory for storing executable instructions; Wherein, the processor is used to read the executable instructions from the memory and execute the executable instructions to implement the data processing method based on serverless computing as described in any one of claims 1-6 above.
10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the processor implements the data processing method based on serverless computing as described in any one of claims 1 to 6.