A data processing task execution method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a method, device, electronic device and storage medium for executing a data processing task. The method includes: obtaining a data processing task to be executed submitted by a user, and determining the target tenant to which the user belongs; determining a target data operating environment for executing the data processing task based on the correspondence between the target tenant and the preset tenant and the pre-built data operating environment, wherein the data operating environment includes pre-virtualized storage and computing resources, and virtual isolation between actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants; calling the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task. Thereby, the complexity and difficulty of executing the data processing task are reduced, and the resource utilization rate is improved.

Description

Technical Field

[0001] The present application relates to the field of resource management technology, and in particular to a method, device, electronic device and storage medium for executing a data processing task. Background Art

[0002] For each data processing task to be processed, it is usually necessary to call computing resources and storage resources so that the application can use the called computing resources and storage resources to perform the data processing task. Among them, computing resources can include distributed resource management Yarn, K8s (Kubernetes), physical servers, etc.; storage resources can include databases, object storage, cloud disks, etc.

[0003] When using applications to perform data processing tasks, since various types of computing resources and storage resources are managed independently, different resources are used in different ways. Users need to use each type of resource separately according to the different ways of using each resource. The complexity and difficulty of using applications to perform tasks are high, which reduces the utilization rate of various resources. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for executing data processing tasks, so as to reduce the complexity and difficulty of executing data processing tasks and improve resource utilization. The specific technical solutions are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for executing a data processing task, the method comprising:

[0006] Obtaining the data processing task to be executed submitted by the user, and determining the target tenant to which the user belongs;

[0007] Based on the correspondence between the target tenant and the preset tenant and the pre-built data operating environment, determine the target data operating environment for executing the data processing task, wherein the data operating environment includes pre-virtualized storage and computing resources, and the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants are virtually isolated from each other;

[0008] The actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment are called to execute the data processing task.

[0009] Optionally, before the step of calling the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task, the method further includes:

[0010] Displaying pre-virtualized storage and computing resources included in the target data operating environment, wherein the storage and computing resources include computing resources and storage resources;

[0011] In the case of obtaining a resource selection instruction issued by a user based on the displayed storage and computing resources, determining the target storage resource selected by the user from the storage resources and the target computing resource selected by the user from the computing resources, wherein the target storage resource at least includes an output storage resource for storing the processing result of the data processing task;

[0012] The step of calling the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task includes:

[0013] Obtaining the data to be processed corresponding to the data processing task;

[0014] Calling the actual computing resource corresponding to the target computing resource to process the data to be processed to obtain a processing result;

[0015] The processing result is stored in the actual storage resource corresponding to the output storage resource.

[0016] Optionally, the target storage resource further includes an input storage resource for storing the to-be-processed data corresponding to the data processing task;

[0017] The step of obtaining the to-be-processed data corresponding to the data processing task includes:

[0018] The to-be-processed data corresponding to the data processing task is obtained from the actual storage resources corresponding to the input storage resources.

[0019] Optionally, the manner of constructing the correspondence between the preset tenant and the pre-built data operating environment includes:

[0020] Obtain resource information of each actual storage and computing resource;

[0021] For each tenant, determine the actual storage and computing resources required by the tenant based on the tenant's corresponding business needs;

[0022] For each tenant, according to the preset virtualization rules and based on the resource information, the actual storage and computing resources required by the tenant are virtualized to obtain a virtualized storage and computing resource set as the resource pool corresponding to the tenant;

[0023] From the resource pool of the tenant, virtualized storage and computing resources are imported into the data operating environment corresponding to the tenant, so as to obtain the corresponding relationship between the preset tenant and the pre-built data operating environment.

[0024] Optionally, the pre-virtualized storage and computing resources include pre-virtualized distributed computing resources, pre-virtualized server computing resources, and pre-virtualized storage resources;

[0025] The preset virtualization rules of the distributed computing resources include: the virtualized distributed computing resources include multiple virtualization queues, and for each distributed computing resource cluster, the virtualization queue corresponding to the cluster includes a tenant queue, a cluster queue, a data operating environment queue, and a custom queue;

[0026] The preset virtualization rules of the server computing resources include: the virtualized server computing resources include data integration instances and server local queues, wherein the data integration instance is a resource required to extract the data to be processed from the source end, convert the data to be processed and load it to the destination end, and the data integration instance is represented by the instance name, CPU specification, memory specification and the server to which it belongs; the server local queue is represented by the queue name, CPU specification and memory specification;

[0027] The preset virtualization rules of the storage resources include: the virtualized storage resources are represented by storage resource names, storage resource types, storage resource URL connection strings, authentication methods, and attribute configuration information.

[0028] Optionally, the preset virtualization rules of the distributed computing resources specifically include:

[0029] The virtualized distributed computing resources include multiple virtualization queues. For each distributed computing resource cluster, the virtualization queue corresponding to the cluster has the tenant queue as the first-level parent queue, the cluster queue as the second-level parent queue, the data operating environment queue as the third-level queue, and the custom queue as the fourth-level queue; the custom queue is represented by the queue name, CPU specifications, and memory specifications of the actual queue corresponding to the custom queue.

[0030] Optionally, the method further includes:

[0031] After obtaining an update instruction for the data operating environment, the update instruction is used to instruct to create a new data operating environment for the target tenant or to expand the existing data operating environment;

[0032] In a case where the update instruction is used to instruct to create a new data operating environment for the target tenant, determining whether the number of data operating environments corresponding to the target tenant reaches a first preset number, and in a case where the number of data operating environments corresponding to the target tenant does not reach the first preset number, creating a new data operating environment for the target tenant;

[0033] In the case where the update instruction is used to indicate the expansion of an existing data operating environment, it is determined whether the number of pre-virtualized storage and computing resources included in the existing data operating environment has reached a second preset number, and if the number of pre-virtualized storage and computing resources included in the existing data operating environment has not reached the second preset number, the existing data operating environment is expanded.

[0034] In a second aspect, an embodiment of the present application provides a data processing task execution device, the device comprising:

[0035] A task acquisition module is used to acquire the data processing tasks to be executed submitted by the user and determine the target tenant to which the user belongs;

[0036] A data operating environment determination module, used to determine a target data operating environment for executing the data processing task based on the correspondence between the target tenant and the preset tenant and the pre-built data operating environment, wherein the data operating environment includes pre-virtualized storage and computing resources, and actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants are virtually isolated from each other;

[0037] The task execution module is used to call the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task.

[0038] Optionally, the device further comprises:

[0039] A resource display module, used for displaying the pre-virtualized storage and computing resources included in the target data operating environment before the step of calling the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task, wherein the storage and computing resources include computing resources and storage resources;

[0040] A resource determination module, configured to determine, upon obtaining a resource selection instruction issued by a user based on the displayed storage and computing resources, a target storage resource selected by the user from the storage resources and a target computing resource selected by the user from the computing resources, wherein the target storage resource at least includes an output storage resource for storing a processing result of the data processing task;

[0041] The task execution module comprises:

[0042] A data acquisition unit, used to acquire the to-be-processed data corresponding to the data processing task;

[0043] A resource calling unit, used to call the actual computing resource corresponding to the target computing resource, process the data to be processed, and obtain a processing result;

[0044] A storage unit, used for storing the processing result in an actual storage resource corresponding to the output storage resource;

[0045] The target storage resource also includes an input storage resource for storing the to-be-processed data corresponding to the data processing task; the data acquisition unit includes:

[0046] A data acquisition subunit, used to acquire the to-be-processed data corresponding to the data processing task from the actual storage resources corresponding to the input storage resources;

[0047] The device further includes a corresponding relationship building module, and the corresponding relationship building module includes:

[0048] An information acquisition unit, used to acquire resource information of each actual storage and computing resource;

[0049] A resource determination unit, used to determine, for each tenant, the actual storage and computing resources required by the tenant according to the business requirements of the tenant;

[0050] A virtualization unit is used to virtualize the actual storage and computing resources required by each tenant according to preset virtualization rules and based on the resource information to obtain a virtualized storage and computing resource set as a resource pool corresponding to the tenant;

[0051] A resource import unit, used to import virtualized storage and computing resources from the resource pool of the tenant into the constructed data operating environment corresponding to the tenant, so as to obtain a correspondence between the preset tenant and the pre-constructed data operating environment;

[0052] The pre-virtualized storage and computing resources include pre-virtualized distributed computing resources, pre-virtualized server computing resources and pre-virtualized storage resources;

[0053] The preset virtualization rules of the distributed computing resources include: the virtualized distributed computing resources include multiple virtualization queues, and for each distributed computing resource cluster, the virtualization queue corresponding to the cluster includes a tenant queue, a cluster queue, a data operating environment queue, and a custom queue;

[0054] The preset virtualization rules of the server computing resources include: the virtualized server computing resources include data integration instances and server local queues, wherein the data integration instance is a resource required to extract the data to be processed from the source end, convert the data to be processed and load it to the destination end, and the data integration instance is represented by the instance name, CPU specification, memory specification and the server to which it belongs; the server local queue is represented by the queue name, CPU specification and memory specification;

[0055] The preset virtualization rules of the storage resource include: the virtualized storage resource is represented by the storage resource name, storage resource type, storage resource URL connection string, authentication method and attribute configuration information;

[0056] The preset virtualization rules of the distributed computing resources specifically include:

[0057] The virtualized distributed computing resources include multiple virtualized queues. For each distributed computing resource cluster, the virtualized queue corresponding to the cluster uses the tenant queue as the first-level parent queue, the cluster queue as the second-level parent queue, the data operating environment queue as the third-level queue, and the custom queue as the fourth-level queue; the custom queue is represented by the queue name, CPU specification, and memory specification of the actual queue corresponding to the custom queue;

[0058] The device also includes:

[0059] An instruction acquisition module is used to acquire an update instruction for the data operating environment, wherein the update instruction is used to instruct to create a new data operating environment for the target tenant or to expand the existing data operating environment;

[0060] an environment creation module, configured to determine whether the number of data operating environments corresponding to the target tenant reaches a first preset number when the update instruction is used to instruct to create a new data operating environment for the target tenant, and to create a new data operating environment for the target tenant when the number of data operating environments corresponding to the target tenant does not reach the first preset number;

[0061] An expansion module is used to determine whether the number of pre-virtualized storage and computing resources included in the existing data operating environment has reached a second preset number when the update instruction is used to indicate the expansion of the existing data operating environment, and to expand the existing data operating environment when the number of pre-virtualized storage and computing resources included in the existing data operating environment has not reached the second preset number.

[0062] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0063] Memory, used to store computer programs;

[0064] The processor is used to implement any of the data processing task execution methods described in the first aspect when executing the program stored in the memory.

[0065] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing task execution method described in any one of the first aspects above is implemented.

[0066] In a fifth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the data processing task execution methods described in the first aspect above.

[0067] Beneficial effects of the embodiments of the present application:

[0068] In the solution provided by the embodiment of the present application, the electronic device can obtain the data processing task to be executed submitted by the user, and determine the target tenant to which the user belongs; based on the correspondence between the target tenant and the preset tenant and the pre-built data operating environment, determine the target data operating environment for executing the data processing task, wherein the data operating environment includes pre-virtualized storage and computing resources, and the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants are virtually isolated; call the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task.

[0069] Since the data operation environment corresponding to each tenant is pre-built, and the pre-virtualized storage and computing resources included in the data operation environment correspond to actual storage and computing resources, when the user wants to perform a data processing task, he only needs to submit the data processing task, and the electronic device can directly call the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the data operation environment corresponding to the tenant to which the user belongs to perform the data processing task. Since there is no need for the user to use various types of resources to perform data processing tasks according to the usage of various types of storage and computing resources, the complexity and difficulty of executing tasks are reduced, and the resource utilization rate is improved. And since each tenant corresponds to a different data operation environment, the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in each data operation environment are also different, that is, the actual storage and computing resources actually used to perform data processing tasks of different tenants are different, and virtual isolation between the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants is achieved. In addition, by constructing each data operation environment, the storage and computing resources can be divided into fine granularity, further improving the resource utilization rate. Of course, it is not necessary to achieve all the advantages described above at the same time for implementing any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0071] Figure 1 A flowchart of a method for executing a data processing task provided in an embodiment of the present application;

[0072] Figure 2 A schematic diagram of a data operating environment model provided in an embodiment of the present application;

[0073] Figure 3 Another flowchart of the data processing task execution method provided in the embodiment of the present application;

[0074] FIG4 (a) is a schematic diagram of a display interface for displaying virtualized storage and computing resources included in a DRE provided in an embodiment of the present application;

[0075] FIG4( b ) is a schematic diagram of a display interface for computing resource details information provided in an embodiment of the present application;

[0076] FIG4 (c) is a schematic diagram of another display interface for computing resource details information provided in an embodiment of the present application;

[0077] Figure 5 A flowchart of a method for constructing a correspondence between a tenant and a pre-built data operating environment provided in an embodiment of the present application;

[0078] FIG. 6 (a) is a schematic diagram of an example of a process of constructing a correspondence relationship between a tenant and a pre-built data operating environment provided in an embodiment of the present application;

[0079] FIG6 (b) is another schematic diagram of an example of a process of constructing a correspondence relationship between a tenant and a pre-built data operating environment provided in an embodiment of the present application;

[0080] FIG. 7 (a) is a schematic diagram of a visual interface for editing a distributed resource queue provided in an embodiment of the present application;

[0081] FIG7( b ) is a schematic diagram of a visual interface for editing an ETL instance provided in an embodiment of the present application;

[0082] FIG7( c ) is a schematic diagram of a visual interface for editing a local queue of a server provided in an embodiment of the present application;

[0083] FIG7( d ) is a schematic diagram of storage resources provided in an embodiment of the present application;

[0084] Figure 8 A schematic diagram of the management process of the data operating environment provided in the embodiment of the present application;

[0085] Fig. 9 A schematic diagram of a data processing task execution device provided in an embodiment of the present application;

[0086] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0088] In order to reduce the complexity and difficulty of executing data processing tasks and improve resource utilization, the present application provides a data processing task execution method, device, electronic device, computer-readable storage medium and computer program product. The following is first an introduction to a data processing task execution method provided in an embodiment of the present application.

[0089] like Figure 1 As shown, the method includes:

[0090] S101: Acquire a data processing task to be executed submitted by a user, and determine a target tenant to which the user belongs.

[0091] S102: Determine a target data operating environment for executing the data processing task based on the correspondence between the target tenant and the preset tenant and the pre-built data operating environment.

[0092] The data operating environment includes pre-virtualized storage and computing resources, and actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants are virtually isolated from each other.

[0093] S103: Calling actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task.

[0094] It can be seen that in the solution provided by the embodiment of the present application, the electronic device can obtain the data processing task to be executed submitted by the user, and determine the target tenant to which the user belongs; based on the correspondence between the target tenant and the preset tenant and the pre-built data operating environment, determine the target data operating environment for executing the data processing task, wherein the data operating environment includes pre-virtualized storage and computing resources, and the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants are virtually isolated; call the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task.

[0095] Since a data operation environment corresponding to each tenant is pre-built, and the pre-virtualized storage and computing resources included in the data operation environment correspond to actual storage and computing resources, when the user wants to perform a data processing task, he only needs to submit the data processing task, and the electronic device can directly call the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the data operation environment corresponding to the tenant to which the user belongs to perform the data processing task. Since there is no need for the user to use various types of resources to perform data processing tasks according to the usage methods of various types of storage and computing resources, the complexity and difficulty of executing tasks are reduced, and the resource utilization rate is improved. And since each tenant corresponds to a different data operation environment, the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in each data operation environment are also different, that is, the actual storage and computing resources actually used to execute data processing tasks of different tenants are different, and virtual isolation between the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants is achieved. In addition, by constructing each data operation environment, the storage and computing resources are fine-grained, further improving the resource utilization rate.

[0096] In order to improve the execution efficiency of data processing tasks related to various businesses, a data runtime environment (DRE) for executing the data processing tasks required by the tenant can be created in advance for the preset tenants corresponding to each business, and a corresponding relationship between the tenant and the data processing tasks created for the tenant can be established.

[0097] The data operation environment may include pre-virtualized storage and computing resources. For example, the data operation environment includes pre-virtualized computing resources and storage resources such as data source information, distributed computing queues, and physical servers. A unified resource virtualization description method is defined in advance for different computing resources and storage resources. By virtualizing storage and computing resources, the virtualized computing resources and virtualized storage resources required for each tenant to perform data processing tasks are loaded into a data operation environment. When executing data processing tasks, the application program only needs to submit the data processing tasks to the data operation environment, which improves convenience and efficiency.

[0098] For each tenant, by pre-building a data operating environment for the tenant to execute the tenant's data processing tasks, the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the data operating environment corresponding to the tenant can be virtually isolated from other actual storage and computing resources. By creating different data operating environments for different tenants, virtual isolation between the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants can be achieved.

[0099] In the above step S101, the electronic device may obtain the data processing task to be executed submitted by the user, and determine the target tenant to which the user belongs.

[0100] When a data processing task is desired to be executed, the user may submit the data processing task to be executed, so that the electronic device may obtain the data processing task to be executed submitted by the user and determine the target tenant to which the user belongs.

[0101] The services corresponding to different tenants are different, and the corresponding users belonging to the tenants are users who process the services corresponding to the tenants.

[0102] As an implementation method, the electronic device may record the business identifier corresponding to each tenant, and when the user submits a data processing task to be executed, the data processing task sent may carry the business identifier corresponding to the user. In this way, the electronic device may receive the business identifier corresponding to the user sent by the user, and determine the target tenant to which the user belongs based on the business identifier corresponding to the user and the business identifiers corresponding to each tenant stored in the electronic device.

[0103] Furthermore, in the above step S102, the electronic device can determine the target data operating environment for executing the data processing task based on the correspondence between the target tenant and the preset tenant and the pre-built data operating environment. The data operating environment includes pre-virtualized storage and computing resources, and the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants are virtually isolated from each other.

[0104] Since the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants are virtually isolated, for each tenant, the actual storage and computing resources used to execute data processing tasks of each user belonging to the tenant are fixed, that is, the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the data operating environment corresponding to the tenant.

[0105] Based on this, after determining the target tenant to which the user belongs, the electronic device can search for the target data operating environment corresponding to the target tenant in the correspondence between the preset tenant and the pre-built data operating environment. The found target data operating environment is the data operating environment used to execute the data processing task submitted by the user.

[0106] In the above step S103, the electronic device can call the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to perform the data processing task.

[0107] When determining the target data operating environment for executing the data processing task submitted by the user, the electronic device can call the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment, and use the above-mentioned actual storage resources to execute the data processing task.

[0108] The actual storage and computing resources include storage resources and computing resources. When executing the data processing task, the computing resources are used to calculate the data to be calculated in the data processing task to obtain the processing result of the data processing task; the storage resources are at least used to store the processing result of the data processing task.

[0109] For example, Figure 2 As shown, the running resource pool may include the actual storage and computing resources required to perform data processing tasks, that is, the storage resources required for data query and data warehousing during the execution of data processing tasks, the distributed computing resources required for big data computing, and the server computing resources required for local execution of batch tasks.

[0110] For different tenants, the corresponding data processing environment DRE is pre-built according to the specific business scope of the tenant. By virtualizing the actual storage and computing resources in the operation resource pool, the computing resources and storage resources used to execute the data processing tasks in the specific business scope of each tenant are virtually encapsulated, and the virtualized computing resources and storage resources constitute the data operation environment corresponding to the tenant.

[0111] In this way, for tenants 1 to N, a data operating environment corresponding to the tenant is pre-built for each tenant. That is, a data operating environment DRE1 is pre-built for tenant 1, a data operating environment DRE2 is pre-built for tenant 2, and a data operating environment DREN is pre-built for tenant N, thereby obtaining a corresponding relationship between each tenant and the pre-built data operating environment.

[0112] Each data operation environment includes virtualized storage resources and computing resources. For example, data operation environment DRE1 includes storage resources and computing resources. The storage resources include: Hbase, HDFS (Hadoop Distributed FileSystem), Kudu and other big data storage resources; greenplum, Haiyan and other distributed storage resources; and other storage media and object storage. Computing resources include bare metal, virtual machines, physical servers, cloud servers and other server computing resources, as well as distributed computing resources such as Yarn queues.

[0113] The data processing task is a data development task, which may include data integration and data calculation. For each tenant, the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the data operating environment corresponding to the tenant can be used to execute the data development tasks of each user belonging to the tenant.

[0114] When receiving a data development task submitted by a user belonging to tenant 1 to the data operating environment DRE1 corresponding to tenant 1, resources are obtained in the operating resource pool according to the data operating environment DRE1 corresponding to tenant 1 and the obtained resources are used to run the task, that is, the electronic device can call the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the data operating environment DRE1 in the operating resource pool, and use the called actual storage and computing resources to perform the above data development task. Specifically, the electronic device can call the actual storage resources corresponding to the pre-virtualized storage resources included in the data operating environment DRE1, and retrieve the data source required to perform the data development task, so as to use the data source to realize data integration; and, the actual computing resources corresponding to the pre-virtualized computing resources included in the data operating environment DRE1 can be called to perform data computing.

[0115] Since a data operation environment corresponding to each tenant is pre-built, and the pre-virtualized storage and computing resources included in the data operation environment correspond to actual storage and computing resources, when the user wants to perform a data processing task, he only needs to submit the data processing task, and the electronic device can directly call the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the data operation environment corresponding to the tenant to which the user belongs to perform the data processing task. Since there is no need for the user to use various types of resources to perform data processing tasks according to the usage methods of various types of storage and computing resources, the complexity and difficulty of executing tasks are reduced, and the resource utilization rate is improved. And since each tenant corresponds to a different data operation environment, the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in each data operation environment are also different, that is, the actual storage and computing resources actually used to execute data processing tasks of different tenants are different, and virtual isolation between the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants is achieved. In addition, by constructing each data operation environment, the storage and computing resources are fine-grained, further improving the resource utilization rate.

[0116] As an implementation method of the present application, Figure 3 As shown, before the above step S103, i.e., the step of calling the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task, a data processing task execution method provided in an embodiment of the present application may further include:

[0117] S301: Displaying pre-virtualized storage and computing resources included in the target data operating environment, wherein the storage and computing resources include computing resources and storage resources.

[0118] After determining the target data operating environment for executing the data processing task to be executed submitted by the user, the electronic device can display the pre-virtualized storage and computing resources included in the target data operating environment on the display screen corresponding to the electronic device. The storage resources may include computing resources and storage resources.

[0119] As an implementation mode, the electronic device itself is equipped with a display screen. After determining the target data operating environment, the electronic device can display the pre-virtualized storage and computing resources included in the target data operating environment on the display screen carried by itself.

[0120] As an implementation method, after determining the target data operating environment, the electronic device can send a resource display instruction carrying the pre-virtualized storage and computing resources included in the above-mentioned target data operating environment to a communicating user device, so that the user device can display the pre-virtualized storage and computing resources included in the target data operating environment on its own display screen based on the above-mentioned resource display instruction.

[0121] As an implementation mode, for each tenant, the tenant may correspond to at least one data operating environment. The electronic device may display the data operating environments corresponding to the tenant, so that the user device selects a data operating environment for executing the data processing task based on the displayed data operating environment.

[0122] As shown in FIG4 (a), the electronic device itself is equipped with a display screen, which can display the pre-virtualized storage and computing resources included in each data operation environment, as well as the basic information, parameter settings, operation strategies and operation history of the data operation environment. The user can select Hive as input 1 based on the pre-virtualized storage and computing resources included in the data operation environment displayed by the electronic device. When the electronic device receives the selection instruction sent by the user, it can display the parameter setting information of the Hive input 1. The parameter information specifically includes information such as driver (running) memory / core number, excetor (executor) number, excetor (executor) memory / core number, DRE options and computing resource details. Among them, the current driver memory / core number is 1024MB and 1 core, the excetor number is 1, and the excetor memory / core number is 1024MB and 1 core respectively. The user can edit the driver memory / core number, excetor number, excetor memory / core number, and DRE through the drop-down options corresponding to each. For example, the user can click on the drop-down menu of the DRE option and select the data operating environment that is desired to be used to perform the data processing task from the development environment and the default operating environment included in the drop-down menu. In addition, the user can also perform operations such as searching attributes in the display interface.

[0123] The user can also select computing resources in the selected data operating environment, as shown in FIG4 (b). After selecting computing resources in the data operating environment selected by the user, the user can click the details option of the computing resource to view the details of the computing resource. When the electronic device detects the user's clicking operation on the details option, it can display the details of the computing resource, wherein the name of the data operating environment to which the computing resource belongs is dre001, CPU (Central Processing Unit) 34 cores, memory 166GB, remaining CPU 34 cores, remaining memory 166GB (Gigabyte), used CPU 0 cores and used memory 0GB. In addition, the current number of excetors is 1, and the number of excetor memory / cores is 1024MB and 1 core respectively.

[0124] In addition, the user can also select the storage resource as the input source from the pre-virtualized storage and computing resources included in the data operation environment displayed by the electronic device, and view the input source configuration information after selecting the input source. As shown in Figure 4 (c), after the user selects Kafka as input 1, the user can view the input source configuration information, operation column configuration, attribute configuration, and operation status of the Kafka input 1. When viewing the input source configuration, the user can view the input source / table configuration, input field view, and input source monitoring configuration, and can perform operations such as referencing and adding public data sources. When detecting that the user clicks the details option, the electronic device can display the name of the input source vkvmlkemwvkloemwvl, the data usage mark as the input and output source, the connection string: kafka2: / / 10.184.52.112:31993,10.184.52.113:31993..., host: 10.184.52.112, PORT (computer port) 31993, DB (Database) (SID (Security Identify), INDEX (index)...), user name: admin (Administrators) and comments and other information, and provide the user with an editing option for the user to modify the name of the input source.

[0125] S302: When a resource selection instruction issued by a user based on the displayed storage and computing resources is obtained, determine the target storage resource selected by the user from the storage resources and the target computing resource selected by the user from the computing resources. The target storage resource at least includes an output storage resource for storing the processing result of the data processing task.

[0126] The user can determine the target storage computing resource for executing the data processing task from the pre-virtualized storage computing resources included in the displayed target data operating environment, and send a resource selection instruction about the target storage computing resource to the electronic device. For example, the user can click on the target storage computing resource that he wants to select on the display screen and click on the selection button to send a resource selection instruction about the target storage computing resource to the electronic device.

[0127] As an implementation mode, when the screen displaying the pre-virtualized storage and computing resources included in the target data operating environment is a display screen of an electronic device, the user can directly send a resource selection instruction regarding the target storage and computing resources to the electronic device by performing a point selection operation.

[0128] As an implementation mode, when the screen displaying the pre-virtualized storage and computing resources included in the target data operating environment is the display screen of a user device, the user can send a resource selection instruction regarding the target storage and computing resources to the user device by performing a point selection operation. When the user device receives the resource selection instruction sent by the user, the user device can forward the received resource selection instruction to the electronic device.

[0129] Among them, the target storage and computing resources may include target storage resources and target computing resources; the target storage resources may at least include output storage resources for storing the processing results of the data processing tasks.

[0130] For example, a user can determine the output storage resources for storing the processing results of a data processing task and the target computing resources for calculating the data to be processed from the pre-virtualized storage and computing resources included in the displayed target data operating environment, and send resource selection instructions regarding the target storage resources and target computing resources to the electronic device.

[0131] As an implementation manner, the target storage resource may include an input storage resource for storing to-be-processed data corresponding to the data processing task and an output storage resource for storing a processing result of the data processing task.

[0132] The electronic device can obtain the resource selection instruction issued by the user, and determine the target storage resource selected by the user from the pre-virtualized storage resources included in the target data operating environment, and the target computing resource selected by the user from the pre-virtualized computing resources included in the target data operating environment.

[0133] Accordingly, the above step S103, i.e., the step of calling the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task, may include:

[0134] S303: Obtain the data to be processed corresponding to the data processing task.

[0135] S304: calling the actual computing resource corresponding to the target computing resource, processing the data to be processed, and obtaining a processing result.

[0136] S305: storing the processing result in the actual storage resource corresponding to the output storage resource.

[0137] When executing a data processing task, the electronic device can first obtain the data to be processed corresponding to the data processing task, and then the electronic device can call the actual computing resources corresponding to the target computing resources selected by the user, use the actual computing resources to process the data to be processed, obtain the processing results and store the processing results in the actual storage resources corresponding to the output storage resources selected by the user.

[0138] In this embodiment, the electronic device can display the pre-virtualized storage and computing resources included in the target data operating environment, so that the user can select the storage resource storage task output results under the target data operating environment, and select the specified distributed computing resources or server computing resources to perform data processing tasks. The electronic device can call the actual computing resources corresponding to the computing resources selected by the user, process the pending data corresponding to the data processing task, and store the data processing results in the actual storage resources corresponding to the storage resources selected by the user. In this way, by displaying the pre-virtualized storage and computing resources included in the target data operating environment to the user, the visualization of the pre-virtualized storage and computing resources included in the target data operating environment is achieved, which is convenient for the user to understand the pre-virtualized storage and computing resources included in the target data operating environment, and is convenient for the user to personalize the actual storage and computing resources corresponding to the virtualized storage and computing resources that he wants to use, and the actual storage and computing resources used to perform data processing tasks are all based on the resources selected by the user, which improves the user's experience.

[0139] As an implementation method of the embodiment of the present application, the target storage resource also includes an input storage resource for storing the to-be-processed data corresponding to the data processing task;

[0140] The above step S303, i.e. the step of obtaining the to-be-processed data corresponding to the data processing task, may include:

[0141] The to-be-processed data corresponding to the data processing task is obtained from the actual storage resources corresponding to the input storage resources.

[0142] The user can determine the target storage resource for executing the data processing task from the pre-virtualized storage resources included in the displayed target data operating environment, and send a resource selection instruction about the target storage resource to the electronic device. The target storage resource may include an input storage resource for storing the to-be-processed data corresponding to the data processing task and an output storage resource for storing the processing result of the data processing task.

[0143] For example, a user can determine, from the pre-virtualized storage and computing resources included in the displayed target data operating environment, input storage resources for storing the data to be processed corresponding to the data processing task, output storage resources for storing the processing results of the data processing task, and target computing resources for calculating the data to be processed, and send resource selection instructions regarding the input storage resources, output storage resources, and target computing resources to the electronic device.

[0144] When the electronic device receives a resource selection instruction issued by a user, it can obtain the data to be processed corresponding to the data processing task in the actual storage resources corresponding to the input storage resources; then call the actual computing resources corresponding to the target computing resources, use the actual computing resources to process the data to be processed, and obtain the processing results; and store the processing results in the actual storage resources corresponding to the output storage resources.

[0145] The input storage resource and the output storage resource may be the same storage resource or different storage resources, which is not specifically limited here.

[0146] Furthermore, the input storage resources and the output storage resources may be used as task input and task output, respectively.

[0147] As an implementation method, the data processing task to be executed submitted by the user carries the data to be processed corresponding to the data processing task, and the electronic device can obtain the data to be processed carried by the data processing task. Then, the actual computing resource corresponding to the target computing resource is called, and the data to be processed is processed using the actual computing resource to obtain the processing result; and the processing result is stored in the actual storage resource corresponding to the output storage resource.

[0148] In this embodiment, the target storage resource may include an input storage resource for storing the data to be processed corresponding to the data processing task and an output storage resource for storing the processing result of the data processing task. In this way, the electronic device can obtain the data to be processed corresponding to the data processing task from the input storage resource, call the target computing resource to process the data to be processed, and store the processing result to the output storage resource. According to the storage computing resource selected by the user, the input of the data processing task can be determined and the output of the data processing task can be obtained, which simplifies the user operation, reduces the complexity and difficulty of the data processing task, and improves the user experience.

[0149] As an implementation method of the present application, Figure 5 As shown, the construction method of the correspondence between the preset tenant and the pre-built data operating environment may include:

[0150] S501: Obtain resource information of each actual storage and computing resource.

[0151] Before building the data operating environment corresponding to each tenant, the electronic device may first obtain resource information of each actual storage and computing resource.

[0152] Among them, storage and computing resources may include storage resources and computing resources, and computing resources may include distributed computing resources and server computing resources. The electronic device may extract resource information of computing resources of different big data basic platforms that can be called, resource information of different server computing resources, and resource information of different storage resources to obtain resource information of each actual storage and computing resource that can be called.

[0153] Distributed computing resources may include Yarn queue resources in a Hadoop (Hadoop Distributed File System) cluster for executing data processing tasks such as data development tasks. Data processing tasks may include SQL (Structured Query Language) tasks, Shell tasks, Python tasks, data quality tasks, metadata collection tasks, and other tasks.

[0154] Server computing resources can include CPU resources and memory resources provided by various types of servers such as bare metal, virtual machines, physical servers, and cloud servers.

[0155] Storage resources can include various database resources, such as PostgreSQL, Elasticsearch, Hive, Kafka, Kudu, SparkSQL, Hbase, Cassandra, HDFS, petreldb, egretdb, stargraph, DM, kingbase, Greenplum, MySQL, Oracle, SQLServer, Hudi, FTP, Redis, etc.

[0156] S502: For each tenant, determine the actual storage and computing resources required by the tenant according to the business needs of the tenant.

[0157] For each tenant, the type and specifications of the data processing tasks that the tenant needs to perform to meet the above business needs can be determined based on the tenant's business needs, and then the actual storage and computing resources required for the tenant to perform the above data processing tasks can be determined.

[0158] S503: For each tenant, according to preset virtualization rules and based on the resource information, the actual storage and computing resources required by the tenant are virtualized to obtain a virtualized storage and computing resource set as the resource pool corresponding to the tenant.

[0159] Since different resources are used in different ways, in order to shield the differences in the underlying big data infrastructure platform, server types, and database types, each category of resources can be described and defined according to the same preset virtualization rules to virtualize the resources of that category.

[0160] For the sake of clarity, the preset virtualization rules for virtualizing various categories of resources will be introduced below.

[0161] In addition, in order to simplify the operation of executing the data processing tasks of each tenant, the resource information of the storage resources and the resource information of the computing resources required for the data processing tasks of each tenant can be packaged in a virtual environment, and the resource information of the storage resources and the resource information of the computing resources required for executing the data processing tasks of each tenant can be unified managed by managing the virtual environment.

[0162] Based on this, for each tenant, the actual storage and computing resources required by the tenant can be virtualized according to the preset virtualization rules and based on the resource information of each actual storage and computing resource, and the virtualized storage and computing resource set corresponding to the tenant is obtained as the resource pool corresponding to the tenant. By constructing the resource pool corresponding to each tenant, virtual isolation between the actual storage and computing resources corresponding to different tenants is achieved.

[0163] S504: Import virtualized storage and computing resources from the tenant's resource pool into the constructed data operating environment corresponding to the tenant, and obtain a correspondence between the preset tenant and the pre-constructed data operating environment.

[0164] The process of packaging the resource information of storage resources and the resource information of computing resources required for the data processing tasks of each tenant in a virtual environment is the process of building the data operating environment corresponding to the tenant.

[0165] After the virtualization is completed, the virtualized storage and computing resources can be imported from the tenant's resource pool into the constructed data operating environment corresponding to the tenant. In this way, the pre-virtualized storage and computing resources included in the tenant's data operating environment can be determined, and the correspondence between the preset tenant and the pre-built data operating environment can be obtained. Through the correspondence between the preset tenant and the pre-built data operating environment, virtual isolation between the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the data operating environments corresponding to different tenants is achieved.

[0166] For each tenant, according to the number of data types of data processing tasks that the tenant needs to perform and the specifications of the actual storage and computing resources required for each data processing task, the data operating environment corresponding to the tenant can be one or more, which is not specifically limited here. For example, according to the business needs of the tenant, three data operating environments can be constructed for the tenant, namely DRE1, DRE2 and DRE3. The actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in DRE1 are used to perform data extraction related tasks for the tenant, the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in DRE2 are used to perform data analysis related tasks for the tenant, and the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in DRE3 are used to perform data calculation related tasks for the tenant.

[0167] The following is a detailed description of the process of constructing the correspondence between the above-mentioned preset tenants and the pre-built data operating environment in conjunction with FIG. 6 (a)-FIG. 6 (b).

[0168] As shown in FIG6 (a), when starting to build a correspondence between a preset tenant and a pre-built data operating environment, the electronic device can obtain tenant information of the preset tenant and determine whether there are multiple tenants based on the tenant information. If not, that is, when the tenant is a single tenant, enter the default resource pool. If yes, that is, when the tenant is multiple tenants, a tenant can be selected and enter the tenant resource pool.

[0169] Next, for each tenant, according to the business needs of the tenant, the actual storage and computing resources required by the tenant are determined, and the resource information of each actual storage and computing resource is obtained by configuring the big data cluster and the configuration server. Next, for each tenant, according to the preset virtualization rules, based on the obtained resource information, the actual storage and computing resources required by the tenant are virtualized to obtain virtualized distributed computing resources, storage resources and server computing resources, and the data operating environment corresponding to the tenant is constructed to obtain the corresponding relationship between the preset tenant and the pre-constructed data operating environment, and the construction process is terminated.

[0170] As shown in FIG6(b), when creating each data runtime environment DRE, the electronic device may first start initialization, that is, the electronic device may first determine whether there is a default resource pool. If there is no default resource pool, the electronic device may initialize the default resource pool and respectively perform default cluster initialization, default storage resource initialization, default distributed computing resource initialization, and default server resource initialization.

[0171] In the case of a default resource pool, the electronic device can create a default DRE. Initialize the default DRE and obtain virtual resources from the resource pool. Then, the electronic device confirms whether there are server computing resources, storage resources, and distributed computing resources in the obtained virtual resources according to the business needs of the tenant. For each of the server computing resources, storage resources, and distributed computing resources, if the resource exists, the virtualized resource can be imported and a DRE can be created. If the resource does not exist, the resource will not be imported.

[0172] Specifically, if there are server computing resources, import the server computing resources and create a DRE local queue; if there are storage resources, import the storage resources and create a DRE storage resource; if there are distributed computing resources, import the distributed computing resources and create a Yarn resource queue. Determine whether the DRE initialization is complete. If not, wait for initialization. If the DRE initialization is complete, you can activate the DRE and use it to execute the data processing tasks corresponding to the tenant.

[0173] In this embodiment, the electronic device can obtain resource information of each actual storage and computing resource, and for each tenant, according to the business needs corresponding to the tenant, determine the actual storage and computing resources required by the tenant, and then virtualize the actual storage and computing resources required by the tenant based on the resource information according to the preset virtualization rules, and obtain a virtualized storage and computing resource set as the resource pool corresponding to the tenant. Then, from the resource pool of the tenant, the virtualized storage and computing resources are imported into the constructed data operation environment corresponding to the tenant, and the corresponding relationship between the preset tenant and the pre-constructed data operation environment is obtained. In this way, by incorporating the actual storage and computing resources required by the tenant into the resource pool corresponding to the tenant according to the business needs of each tenant, customized resource use is achieved. By realizing the reintegration of resources, the data operation environment for specific business is constructed in combination with the project, and the resources are uniformly described and managed in the form of the data operation environment. In addition, by constructing the resource pool and data operation environment corresponding to each tenant, virtual isolation between the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants is achieved.

[0174] As an implementation of an embodiment of the present application, the pre-virtualized storage and computing resources include pre-virtualized distributed computing resources, pre-virtualized server computing resources and pre-virtualized storage resources.

[0175] The preset virtualization rules of the distributed computing resources include: the virtualized distributed computing resources include multiple virtualization queues, and for each distributed computing resource cluster, the virtualization queue corresponding to the cluster includes a tenant queue, a cluster queue, a data operating environment queue, and a custom queue;

[0176] The preset virtualization rules of the server computing resources include: the virtualized server computing resources include data integration instances and server local queues, wherein the data integration instance is a resource required to extract the data to be processed from the source end, convert the data to be processed and load it to the destination end, and the data integration instance is represented by the instance name, CPU specification, memory specification and the server to which it belongs; the server local queue is represented by the queue name, CPU specification and memory specification;

[0177] The preset virtualization rules of the storage resources include: the virtualized storage resources are represented by storage resource names, storage resource types, storage resource URL connection strings, authentication methods, and attribute configuration information.

[0178] For the data processing tasks that each tenant needs to perform, the actual storage and computing resources required to perform the data processing tasks may include: distributed computing resources required for big data computing, server computing resources required for local batch tasks, and storage resources required for data query and warehousing.

[0179] Based on this, the pre-virtualized storage and computing resources included in the pre-built data operating environment corresponding to each tenant may include pre-virtualized distributed computing resources, pre-virtualized server computing resources and pre-virtualized storage resources.

[0180] The preset virtualization rules of the distributed computing resources may include: the virtualized distributed computing resources may include multiple virtualization queues, and for each distributed computing resource cluster, the virtualization queue corresponding to the cluster may include a tenant queue, a cluster queue, a data operating environment queue, and a custom queue. The virtualization definition of the initialization queue is performed by the queue management in the data operating environment.

[0181] The underlying layer of distributed computing resources is the computing queue resources in the distributed computing resource cluster. For each tenant, the number of queues can be initialized according to the number of clusters of the distributed computing resource cluster under the resource pool corresponding to the tenant. For example, the computing queue resources in a distributed computing resource cluster can be used as initialization queues, so that the number of initialization queues is equal to the number of clusters of the distributed computing resource cluster; the computing queue resources in a distributed computing resource cluster can also be divided into multiple initialization queues, so that the number of initialization queues is more than the number of clusters of the distributed computing resource cluster, which is not specifically limited here.

[0182] To virtualize the distributed computing resources is to virtualize each initialization queue separately, so that the virtualized distributed computing resources can include multiple virtualized queues. In addition, in order to facilitate the management of the queues, the virtualized queues can be planned according to the tenants, clusters, data operating environments, and user customizations. In this way, for each distributed computing resource cluster, the virtualized queues corresponding to the cluster can include tenant queues, cluster queues, data operating environment queues, and custom queues.

[0183] As an implementation method, for each distributed computing resource cluster, the virtualization queue corresponding to the cluster can have the tenant queue as the first-level parent queue, the cluster queue as the second-level parent queue, the data operating environment queue as the third-level queue, and the custom queue as the fourth-level queue.

[0184] As an implementation method, for each distributed computing resource cluster, the virtualization queue corresponding to the cluster can have the tenant queue as the first-level parent queue, the data operating environment queue as the second-level parent queue, the cluster queue as the third-level queue, and the custom queue as the fourth-level queue.

[0185] In addition, the custom queue can be represented by the queue name, CPU specification, and memory specification of the actual queue corresponding to the custom queue, and the custom queue can be managed by the administrator.

[0186] As an implementation mode, the custom queue may be represented by the queue name, CPU specification, memory specification, and the cluster to which the actual queue corresponds.

[0187] Exemplarily, as shown in FIG. 7 (a), the bottom layer of the distributed computing resource is a Yarn queue in a hadoop cluster, and the custom queue can be represented by the queue name, CPU specification, memory specification, and the cluster to which the Yarn queue corresponds to the custom queue. That is, the electronic device can display an editing interface for editing the Yarn queue, and the editing interface is used to edit the data operating environment to which the Yarn queue belongs, the specifications of the Yarn queue, and the cluster to which the Yarn queue belongs. In this way, the administrator can edit the data operating environment to which the Yarn queue belongs to dre001 in the editing interface, select the specification 34 cores / 166GB of the Yarn queue from the displayed specifications 34 cores / 166GB, 3 cores / 8GB, and 6 cores / 16GB, and edit the cluster to which the Yarn queue belongs to as the default cluster, and then click the OK option to complete the custom settings for the Yarn queue, or click the Cancel option to end the editing process.

[0188] After virtualizing each initialization queue respectively, the electronic device can allocate mapped queue resources in the actual computing resources under the actual distributed computing resource cluster.

[0189] The preset virtualization rules of the server computing resources may include: the virtualized server computing resources include a data integration instance and a server local queue.

[0190] The underlying server computing resources are CPU resources and memory resources provided by bare metal, virtual machines, physical servers, and cloud servers, which are used to run data integration instance programs, data computing task local mode (sparklocal), python tasks, shell tasks, etc. The virtualization of server computing resources is defined by the data integration instances and server local queues in the data operation environment.

[0191] Among them, the data integration instance (Extraction-Transformation-Loading, ETL) is the resource required to extract the data to be processed from the source end, transform the data to be processed and load it to the destination end, and the above data integration instance can be represented by the instance name, CPU specification, memory specification and the server to which it belongs.

[0192] The server local queue is the server computing resources required by the server to run batch tasks. The server local queue can be represented by queue name, CPU specifications, and memory specifications.

[0193] After virtualizing the server computing resources, the electronic device can allocate corresponding CPU resources and memory resources on the actual server computing resources.

[0194] Exemplarily, as shown in FIG. 7 (b) and FIG. 7 (c), the underlying server computing resources are CPU resources and memory resources provided for bare metal, virtual machines, physical servers, and cloud servers, etc. Administrators can customize ETL instances and server local queues, that is, administrators can define the ETL name, CPU specifications, memory specifications, and the server to which they belong; and define the local queue name, CPU specifications, and memory specifications. In other words, the electronic device can display an editing interface for editing an ETL instance and an editing interface for editing a server local queue Udps (Unified Data Processing Service). The interface for editing an ETL instance is used to edit the ETL instance name, specifications, and server to which the ETL instance belongs. In this way, the administrator can edit the ETL instance name to 122222 in the editing interface, select the specification 1 core / 1GB from the displayed specifications 1 core / 1GB, 1 core / 8GB, and 8 cores / 16GB, and edit the server to which the ETL instance belongs to 10.184.52.234 (root (root user) 1), and then click the OK option to complete the custom settings for the ETL instance, or click the Cancel option to end the editing process. The interface for editing the Udps queue is used to edit the Udps queue name and specifications. In this way, the administrator can edit the Udps queue name to 11 in the editing interface, select the specification 5 core / 16GB from the displayed specifications 5 core / 10GB, 5 core / 16GB, and 8 core / 41GB, and then click the OK option to complete the custom settings for the Udps queue, or click the Cancel option to end the editing process.

[0195] The underlying storage resources are various database resources, which are used to provide data query and storage when executing data processing tasks, and support adding connection information of all storage resources in the current cluster and platform environment. The preset virtualization rules of storage resources can include: the virtualized storage resources are represented by storage resource name, storage resource type, storage resource URL (Universal Resource Locator) connection string, authentication method, and attribute configuration information. The virtualization of storage resources is performed by storage resources in the data operation environment.

[0196] For example, as shown in FIG7 (d), the administrator can view computing resources and storage resources in the electronic device, and edit the storage resource name, storage resource type, storage resource URL connection string, authentication method and attribute configuration information of various database resources in the display interface of storage resources. After the administrator completes the editing, the electronic device can display all 10 pre-virtualized storage resources included in the data operation environment, including 2 Kafka, 2 Elasticsearch, 2 Kudu, 1 PostgreSQL, 1 Redis, 1 SparkSQL and 1 HDFS, and display the classification, creator, authentication information, connection string and update time of the 3 system data sources in the display interface. Specifically, the classification of system data source 1 is Kudu, the creator is admin, and the connection string is kudu: / / dfthbp114.bigdata.hikvision,com:7051,dfthbp113.bigdata.hikvision,com:7051,dfthbp112...; the classification of system data source 2 is Kafka, the creator is admin, and the connection string is kafka2: / / dfthbp114.bigdata.hikvision.com:31993,dfthbp113.bigdata.hikvision.com:31993,dfthbp...; the classification of system data source 3 is Elasticsearch, the creator is admin, and the connection string is es:rest: / / dfthbp114.bigdata.hikvision.com:31920,dfthbp113.bigdata.hikvision.com:31920,dfthbp.... In addition, the electronic device also supports the continued addition of system data sources.

[0197] In this embodiment, each storage and computing resource is virtualized according to its own preset virtualization rules, supports adaptation to different big data basic platforms and databases, shields the differences between data platforms and databases, and extracts unified storage and computing resources; supports the management of bare metal, virtual machines, physical servers and cloud servers, extracts unified server computing resources, and provides a cluster management method to achieve unified management of storage and computing resources, thereby simplifying the operation of data processing tasks.

[0198] As an implementation method of the embodiment of the present application, the preset virtualization rules of the distributed computing resources specifically include:

[0199] The virtualized distributed computing resources include multiple virtualization queues. For each distributed computing resource cluster, the virtualization queue corresponding to the cluster has the tenant queue as the first-level parent queue, the cluster queue as the second-level parent queue, the data operating environment queue as the third-level queue, and the custom queue as the fourth-level queue; the custom queue is represented by the queue name, CPU specifications, and memory specifications of the actual queue corresponding to the custom queue.

[0200] The preset virtualization rules of the distributed computing resources may specifically include: the virtualized distributed computing resources include multiple virtualization queues, and for each distributed computing resource cluster, the virtualization queue corresponding to the cluster uses the tenant queue as the first-level parent queue, the cluster queue as the second-level parent queue, the data operating environment queue as the third-level queue, and the custom queue as the fourth-level queue. In addition, the custom queue can be represented by the queue name, CPU specification, and memory specification of the actual queue corresponding to the custom queue.

[0201] For example, tenant default has a cluster c1 in the data operating environment, and the queue name of the actual queue of the custom queue is xx, where the identifier of the data operating environment is d1. The virtualized queues after virtualization are as follows: the first-level queue is the tenant queue default, the second-level queue is the cluster queue default.c1, the third-level queue is the data operating environment queue default.c1.d1, and the fourth-level queue is the user-defined queue default.c1.d1.xx.

[0202] In this embodiment, for each distributed computing resource cluster, the virtualization queue corresponding to the cluster uses the tenant queue as the first-level parent queue, the cluster queue as the second-level parent queue, the data operating environment queue as the third-level queue, and the custom queue as the fourth-level queue. In addition, the custom queue can be represented by the queue name, CPU specification, and memory specification of the actual queue corresponding to the custom queue. In this way, by dividing the virtualization queue according to the queue level, the hierarchical relationship between the queues can be clarified, which facilitates the management of each queue resource according to the level.

[0203] As an implementation of an embodiment of the present application, a data processing task execution method provided in an embodiment of the present application may also include:

[0204] After obtaining an update instruction for the data operating environment, the update instruction is used to instruct to create a new data operating environment for the target tenant or to expand the existing data operating environment;

[0205] In a case where the update instruction is used to instruct to create a new data operating environment for the target tenant, determining whether the number of data operating environments corresponding to the target tenant reaches a first preset number, and in a case where the number of data operating environments corresponding to the target tenant does not reach the preset number, creating a new data operating environment for the target tenant;

[0206] In the case where the update instruction is used to indicate the expansion of an existing data operating environment, it is determined whether the number of pre-virtualized storage and computing resources included in the existing data operating environment has reached a second preset number, and if the number of pre-virtualized storage and computing resources included in the existing data operating environment has not reached the second preset number, the existing data operating environment is expanded.

[0207] After constructing the data operating environment corresponding to each tenant, the administrator can manage the data operating environment corresponding to the tenant according to the changes in the tenant's business needs in actual applications. When it is desired to create a new data operating environment for a target tenant or expand the capacity of an existing data operating environment, the administrator can send an update instruction for the data operating environment to the electronic device, wherein the update instruction is used to instruct to create a new data operating environment for the target tenant or expand the capacity of an existing data operating environment.

[0208] After constructing the data operating environment corresponding to each tenant, the constructed data operating environment is integrated with the tenant system, and the tenant is associated with the data operating environment. Each user in the tenant has the authority to the computing resources and storage resources included in the data operating environment. However, for each tenant, there are resource restrictions on the storage and computing resources corresponding to the tenant, that is, the authorized resource capacity of each tenant is controlled, the number of data operating resources corresponding to the tenant is controlled, and the number of virtualized storage and computing resources included in each data operating resource of the tenant is also controlled. For example, the authorized resource capacity of the data operating environment for the data processing task of the information and communication platform is set. When the number of resources required by the information and communication platform exceeds the authorized resource capacity, even if the physical server or Yarn queue has non-virtualized resources, it is restricted to create a new data operating environment for the information and communication platform or expand the capacity of the existing data operating environment.

[0209] Based on this, when an electronic device obtains an update instruction for a data operating environment, it can determine the method of responding to the update instruction based on the instruction content of the update instruction, the number of data operating environments corresponding to the tenants, and the number of virtualized storage and computing resources included in each data operating environment, respectively, and the quantitative relationship with the authorized resource capacity.

[0210] Specifically, when the update instruction is used to indicate the creation of a new data operating environment for the target tenant, the electronic device can determine whether the number of data operating environments corresponding to the target tenant has reached a first preset number. When the number of data operating environments corresponding to the target tenant has not reached the first preset number, the electronic device can create a new data operating environment for the target tenant based on the specifications of the data operating environment to be created indicated by the update instruction, such as the type of pre-virtualized storage and computing resources included in the data operating environment to be created. The first preset number is the authorized resource capacity of the number of data operating environments corresponding to the tenant, and the first preset number can be set according to the business type and tenant level of the tenant, for example, 5, 10, etc., which is not specifically limited here.

[0211] For example, when the number of data operating environments corresponding to the target tenant does not reach a first preset number, the electronic device may, according to the type of pre-virtualized storage and computing resources included in the data operating environment to be created indicated by the update instruction, in accordance with preset virtualization rules, and based on the resource information in the resource pool corresponding to the target tenant, virtualize the actual storage and computing resources corresponding to the type of pre-virtualized storage and computing resources included in the data operating environment to be created, import the virtualized storage and computing resources from the resource pool of the target tenant, obtain a correspondence between the preset tenant and the pre-built data operating environment, obtain a data operating environment including virtualized storage and computing resources, and build a correspondence between the target tenant and the newly created data operating environment.

[0212] On the contrary, when the number of data operating environments corresponding to the target tenant reaches a first preset number, the electronic device can output a notification message that the data operating environment corresponding to the target tenant has reached the upper limit, so that the administrator can check and manage the existing data operating environment of the target tenant.

[0213] In the case where the update instruction is used to instruct the expansion of the existing data operating environment, it is determined whether the number of pre-virtualized storage and computing resources included in the existing data operating environment has reached a second preset number, and if the number of pre-virtualized storage and computing resources included in the existing data operating environment has not reached the second preset number, the existing data operating environment is expanded. The second preset number is the authorized resource capacity of the number of pre-virtualized storage and computing resources included in each data operating environment corresponding to the tenant, and the second preset number can be set according to the tenant's business type and tenant level, for example, 5, 10, etc.; and the first preset number and the second preset number can be the same or different, and are not specifically limited here.

[0214] For example, when the number of pre-virtualized storage and computing resources included in the existing data operating environment indicated by the update instruction does not reach the second preset number, the electronic device can expand the existing data operating environment according to the type and number of resources to be expanded indicated by the update instruction. Specifically, when the update instruction indicates that the resources to be expanded are storage resources, the electronic device expands the storage resources of the existing data operating environment; when the update instruction indicates that the resources to be expanded are distributed computing resources, the electronic device expands the distributed computing resources of the existing data operating environment. When expanding the existing data operating environment, the storage and computing resources in the resource pool of the target tenant can be virtualized, and the virtualized storage and computing resources can be imported into the existing data operating environment to obtain the expanded data operating environment.

[0215] On the contrary, when the number of pre-virtualized storage and computing resources included in the existing data operating environment reaches a second preset number, the electronic device can output a notification message that the number of pre-virtualized storage and computing resources included in the existing data operating environment has reached an upper limit, so that the administrator can check and manage the existing data operating environment of the target tenant.

[0216] In this embodiment, when an update instruction for instructing to create a new data operating environment for a target tenant or to expand the capacity of an existing data operating environment is obtained, the method of responding to the update instruction can be determined according to the instruction content of the update instruction, and the quantitative relationship between the number of data operating environments corresponding to the tenant and the number of virtualized storage and computing resources included in each data operating environment and the authorized resource capacity. In the case where the update instruction is used to instruct to create a new data operating environment for the target tenant, it is determined whether the number of data operating environments corresponding to the target tenant has reached a first preset number, and if the number of data operating environments corresponding to the target tenant has not reached the first preset number, a new data operating environment is created for the target tenant; in the case where the update instruction is used to instruct to expand the capacity of an existing data operating environment, it is determined whether the number of pre-virtualized storage and computing resources included in the existing data operating environment has reached a second preset number, and if the number of pre-virtualized storage and computing resources included in the existing data operating environment has not reached the second preset number, the existing data operating environment is expanded. In this way, we first determine the quantitative relationship between the number of data operating environments corresponding to the tenant and the number of virtualized storage and computing resources included in each data operating environment and the authorized resource capacity to judge whether the amount of resources corresponding to the tenant has reached the authorized resource limit, and perform subsequent operations based on the determined quantitative relationship. This can effectively avoid resource call errors caused by the resources corresponding to the tenant exceeding the authorized resource limit, thereby improving task execution efficiency and accuracy.

[0217] Of course, managers can also perform other management operations on the data operating environment. Figure 8 Provides specific instructions for the management process of the data operating environment.

[0218] The development administrator can start the data operating environment platform and enter the DRE management interface. The development administrator can query the DRE according to the DRE name, mode type and description of the DRE to be managed.

[0219] When you want to edit or add a DRE, you can import storage resources, distributed computing resources, and server computing resources from the resource pool of the tenant corresponding to the DRE. In addition, you can also edit the environment management members of the DRE, where the environment management members of the DRE are users belonging to the tenant corresponding to the DRE, that is, users associated with the DRE. Users associated with the DRE have the right to use the actual computing resources and actual storage resources corresponding to the pre-virtualized computing resources and pre-virtualized storage resources included in the DRE. For example, the environment management members of DRE1 include user A. When using DRE1 to execute the development data task of user A, user A can select the actual storage resources corresponding to the pre-virtualized storage resources included in the DRE as input and output, and select the Yarn queue corresponding to the pre-virtualized computing resources included in the DRE to execute the development data task.

[0220] After importing storage resources, distributed computing resources, and server computing resources, the electronic device can determine whether the DRE is a new DRE based on the DRE name, mode type, and description of the DRE to be managed. If the DRE is a new and uncreated DRE, the electronic device can virtualize the imported storage resources, distributed computing resources, and server computing resources, create the new DRE, and complete the initialization of the DRE.

[0221] If the DRE is not a new DRE, the electronic device edits the DRE according to the editing instruction. When the editing instruction indicates to expand the capacity of the DRE, the electronic device expands the capacity of at least one of the storage resources, distributed computing resources, and server computing resources of the DRE according to the editing instruction. When the editing instruction indicates to reduce the capacity of the DRE, the electronic device reduces the capacity of at least one of the storage resources, distributed computing resources, and server computing resources of the DRE according to the editing instruction, thereby completing the editing or updating.

[0222] In case of receiving a disabling instruction for the data operating environment, the electronic device may disable the data operating environment.

[0223] When receiving an activation instruction for a currently disabled data execution environment, the electronic device may activate the data execution environment so as to subsequently use the DRE to perform data processing tasks.

[0224] Upon receiving a deletion instruction on the data operating environment, the electronic device can delete the DRE and detect whether the data operating environment is in operation, that is, check whether there are any computing resources or storage resources in use. If at least one of the distributed computing resources, server computing resources, and storage resources is in use, a resource recycling operation is performed, and after the recycling is completed, an operation of deleting the DRE is performed, and the DRE and the corresponding relationship between the DRE and the tenant are destroyed when the deletion is completed.

[0225] The electronic device can perform automatic inspection tasks according to the preset inspection cycle. Query the DRE and test the server connectivity to determine whether the server is successfully connected. In the case of disconnection, output a notification message indicating an abnormality so that the staff can check the abnormality of the data operating environment; for each successfully connected data operating environment, continue to determine whether it is abnormal. If it is detected that the data operating environment is not activated, activate the data operating environment and end.

[0226] Corresponding to the above-mentioned method for executing a data processing task, an embodiment of the present application further provides a data processing task execution device. The following is an introduction to a data processing task execution device provided in an embodiment of the present application.

[0227] like Fig. 9 As shown, a data processing task execution device includes:

[0228] The task acquisition module 910 is used to acquire the data processing task to be executed submitted by the user and determine the target tenant to which the user belongs;

[0229] A data operating environment determination module 920 is used to determine a target data operating environment for executing the data processing task based on the correspondence between the target tenant and the preset tenant and the pre-built data operating environment, wherein the data operating environment includes pre-virtualized storage and computing resources, and actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants are virtually isolated from each other;

[0230] The task execution module 930 is used to call the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task.

[0231] In the solution provided by the embodiment of the present application, the electronic device can obtain the data processing task to be executed submitted by the user, and determine the target tenant to which the user belongs; based on the correspondence between the target tenant and the preset tenant and the pre-built data operating environment, determine the target data operating environment for executing the data processing task, wherein the data operating environment includes pre-virtualized storage and computing resources, and the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants are virtually isolated; call the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task.

[0232] Since a data operation environment corresponding to each tenant is pre-built, and the pre-virtualized storage and computing resources included in the data operation environment correspond to actual storage and computing resources, when the user wants to perform a data processing task, he only needs to submit the data processing task, and the electronic device can directly call the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the data operation environment corresponding to the tenant to which the user belongs to perform the data processing task. Since there is no need for the user to use various types of resources to perform data processing tasks according to the usage methods of various types of storage and computing resources, the complexity and difficulty of executing tasks are reduced, and the resource utilization rate is improved. And since each tenant corresponds to a different data operation environment, the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in each data operation environment are also different, that is, the actual storage and computing resources actually used to execute data processing tasks of different tenants are different, and virtual isolation between the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants is achieved. In addition, by constructing each data operation environment, the computing resources are divided into smaller granularity, and the resources can be controlled more finely, further improving the resource utilization rate.

[0233] As an implementation of the embodiment of the present application, the device further includes:

[0234] A resource display module, used for displaying the pre-virtualized storage and computing resources included in the target data operating environment before the step of calling the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task, wherein the storage and computing resources include computing resources and storage resources;

[0235] A resource determination module, configured to determine, upon obtaining a resource selection instruction issued by a user based on the displayed storage and computing resources, a target storage resource selected by the user from the storage resources and a target computing resource selected by the user from the computing resources, wherein the target storage resource at least includes an output storage resource for storing a processing result of the data processing task;

[0236] The task execution module 930 includes:

[0237] A data acquisition unit, used to acquire the to-be-processed data corresponding to the data processing task;

[0238] A resource calling unit, used to call the actual computing resource corresponding to the target computing resource, process the data to be processed, and obtain a processing result;

[0239] The storage unit is used to store the processing result in the actual storage resource corresponding to the output storage resource.

[0240] As an implementation method of the embodiment of the present application, the target storage resource also includes an input storage resource for storing the to-be-processed data corresponding to the data processing task;

[0241] The data acquisition unit comprises:

[0242] The data acquisition subunit is used to acquire the to-be-processed data corresponding to the data processing task from the actual storage resources corresponding to the input storage resources.

[0243] As an implementation manner of the embodiment of the present application, the device further includes a corresponding relationship building module, and the corresponding relationship building module includes:

[0244] An information acquisition unit, used to acquire resource information of each actual storage and computing resource;

[0245] A resource determination unit, used to determine, for each tenant, the actual storage and computing resources required by the tenant according to the business requirements of the tenant;

[0246] A virtualization unit is used to virtualize the actual storage and computing resources required by each tenant according to preset virtualization rules and based on the resource information to obtain a virtualized storage and computing resource set as a resource pool corresponding to the tenant;

[0247] The resource import unit is used to import virtualized storage and computing resources from the tenant's resource pool into the data operating environment constructed corresponding to the tenant, so as to obtain the corresponding relationship between the preset tenant and the pre-built data operating environment.

[0248] As an implementation method of the embodiment of the present application, the pre-virtualized storage and computing resources include pre-virtualized distributed computing resources, pre-virtualized server computing resources and pre-virtualized storage resources;

[0249] The preset virtualization rules of the distributed computing resources include: the virtualized distributed computing resources include multiple virtualization queues, and for each distributed computing resource cluster, the virtualization queue corresponding to the cluster includes a tenant queue, a cluster queue, a data operating environment queue, and a custom queue;

[0250] The preset virtualization rules of the server computing resources include: the virtualized server computing resources include data integration instances and server local queues, wherein the data integration instance is a resource required to extract the data to be processed from the source end, convert the data to be processed and load it to the destination end, and the data integration instance is represented by the instance name, CPU specification, memory specification and the server to which it belongs; the server local queue is represented by the queue name, CPU specification and memory specification;

[0251] The preset virtualization rules of the storage resources include: the virtualized storage resources are represented by storage resource names, storage resource types, storage resource URL connection strings, authentication methods, and attribute configuration information.

[0252] As an implementation method of the embodiment of the present application, the preset virtualization rules of the distributed computing resources specifically include:

[0253] The virtualized distributed computing resources include multiple virtualization queues. For each distributed computing resource cluster, the virtualization queue corresponding to the cluster has the tenant queue as the first-level parent queue, the cluster queue as the second-level parent queue, the data operating environment queue as the third-level queue, and the custom queue as the fourth-level queue; the custom queue is represented by the queue name, CPU specifications, and memory specifications of the actual queue corresponding to the custom queue.

[0254] As an implementation of the embodiment of the present application, the device further includes:

[0255] An instruction acquisition module is used to acquire an update instruction for the data operating environment, wherein the update instruction is used to instruct to create a new data operating environment for the target tenant or to expand the existing data operating environment;

[0256] an environment creation module, configured to determine whether the number of data operating environments corresponding to the target tenant reaches a first preset number when the update instruction is used to instruct to create a new data operating environment for the target tenant, and to create a new data operating environment for the target tenant when the number of data operating environments corresponding to the target tenant does not reach the preset number;

[0257] An expansion module is used to determine whether the number of pre-virtualized storage and computing resources included in the existing data operating environment has reached a second preset number when the update instruction is used to indicate the expansion of the existing data operating environment, and to expand the existing data operating environment when the number of pre-virtualized storage and computing resources included in the existing data operating environment has not reached the second preset number.

[0258] The present application also provides an electronic device, such as Fig.10 As shown, including:

[0259] Memory 1001, used for storing computer programs;

[0260] The processor 1002 is used to implement the above-mentioned data processing task execution method when executing the program stored in the memory 1001.

[0261] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1002, the communication interface, and the memory 1001 communicate with each other via the communication bus.

[0262] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0263] The communication interface is used for communication between the above electronic device and other devices.

[0264] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0265] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0266] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data processing task execution methods are implemented.

[0267] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the data processing task execution methods in the above embodiments.

[0268] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0269] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0270] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0271] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for executing a data processing task, characterized in that: The method comprises: Obtaining the data processing task to be executed submitted by the user, and determining the target tenant to which the user belongs; Based on the correspondence between the target tenant and the preset tenant and the pre-built data operating environment, determine the target data operating environment for executing the data processing task, wherein the data operating environment includes pre-virtualized storage and computing resources, and the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants are virtually isolated; Calling actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task; Obtaining an update instruction for the data operating environment, wherein the update instruction is used to instruct the creation of a new data operating environment for the target tenant or the expansion of an existing data operating environment; In a case where the update instruction is used to instruct to create a new data operating environment for the target tenant, determining whether the number of data operating environments corresponding to the target tenant reaches a first preset number, and in a case where the number of data operating environments corresponding to the target tenant does not reach the first preset number, creating a new data operating environment for the target tenant; In the case where the update instruction is used to indicate the expansion of an existing data operating environment, it is determined whether the number of pre-virtualized storage and computing resources included in the existing data operating environment has reached a second preset number, and if the number of pre-virtualized storage and computing resources included in the existing data operating environment has not reached the second preset number, the existing data operating environment is expanded.

2. The method according to claim 1, characterized in that Before the step of calling the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task, the method further includes: Displaying pre-virtualized storage and computing resources included in the target data operating environment, wherein the storage and computing resources include computing resources and storage resources; In the case of obtaining a resource selection instruction issued by a user based on the displayed storage and computing resources, determining the target storage resource selected by the user from the storage resources and the target computing resource selected by the user from the computing resources, wherein the target storage resource at least includes an output storage resource for storing the processing result of the data processing task; The step of calling the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task includes: Obtaining the data to be processed corresponding to the data processing task; Calling the actual computing resource corresponding to the target computing resource to process the data to be processed to obtain a processing result; The processing result is stored in the actual storage resource corresponding to the output storage resource.

3. The method according to claim 2, characterized in that The target storage resource also includes an input storage resource for storing the to-be-processed data corresponding to the data processing task; The step of obtaining the to-be-processed data corresponding to the data processing task includes: The to-be-processed data corresponding to the data processing task is obtained from the actual storage resources corresponding to the input storage resources.

4. The method according to claim 1, characterized in that: The method for constructing the correspondence between the preset tenant and the pre-built data operating environment includes: Obtain resource information of each actual storage and computing resource; For each tenant, determine the actual storage and computing resources required by the tenant based on the tenant's corresponding business needs; For each tenant, according to the preset virtualization rules and based on the resource information, the actual storage and computing resources required by the tenant are virtualized to obtain a virtualized storage and computing resource set as the resource pool corresponding to the tenant; From the resource pool of the tenant, virtualized storage and computing resources are imported into the data operating environment corresponding to the tenant, so as to obtain the corresponding relationship between the preset tenant and the pre-built data operating environment.

5. The method according to claim 4, characterized in that The pre-virtualized storage and computing resources include pre-virtualized distributed computing resources, pre-virtualized server computing resources and pre-virtualized storage resources; The preset virtualization rules of the distributed computing resources include: the virtualized distributed computing resources include multiple virtualization queues, and for each distributed computing resource cluster, the virtualization queue corresponding to the cluster includes a tenant queue, a cluster queue, a data operating environment queue, and a custom queue; The preset virtualization rules of the server computing resources include: the virtualized server computing resources include data integration instances and server local queues, wherein the data integration instance is a resource required to extract the data to be processed from the source end, convert the data to be processed and load it to the destination end, and the data integration instance is represented by the instance name, CPU specification, memory specification and the server to which it belongs; the server local queue is represented by the queue name, CPU specification and memory specification; The preset virtualization rules of the storage resources include: the virtualized storage resources are represented by storage resource names, storage resource types, storage resource URL connection strings, authentication methods, and attribute configuration information.

6. The method according to claim 5, characterized in that The preset virtualization rules of the distributed computing resources specifically include: The virtualized distributed computing resources include multiple virtualization queues. For each distributed computing resource cluster, the virtualization queue corresponding to the cluster has the tenant queue as the first-level parent queue, the cluster queue as the second-level parent queue, the data operating environment queue as the third-level queue, and the custom queue as the fourth-level queue; the custom queue is represented by the queue name, CPU specifications, and memory specifications of the actual queue corresponding to the custom queue.

7. A data processing task execution device, characterized in that: The device comprises: A task acquisition module is used to acquire the data processing tasks to be executed submitted by the user and determine the target tenant to which the user belongs; A data operating environment determination module, used to determine a target data operating environment for executing the data processing task based on the correspondence between the target tenant and the preset tenant and the pre-built data operating environment, wherein the data operating environment includes pre-virtualized storage and computing resources, and actual storage and computing resources corresponding to the pre-virtualized storage and computing resources corresponding to different tenants are virtually isolated from each other; A task execution module, used to call the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task; An instruction acquisition module, used to acquire an update instruction for a data operating environment, wherein the update instruction is used to instruct to create a new data operating environment for a target tenant or to expand the capacity of an existing data operating environment; an environment creation module, configured to determine whether the number of data operating environments corresponding to the target tenant reaches a first preset number when the update instruction is used to instruct to create a new data operating environment for the target tenant, and to create a new data operating environment for the target tenant when the number of data operating environments corresponding to the target tenant does not reach the first preset number; An expansion module is used to determine whether the number of pre-virtualized storage and computing resources included in the existing data operating environment has reached a second preset number when the update instruction is used to indicate the expansion of the existing data operating environment, and to expand the existing data operating environment when the number of pre-virtualized storage and computing resources included in the existing data operating environment has not reached the second preset number.

8. The device according to claim 7, characterized in that The device also includes: A resource display module, used for displaying the pre-virtualized storage and computing resources included in the target data operating environment before the step of calling the actual storage and computing resources corresponding to the pre-virtualized storage and computing resources included in the target data operating environment to execute the data processing task, wherein the storage and computing resources include computing resources and storage resources; A resource determination module, configured to determine, upon obtaining a resource selection instruction issued by a user based on the displayed storage and computing resources, a target storage resource selected by the user from the storage resources and a target computing resource selected by the user from the computing resources, wherein the target storage resource at least includes an output storage resource for storing a processing result of the data processing task; The task execution module comprises: A data acquisition unit, used to acquire the to-be-processed data corresponding to the data processing task; A resource calling unit, used to call the actual computing resource corresponding to the target computing resource, process the data to be processed, and obtain a processing result; A storage unit, used for storing the processing result in an actual storage resource corresponding to the output storage resource; The target storage resource also includes an input storage resource for storing the to-be-processed data corresponding to the data processing task; the data acquisition unit includes: A data acquisition subunit, used to acquire the to-be-processed data corresponding to the data processing task from the actual storage resources corresponding to the input storage resources; The device further includes a corresponding relationship building module, and the corresponding relationship building module includes: An information acquisition unit, used to acquire resource information of each actual storage and computing resource; A resource determination unit, used to determine, for each tenant, the actual storage and computing resources required by the tenant according to the business requirements of the tenant; A virtualization unit is used to virtualize the actual storage and computing resources required by each tenant according to preset virtualization rules and based on the resource information to obtain a virtualized storage and computing resource set as a resource pool corresponding to the tenant; A resource import unit, used to import virtualized storage and computing resources from the resource pool of the tenant into the constructed data operating environment corresponding to the tenant, so as to obtain a correspondence between the preset tenant and the pre-constructed data operating environment; The pre-virtualized storage and computing resources include pre-virtualized distributed computing resources, pre-virtualized server computing resources and pre-virtualized storage resources; The preset virtualization rules of the distributed computing resources include: the virtualized distributed computing resources include multiple virtualization queues, and for each distributed computing resource cluster, the virtualization queue corresponding to the cluster includes a tenant queue, a cluster queue, a data operating environment queue, and a custom queue; The preset virtualization rules of the server computing resources include: the virtualized server computing resources include data integration instances and server local queues, wherein the data integration instance is a resource required to extract the data to be processed from the source end, convert the data to be processed and load it to the destination end, and the data integration instance is represented by the instance name, CPU specification, memory specification and the server to which it belongs; the server local queue is represented by the queue name, CPU specification and memory specification; The preset virtualization rules of the storage resource include: the virtualized storage resource is represented by the storage resource name, storage resource type, storage resource URL connection string, authentication method and attribute configuration information; The preset virtualization rules of the distributed computing resources specifically include: The virtualized distributed computing resources include multiple virtualization queues. For each distributed computing resource cluster, the virtualization queue corresponding to the cluster has the tenant queue as the first-level parent queue, the cluster queue as the second-level parent queue, the data operating environment queue as the third-level queue, and the custom queue as the fourth-level queue; the custom queue is represented by the queue name, CPU specifications, and memory specifications of the actual queue corresponding to the custom queue.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-6 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images