Task processing method, device and equipment running on cloud computing platform

Abstract

The invention discloses a task processing method, device and equipment running on a cloud computing platform, and relates to the technical field of cloud computing. The data of the target object can be processed by the pod side of the service container group or the intelligent hierarchical cluster, so that the situation that only one cluster or system uniformly processes a large amount of data can be avoided, and the effectiveness of data processing is improved. The method comprises the steps that firstly, a cloud computing platform obtains task information of a to-be-processed task and processing capacity information of a service container group according to task configuration operation of a user on a configuration interface; then, the cloud computing platform allocates the to-be-processed task to target computing equipment for processing based on the task information of the to-be-processed task and the processing capability information of the service container group, and the target computing equipment is the service container group or an intelligent hierarchical cluster; and finally, the intelligent hierarchical cluster generates an access data table of the target object based on a processing result of the to-be-processed task, and the access data table is used for managing a storage area of the target object.

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on August 15, 2023, with application number 202311027148.X and invention name “A method, device and other equipment for data distribution”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of cloud computing technology, and in particular to a task processing method, device and equipment running on a cloud computing platform. Background Art

[0003] In the data storage system, different storage media correspond to different performances. By providing users with data reports, users can be assisted in choosing the medium in which to store their corresponding data. For example, users can choose to store hot data in a medium with higher performance and cold data in a medium with lower performance. In addition, users can also apply for corresponding services based on the data report. For example, if the amount of data in the data report is large, more equipment or resources can be applied to provide storage services for these data.

[0004] Currently, an agent can be deployed in each business container group (pod), and the agent sends the access data corresponding to each business pod to the intelligent hierarchical cluster, and the intelligent hierarchical cluster uniformly processes the access data reported by all business pods. After the intelligent hierarchical cluster completes processing of all access data, a series of data reports can be obtained.

[0005] However, in the above method, due to the large number of business pods, the scale of access data also increases. If only one intelligent hierarchical cluster processes a large amount of access data, the access data cannot be processed in time, affecting the effectiveness of data processing. Summary of the invention

[0006] The present application provides a task processing method, device and equipment running on a cloud computing platform, which solves the technical problem that a large amount of access data can not be processed in time because only one intelligent hierarchical cluster processes the access data.

[0007] In a first aspect, a task processing method running on a cloud computing platform is provided, wherein the cloud computing platform is deployed with a container management system, wherein an intelligent hierarchical cluster and one or more business container groups are deployed in the container management system, and the cloud computing platform also provides a configuration interface. The method comprises: first, the cloud computing platform obtains task information of a task to be processed and processing capacity information of a business container group according to a task configuration operation of a user in the configuration interface, wherein the task configuration operation indicates intelligent hierarchical processing of data of a target object; then, the cloud computing platform allocates the task to be processed to a target computing device for processing based on the task information of the task to be processed and the processing capacity information of the business container group, wherein the target computing device is a business container group or an intelligent hierarchical cluster; finally, the intelligent hierarchical cluster generates an access data table of the target object based on the processing result of the task to be processed, wherein the access data table is used to manage a storage area of ​​the target object.

[0008] It can be understood that the method of the first aspect can be executed by a computing device, which can be a terminal, an apparatus including a terminal, or a chip in a terminal, or the computing device can also be a network device, an apparatus including a network device, or a chip in a network device. For ease of understanding, the following description is taken as an example of execution by a computing device.

[0009] In the present application, after the cloud computing platform obtains the task information of the pending task and the processing capacity information of the business container group deployed in the cloud computing platform, it can allocate the pending task to the target computing device for processing based on the task information and the processing capacity information. Since the target computing device is an intelligent hierarchical cluster deployed in the business container group or the cloud computing platform, the data of the target object can be processed by the business pod side or the intelligent hierarchical cluster, which can avoid only one cluster or system processing a large amount of data uniformly, thereby improving the effectiveness of data processing. And the intelligent hierarchical cluster can generate an access data table for the target object based on the processing results of the pending task. Since the access data table is used to manage the storage area of ​​the target object, the user can apply for more storage resources for the target object's data based on the access data table, or identify hot data based on the access data table and apply to store the hot data in a storage medium with higher performance, which can provide customized service needs for users.

[0010] Optionally, the cloud computing platform allocates the task to be processed to the target computing device for processing based on the task information of the task to be processed and the processing capability information of the business container group, specifically including: the cloud computing platform matches the processing capability information of the business container group with the task information of the task to be processed; the cloud computing platform determines that the processing capability information of the business container group matches the task information of the task to be processed successfully, and allocates the task to be processed to the business container group for processing; the cloud computing platform determines that the processing capability information of the business container group fails to match the task information of the task to be processed, and allocates the task to be processed to the intelligent hierarchical cluster for processing.

[0011] In this application, if the processing capability information of the business container group matches the task information of the task to be processed successfully, it means that the business container group has the processing capability of the task to be processed. At this time, the cloud computing platform assigns the task to be processed to the business container group for processing, which can reduce the processing pressure of the intelligent hierarchical cluster while processing the task to be processed. If the processing capability information of the business container group fails to match the task information of the task to be processed, it means that the business container group does not have the processing capability of the task to be processed. At this time, the cloud computing platform assigns the task to be processed to the intelligent hierarchical cluster for processing, which can ensure the effectiveness of task processing.

[0012] Optionally, the task processing method also includes: the cloud computing platform splits the task to be processed to obtain multiple subtasks to be processed; the cloud computing platform allocates at least one subtask to the business container group for processing based on the processing capability information of the business container group and the task information of the multiple subtasks to be processed, and allocates other subtasks to the intelligent hierarchical cluster for processing.

[0013] In this application, the cloud computing platform can instruct the business container group and the intelligent hierarchical cluster to collaboratively complete the processing of the pending tasks, specifically, the business container group processes at least one subtask, and the intelligent hierarchical cluster processes other subtasks. It can fully utilize the processing capabilities of the intelligent hierarchical cluster and the business pod side, greatly saving the processing cost of the cloud computing platform, avoiding the processing of all data by a single cluster (or system), and improving the efficiency of data processing.

[0014] Optionally, the above-mentioned cloud computing platform is also deployed with a data acquisition system, and the task processing method also includes: the cloud computing platform obtains the acquisition performance information of the data acquisition system; the cloud computing platform instructs the data acquisition system to collect the data of the target object from the business container group based on the acquisition performance information of the data acquisition system; the data acquisition system sends the data of the target object to the intelligent hierarchical cluster.

[0015] In this application, the cloud computing platform can instruct the data collection system to actively collect the data of the target object from the business container group based on the collection performance information of the data collection system, and can effectively control the data collection order to ensure that the data collection system does not exceed its own collection capacity. In addition, the data collection system can send the data of the target object to the intelligent grading system, so that the intelligent grading system can obtain the processing results of the task to be processed based on the data of the target object, which can ensure that the data collected by the data collection system will not exceed the maximum processing capacity of the intelligent grading cluster, reducing the processing pressure of the intelligent grading cluster.

[0016] Optionally, after the cloud computing platform assigns the pending tasks to the business container group for processing, the task processing method further includes: the business container group processes the pending tasks to obtain processing results of the pending tasks.

[0017] In this application, the business container group processes the pending tasks (or the data of the target object) to obtain the processing results of the pending tasks. That is, the processing process of the pending tasks can be completed by the business pod side alone, which improves the effectiveness of data processing. It can also reduce the processing pressure of the intelligent hierarchical cluster and improve the processing capacity of the intelligent hierarchical cluster.

[0018] Optionally, after the cloud computing platform assigns the tasks to be processed to the business container group for processing, the task processing method also includes: the business container group processes the tasks to be processed to obtain intermediate results of the tasks to be processed; the data acquisition system collects the intermediate results of the tasks to be processed from the business container group, and sends the intermediate results of the tasks to be processed to the intelligent grading cluster; the intelligent grading cluster obtains the processing results of the tasks to be processed based on the intermediate results of the tasks to be processed.

[0019] In this application, the business container group and the intelligent hierarchical cluster jointly complete the processing process of the pending task, which can make full use of the data coupling characteristics of the business pod side to obtain the intermediate results of the pending task in advance, so that the intelligent hierarchical cluster can obtain the processing result (or final result) of the pending task based on the intermediate result. It can greatly reduce the processing scale and complexity of the intelligent hierarchical cluster.

[0020] Optionally, the target computing device includes an operator container, which is deployed in the target computing device in the form of a container image. The cloud computing platform allocates the tasks to be processed to the target computing device for processing, specifically including: the cloud computing platform allocates the tasks to be processed to the operator container for processing, and obtains the processing results of the tasks to be processed.

[0021] In this application, the computing device distributes the tasks to be processed to the operator container in the target computing device, and the operator container processes the tasks to be processed to obtain the processing results of the tasks to be processed. It can accurately control the operators in the operator container that specifically participate in the operation, provide processing capabilities that match the tasks to be processed, and avoid insufficient or wasted processing capabilities while effectively completing task processing.

[0022] Optionally, the task information of the above-mentioned task to be processed includes the data volume of the task to be processed, and the task processing method also includes: the cloud computing platform performs an expansion operation or a reduction operation on the operator container included in the target computing device based on the data volume of the task to be processed.

[0023] In this application, when the data volume of the task to be processed is large, the target computing device (specifically, the operator container included in the target computing device) may not be able to meet the processing process of the task to be processed, and the cloud computing platform can perform a capacity expansion operation on the operator container included in the target computing device to improve the processing capacity of the target computing device. Correspondingly, when the data volume of the task to be processed is small, the target computing device (specifically, the operator container included in the target computing device) can easily complete the processing process of the task to be processed, and the cloud computing platform can perform a capacity reduction operation on the operator container included in the target computing device to save the processing resources of the target computing device.

[0024] In the second aspect, a task processing device running on a cloud computing platform is provided, the cloud computing platform is deployed with a container management system, the container management system is deployed with an intelligent hierarchical cluster and one or more business container groups, the cloud computing platform also provides a configuration interface, the task processing device includes an acquisition module and a first processing module, and the intelligent hierarchical cluster includes a second processing module. The acquisition module is used to acquire the task information of the task to be processed and the processing capacity information of the business container group according to the task configuration operation of the user in the configuration interface, wherein the task configuration operation indicates intelligent hierarchical processing of the data of the target object; the first processing module is used to allocate the task to be processed to the target computing device for processing based on the task information of the task to be processed and the processing capacity information of the business container group, the target computing device is a business container group or an intelligent hierarchical cluster; the second processing module is used to generate an access data table of the target object based on the processing result of the task to be processed, and the access data table is used to manage the storage area of ​​the target object.

[0025] In addition, the technical effects of the task processing device described in the second aspect can refer to the technical effects of the method described in the first aspect, and will not be repeated here.

[0026] In a third aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in the first aspect.

[0027] According to a fourth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect.

[0028] In a fifth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram showing the principle of intelligent hierarchical cluster processing of data provided by an embodiment of the present application is shown;

[0030] Figure 2 A schematic diagram of the internal structure of a computing device provided in an embodiment of the present application is shown;

[0031] Figure 3 A schematic diagram of a process flow of a task processing method provided in an embodiment of the present application is shown;

[0032] Figure 4 A schematic diagram of a configuration interface provided by an embodiment of the present application is shown;

[0033] Figure 5 A flowchart of another task processing method provided in an embodiment of the present application is shown;

[0034] Figure 6 A flowchart of another task processing method provided in an embodiment of the present application is shown;

[0035] Figure 7 A schematic diagram showing a principle of performing data cleaning on data in a business container group provided by an embodiment of the present application is shown;

[0036] Figure 8 A schematic diagram showing a principle of data conversion and aggregation processing provided by an embodiment of the present application is shown;

[0037] Fig. 9 A flowchart of another task processing method provided in an embodiment of the present application is shown;

[0038] Fig.10 A schematic diagram showing a principle of expanding the capacity of an operator container provided in an embodiment of the present application is shown;

[0039] Fig.11 A schematic diagram of another configuration interface provided in an embodiment of the present application is shown;

[0040] Fig.12 A schematic diagram showing the principle of a data collection system provided by an embodiment of the present application collecting data in batches from a business container group;

[0041] Fig.13 A schematic diagram showing a principle of allocating multiple subtasks to an intelligent hierarchical cluster and a business container group for processing provided by an embodiment of the present application is shown;

[0042] Fig.14 A schematic diagram of the structure of a task processing device provided in an embodiment of the present application is shown;

[0043] Fig.15 A schematic diagram of the structure of a computing device provided in an embodiment of the present application is shown;

[0044] Fig.16 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application is shown;

[0045] Fig.17 A schematic diagram of a network architecture of a computing device cluster provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0046] To facilitate understanding of the solutions provided by the embodiments of the present application, some concepts involved in the present application are explained before introducing the solutions provided by the embodiments of the present application.

[0047] Kubernetes, or K8S for short, is an open source container management system (or container orchestration engine) for automating the deployment, expansion, and management of containerized applications. In a K8S cluster, a container group (pod) is the basis of all business types and the smallest deployable unit in K8S. It is a combination of one or more containers. A container can be understood as a simplified Linux environment and the applications running in it. Containers are isolated from each other. A K8S cluster can include one or more business container groups, and the one or more business containers included in the business container group are used to store object data.

[0048] The intelligent hierarchical cluster (or central cluster) is the management and control center of the K8S cluster. It consists of master nodes and is responsible for cluster management and scheduling. One or more operator containers can be deployed in the intelligent hierarchical cluster. The operator container can process a task and obtain the processing result of the task. In addition, an operator container can include one or more operators, and an operator is a mapping from a function space to a function space, which is the computing unit that constitutes the algorithm.

[0049] Object storage is a way of storing data. It stores data and its related metadata as a whole in units of objects. A "bucket" is a container used to organize and store objects, which can store multiple objects like a folder. Object storage provides a flat storage method based on buckets and objects, removing the multi-level tree directory structure in traditional file systems, making data storage and management simpler and more efficient.

[0050] In the data storage system, different storage media correspond to different performances. The intelligent hierarchical cluster needs to analyze massive amounts of data, determine the hotness or coldness of the current object to be stored, and provide data reports to users, thereby assisting users in choosing which medium to store their corresponding data. For example, users can choose to store hot data in a medium with higher performance and cold data in a medium with lower performance to solve the cost problem as a whole. In addition, users can also apply for corresponding services based on the data report. For example, if the amount of data in the data report is large, more equipment or resources can be applied to provide storage services for these data.

[0051] Since the scale of data access is extremely large and the execution nodes operating on the same object bucket are scattered, the relevant data may be scattered among many nodes. The intelligent tiered cluster needs to generate various data reports by reading metadata, access data, configuration files and other types of data. In a container-based cloud native environment, a large amount of data is generated in each container, and the scale of the container is much larger than that of the physical machine, so we need a more efficient intelligent tiered processing method.

[0052] In a possible implementation, an agent can be deployed in each business container group. The agent sends the access data corresponding to each business pod to the data collection system. After the data collection system sends the access data to the intelligent hierarchical cluster, the intelligent hierarchical cluster uniformly processes the access data reported by all business pods. After the intelligent hierarchical cluster completes processing of all access data, a series of data reports can be obtained.

[0053] Specifically, Figure 1 As shown, the data processing device includes a business container group 101, a data collection system 102, an intelligent hierarchical cluster 103 and a report display module 104. The business container group 101 includes a business container 1011, a file system 1012 and an agent module 1013.

[0054] Specifically, the business container 1011 is used to store the user's access data. After the file system 1012 obtains the user's access data from the business container 1011, it actively pushes the access data to the data collection system 102 via the proxy module 1013. Then, the intelligent hierarchical cluster 103 uniformly processes the access data sent by the data collection system 102. After the intelligent hierarchical cluster 103 processes the access data, it can generate relevant data reports and send these data reports to the report display module 104. Finally, the report display module 104 displays these data reports.

[0055] In the above implementation, each business container group actively pushes data to the data collection system through its own internally deployed proxy module, and there is no unified data collection logic. In the cloud native container scenario, there are many business container groups. This active push method is likely to cause a large number of business container groups to push data to the data collection system at the same time at a certain moment, exceeding the capacity of the data collection system, resulting in data push queues or even loss.

[0056] In addition, due to the large number of business container groups, the scale of access data also increases. If only one intelligent hierarchical cluster uniformly processes the access data reported by all business container groups, the processing capacity of the intelligent hierarchical cluster will become a bottleneck, and the access data cannot be processed in a timely manner, affecting the effectiveness of data processing.

[0057] At the same time, since the data is generated and reported by the business container group, the data within the business container group is highly coupled. The above implementation method does not fully utilize the processing capabilities of the business container group to process the data in advance, resulting in a considerable portion of the capabilities of the intelligent hierarchical cluster being used for filtering unimportant data, converting data formats, and merging related data, further affecting the processing capabilities of the intelligent hierarchical cluster.

[0058] Based on this, an embodiment of the present application provides a task processing method running on a cloud computing platform. After the cloud computing platform obtains the task information of the task to be processed and the processing capacity information of the business container group deployed in the cloud computing platform, the cloud computing platform can allocate the task to be processed to the target computing device for processing based on the task information and the processing capacity information. Since the target computing device is an intelligent hierarchical cluster deployed in the business container group or the cloud computing platform, the data of the target object can be processed by the business pod side or the intelligent hierarchical cluster, which can avoid only one cluster or system processing a large amount of data uniformly, thereby improving the effectiveness of data processing. And the intelligent hierarchical cluster can generate an access data table of the target object based on the processing results of the task to be processed. Since the access data table is used to manage the storage area of ​​the target object, the user can apply for more storage resources for the data of the target object based on the access data table, or identify hot data based on the access data table and apply to store the hot data in a storage medium with higher performance, which can provide customized service needs for users.

[0059] The task processing method, device and equipment running on the cloud computing platform provided in the embodiments of the present application are applied to task processing (or data processing) scenarios (including but not limited to cloud native container scenarios, container business deployment scenarios and distributed storage scenarios). After the cloud computing platform obtains the task information of the task to be processed and the processing capacity information of the business container group, the task to be processed can be assigned to the target computing device for processing, and the intelligent hierarchical cluster can generate access data of the target object based on the processing results of the task to be processed.

[0060] In order to enable ordinary persons in the art to better understand the technical solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0061] It is understood that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0062] It should also be understood that the term “comprising” indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or components.

[0063] In one example, the cloud computing platform that executes the task processing method provided in the embodiment of the present application can be a terminal, which can also be called user equipment (UE), access terminal, subscriber unit (subscriber unit), user station, mobile station (MS), mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device. The terminal in the embodiments of the present application can be a mobile phone, a cellular phone, a smart phone, a tablet computer, a wireless data card, a personal digital assistant (PDA), a wireless modem, a handset, a laptop computer, a machine type communication (MTC) terminal, a computer with a wireless transceiver function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a smart home device (for example, a refrigerator, a television, an air conditioner, an electric meter, etc.), an intelligent robot, a robotic arm, a workshop equipment, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a vehicle-mounted terminal, a roadside unit with a terminal function, and a wireless terminal in a smart city. unit, RSU), etc., flying equipment (e.g., intelligent robots, hot air balloons, drones, airplanes), etc. The terminal of the present application may also be an on-board module, on-board module, on-board component, on-board chip or on-board unit built into the vehicle as one or more components or units. The terminal may also be other devices with terminal functions, for example, the terminal may also be a device that functions as a terminal in device to device (D2D) communication.

[0064] The embodiments of the present application do not limit the device form of the terminal. The device for realizing the function of the terminal can be a terminal; it can also be a device that can support the terminal to realize the function, such as a chip system. The device can be installed in the terminal, or used in combination with the terminal. In the embodiments of the present application, the chip system can be composed of chips, or it can include chips and other discrete devices.

[0065] In another case, the above-mentioned cloud computing platform can also be a server, which can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (content delivery network, CDN), as well as big data and artificial intelligence platforms.

[0066] like Figure 2 As shown, in one implementation of the embodiment of the present application, the cloud computing platform for executing the task processing method provided in the embodiment of the present application may include a scheduler 201, a business container group 202, a data acquisition system 203, a message queue 204, an intelligent hierarchical cluster 205, and a configuration interface 206. Among them, the business container group 202 includes an operator container 2021, a business container 2022, and a file system 2023, and the intelligent hierarchical cluster 205 includes an operator container 2051.

[0067] Specifically, the scheduler 201 can be understood as a management device (or control device) in the cloud computing platform. The scheduler 201 is used to allocate the tasks to be processed to the business container group 202 or the intelligent hierarchical cluster 205 for processing based on the task information of the tasks to be processed and the processing capacity information of the business container group 202.

[0068] The operator container 2021 includes N operators (N is greater than or equal to 1), which can process pending tasks.

[0069] The business container 2022 is used to store the data of the target object.

[0070] The file system 2023 is a data sharing channel between the business container 2022 and the operator container 2021 , which facilitates the operator container 2021 to obtain the data of the target object from the business container 2022 .

[0071] The data collection system 203 is used to collect data of the target object from the business container group 202 .

[0072] The message queue 204 is a message transmission channel between the data acquisition system 203 and the intelligent grading cluster 205, and is used to transmit the data of the target object and the intermediate results of the task to be processed.

[0073] The intelligent classification cluster 205 is used to generate an access data table of the target object based on the processing result of the task to be processed.

[0074] The operator container 2051 includes M operators (M is greater than or equal to 1), which can also process pending tasks.

[0075] The configuration interface 206 is used to display the access data table of the target object.

[0076] Based on the above Figure 2 The cloud computing platform shown in the figure, the task processing method provided in the embodiment of the present application can be run on the cloud computing platform to realize the process of generating the access data table of the target object. Specifically, the cloud computing platform is deployed with a container management system, in which an intelligent hierarchical cluster and one or more business container groups are deployed, and the cloud computing platform also provides a configuration interface. Figure 3 As shown, the task processing method may include steps S101-S103.

[0077] Step S101: The cloud computing platform obtains task information of the task to be processed and processing capacity information of the business container group according to the task configuration operation of the user in the configuration interface.

[0078] The task configuration operation indicates intelligent classification processing of the target object's data.

[0079] It should be understood that the task configuration operation of the user in the configuration interface can specifically be that the user performs preset operations (such as input, pull-down, click and other selection operations) on the configuration items related to the task to be processed on the configuration interface, thereby determining the configuration information of the task to be processed.

[0080] Optionally, the configuration information of a task may include a combination of one or more of the following information: the identifier of the object (or bucket object), the identifier of the user (or tenant), the time interval corresponding to the object, the number of times the object is accessed, and the region (or area) to which the object belongs, etc.

[0081] For example, in Figure 4 The configuration interface shown includes four configuration items, namely object, tenant, region, and access attribute. The user enters the ID of object A, the ID of tenant B, the names of four regions (including Beijing, Shanghai, Shenzhen, and Xi'an), and the number of visits in the input boxes corresponding to the four configuration items. That is, the configuration information of the task to be processed can be generated, including the ID of object A, the ID of tenant B, the names of the four regions, and the number of visits to object A.

[0082] It should be understood that the task information of the task to be processed may include the configuration information of the task to be processed. The processing capability information of the service container group is used to characterize the task processing capability of the service container group.

[0083] Step S102: The cloud computing platform allocates the task to be processed to the target computing device for processing based on the task information of the task to be processed and the processing capability information of the business container group.

[0084] The target computing device is one or more of the above-mentioned business container groups or intelligent hierarchical clusters.

[0085] like Figure 5 As shown, the cloud computing platform allocates the task to be processed to the target computing device for processing based on the task information of the task to be processed and the processing capability information of the business container group, which may specifically include steps S1021-S1023.

[0086] Step S1021: The cloud computing platform matches the processing capability information of the business container group with the task information of the task to be processed.

[0087] In an optional implementation, the processing capability information of the business container group includes a combination of one or more of the following information: central processing unit (CPU) resources that can be allocated to the business container group, memory resources that can be allocated to the business container group, bandwidth of the business container group, disk pressure of the business container group, queue depth corresponding to the business container group, and operator types included in the business container group.

[0088] In another optional implementation, the task information of the task to be processed may include a combination of one or more of the following information: the CPU resources of the task to be processed, the memory resources occupied by the task to be processed, the bandwidth resources occupied by the task to be processed, the disk pressure occupied by the task to be processed, and the operator type corresponding to the task to be processed.

[0089] Step S1022: The cloud computing platform determines that the processing capability information of the business container group successfully matches the task information of the task to be processed, and allocates the task to be processed to the business container group for processing.

[0090] It should be understood that the cloud computing platform determines that the processing capability information of the business container group matches the task information of the task to be processed, which means that the task processing capability of the business container group matches the task information. It can also be understood that the business container group has the processing capability of the task to be processed. At this time, the cloud computing platform assigns the task to be processed to the business container group for processing, specifically, the business pod side processes the task to be processed. The processing process of the task to be processed can be effectively completed, so that the data of the target object is processed by the business pod side, which can improve the effectiveness of data processing.

[0091] In one case, when there is only one business container group whose processing capability information successfully matches the task information of the task to be processed in the one or more business container groups, the cloud computing platform may assign the task to be processed to the business container group for processing.

[0092] In another case, when the number of business container groups whose processing capability information in the one or more business container groups successfully matches the task information of the task to be processed is more than one (hereinafter referred to as at least two business container groups), the cloud computing platform can determine the capability level (or capability evaluation index) of each business container group based on the processing capability information of each business container group in the at least two business container groups, and assign the task to be processed to the business container group with the largest capability level among the at least two business container groups for processing.

[0093] Step S1023: The cloud computing platform determines that the processing capability information of the business container group fails to match the task information of the task to be processed, and allocates the task to be processed to the intelligent hierarchical cluster for processing.

[0094] It should be understood that the cloud computing platform fails to determine that the processing capacity information of the business container group matches the task information of the task to be processed, indicating that the task processing capacity of the business container group does not match the task information, which can also be understood as the business container group does not have the processing capacity of the task to be processed. At this time, the cloud computing platform assigns the task to be processed to the intelligent hierarchical cluster for processing. The processing efficiency of the task to be processed can be improved, so that the data of the target object is processed by the intelligent hierarchical cluster, which can improve the efficiency of data processing.

[0095] In the embodiment of the present application, if the processing capability information of the business container group matches the task information of the task to be processed successfully, it means that the business container group has the processing capability of the task to be processed. At this time, the cloud computing platform assigns the task to be processed to the business container group for processing, which can reduce the processing pressure of the intelligent hierarchical cluster while processing the task to be processed. If the processing capability information of the business container group fails to match the task information of the task to be processed, it means that the business container group does not have the processing capability of the task to be processed. At this time, the cloud computing platform assigns the task to be processed to the intelligent hierarchical cluster for processing, which can ensure the effectiveness of task processing.

[0096] Optionally, each item of information included in the processing capability information of a business container group may have a priority, and each item of information included in the task information of a task (e.g., a task to be processed) may also have a priority. When multiple business container groups are deployed in the above container management system, the cloud computing platform may screen the multiple business container groups according to the priority order of one or more items of information included in the processing capability information of each of the multiple business container groups and / or the priority order of one or more items of information included in the task information of the task to be processed, so as to determine the business container group that processes the task to be processed.

[0097] Specifically, for one or more items of information included in the processing capability information of each of the multiple business container groups, the cloud computing platform may screen the multiple business container groups in order of priority from high to low, and determine the business container group whose processing capability information among the multiple business container groups successfully matches the task information of the task to be processed as the business container group that processes the task to be processed, and assign the task to be processed to the business container group for processing.

[0098] In one embodiment, if the processing capability information of a business container group includes the CPU resources that can be allocated to the business container group, the memory resources that can be allocated to the business container group, the bandwidth of the business container group, the disk pressure of the business container group, the queue depth corresponding to the business container group, and the operator type included in the business container group, then the priority of the multiple pieces of information included in the processing capability information of the business container group is in the following order from high to low: the operator type included in the business container group, the CPU resources that can be allocated to the business container group, the memory resources that can be allocated to the business container group, the queue depth corresponding to the business container group, the bandwidth of the business container group, and the disk pressure of the business container group.

[0099] In another embodiment, if the task information of the pending task includes the CPU resources occupied by the pending task, the memory resources occupied by the pending task, the bandwidth occupied by the pending task, the disk pressure occupied by the pending task, and the operator type corresponding to the pending task, then the priority of the multiple pieces of information included in the task information is in the following order from high to low: the operator type corresponding to the pending task, the CPU resources occupied by the pending task, the memory resources occupied by the pending task, the bandwidth occupied by the pending task, and the disk pressure occupied by the pending task.

[0100] In the embodiment of the present application, the computing device can screen the multiple business container groups according to the priority order of one or more information included in the processing capability information of each of the multiple business container groups and / or the priority order of one or more information included in the task information of the task to be processed, to determine the business container group that processes the task to be processed. That is, the target computing device that processes the task to be processed can be determined according to a certain priority order or screening order, which can improve the orderliness of determining the target computing device, thereby improving the orderliness of task processing.

[0101] In an optional implementation, the processing capacity information of a business container group includes the CPU resources that can be allocated to the business container group and the operator types included in the business container group, and the task information of the pending task includes the CPU resources occupied by the pending task and the operator type corresponding to the pending task. The above-mentioned cloud computing platform allocates the pending task to the business container group for processing, which can specifically include the cloud computing platform determining at least one business container group from multiple business container groups, and allocating the pending task to a target business container group in the at least one business container group for processing. Among them, the operator types included in the at least one business container group include the operator type corresponding to the pending task, and the target business container group is any business container group in the at least one business container group whose allocable CPU resources meet the CPU resources occupied by the pending task.

[0102] It should be understood that the cloud computing platform can first preliminarily screen multiple business container groups based on the operator types included in each of the multiple business container groups and the operator types corresponding to the tasks to be processed, that is, screen out at least one business container group whose included operator types successfully match the operator types corresponding to the tasks to be processed. Then, from the at least one business container group, a target business container group whose allocatable CPU resources meet the CPU resources occupied by the tasks to be processed is screened out, and the tasks to be processed are assigned to the target business container group for processing. The target computing device for processing the tasks to be processed can be conveniently and quickly determined based on the specific content included in the processing capability information of the business container group and the specific content included in the task information of the tasks to be processed, further improving the efficiency of data processing.

[0103] In one implementation of the embodiment of the present application, after the cloud computing platform assigns the pending task to the business container group for processing, the business container group can complete the processing of the pending task alone, that is, the business container group can process the pending task and obtain the processing result of the pending task.

[0104] It should be understood that the business container group processing the pending task can also be understood as the business container group processing the data of the target object. The processing result of the pending task is the final result obtained after processing the pending task.

[0105] In the embodiment of the present application, the business container group processes the pending task (or the data of the target object) to obtain the processing result of the pending task. That is, the processing process of the pending task can be completed by the business pod side alone, which improves the effectiveness of data processing. It can also reduce the processing pressure of the intelligent hierarchical cluster and improve the processing capacity of the intelligent hierarchical cluster.

[0106] In another implementation of the embodiment of the present application, after the cloud computing platform assigns the pending tasks to the business container group for processing, the business container group and the intelligent hierarchical cluster can jointly complete the processing of the pending tasks. Figure 6 As shown, the task processing method provided in the embodiment of the present application also includes steps S104-S106.

[0107] Step S104: The business container group processes the task to be processed and obtains an intermediate result of the task to be processed.

[0108] Step S105: The data collection system collects the intermediate results of the tasks to be processed from the business container group, and sends the intermediate results of the tasks to be processed to the intelligent grading cluster.

[0109] It should be understood that the data collection system is a data collection system deployed in a cloud computing platform. The data collection system can actively collect intermediate results of tasks to be processed from the business container group.

[0110] Step S106: The intelligent grading cluster obtains the processing result of the task to be processed according to the intermediate result of the task to be processed.

[0111] In one case, the cloud computing platform processes the pending task and specifically includes the cloud computing platform performing data cleaning on the pending task. The data cleaning may be to delete data that the user does not care about and retain data that the user cares about, and the data that the user cares about is the intermediate result of the pending task. After the data collection system collects the cleaned data from the business container group, the intelligent hierarchical cluster aggregates (or integrates) the cleaned data to obtain the processing result of the pending task.

[0112] Optionally, the data that the user is concerned about is data related to the configuration information of the above-mentioned task to be processed.

[0113] For example, Figure 7 As shown, it is assumed that two business container groups are deployed in the cloud computing platform, including business container group 1 and business container group 2. Business container group 1 stores data 1, data 2, and data 3, and business container group 2 stores data 4, data 5, and data 6. The data that the user cares about (or needs to keep) includes object identifiers, user identifiers, and operations performed by users on objects.

[0114] In this way, for business container group 1, the business container group 1 can delete the task identifier (including task-1 and task-3), time (including 100, 200, 300) and result (including success and failure), and the data (or intermediate results) obtained (or retained) by business container group 1 include the object identifier (i.e. 00001), the user identifier (including user 1 and user 2) and the operations performed by the user on the object (including creation and deletion).

[0115] For business container group 2, the business container group 2 can also delete the task identifier (including task-2 and task-4), time (including 300, 200, 100) and result (including success and failure), and the obtained data includes the object identifier (including 00001 and 00002), user identifier (including user 1 and user 2) and the user's operations on the object (including creation and modification).

[0116] In addition, for each of the two business container groups, each business container group can also integrate the cleaned data. For example, object ID 00001, create (2), delete (1) is used to indicate that the relevant user has performed 2 creation operations and 1 deletion operation on the object with ID 00001 in business container group 1.

[0117] Furthermore, the data collection system can collect integrated data from each of the two business container groups, and send the integrated data of each business container group to the intelligent hierarchical cluster. After acquiring the integrated data of each business container group, the intelligent hierarchical cluster can perform aggregation processing. For example, user identification: user 1, create (2), modify (1), delete (1) is used to indicate that user 1 has performed 2 creation operations, 1 modification operation, and 1 deletion operation on the related objects in the two business container groups.

[0118] In another case, the cloud computing platform processes the pending task, specifically including the cloud computing platform performing data conversion on the pending task to generate a basic indicator, which is the intermediate result of the pending task. The data collection system then collects the basic indicator from the business container group and sends the basic indicator to the intelligent grading cluster. Finally, the intelligent grading cluster aggregates the basic indicators to obtain the final indicator. The final indicator is the processing result (or final result) of the pending task.

[0119] For example, Figure 8 As shown, it is assumed that five business container groups are deployed in the cloud computing platform, and each of the five business container groups stores one million pieces of data.

[0120] In one case, the five business container groups first actively push 1 million pieces of data stored in each of them to the data collection system, and then the data collection system sends five pieces of 1 million pieces of data (i.e. 5 million pieces of data) to the intelligent grading cluster. After obtaining the 5 million pieces of data, the intelligent grading cluster processes the 5 million pieces of data and obtains 5,000 indicators.

[0121] In another case, each of the five business container groups first performs data conversion on the 1 million pieces of data stored therein to generate 500 basic indicators. After that, the data collection system actively collects 500 basic indicators (i.e., 2500 basic indicators) of each of the five business container groups from the five business container groups, and sends the 2500 basic indicators to the intelligent grading cluster. After obtaining the 2500 basic indicators, the intelligent grading cluster aggregates the 2500 basic indicators to obtain 5000 final indicators.

[0122] In the embodiment of the present application, the business container group and the intelligent hierarchical cluster jointly complete the processing process of the pending task, which can make full use of the data coupling characteristics of the business pod side to obtain the intermediate result of the pending task in advance, so that the intelligent hierarchical cluster can obtain the processing result (or final result) of the pending task based on the intermediate result. It can greatly reduce the processing scale and complexity of the intelligent hierarchical cluster.

[0123] In one implementation of the embodiment of the present application, the target computing device includes an operator container, which is deployed in the target computing device in the form of a container image. The processing process of the target computing device on the task to be processed can be completed by the operator container. Fig. 9 As shown, the cloud computing platform distributes the tasks to be processed to the target computing device for processing, which may specifically include step S1024.

[0124] Step S1024: The cloud computing platform allocates the tasks to be processed to the operator container for processing, and obtains the processing results of the tasks to be processed.

[0125] It should be understood that the target computing device may include one or more operator containers, and one operator container may include one or more operators. After the cloud computing device assigns the task to be processed to the operator container, the operator container can perform computation processing on the task to be processed based on the operator corresponding to the task to be processed, that is, the processing result of the task to be processed can be obtained.

[0126] It can be understood that the type of operator containers included in the above-mentioned intelligent hierarchical cluster and the type of operator containers included in the business container group may be the same or different, and the number of operator containers included in the intelligent hierarchical cluster and the number of operator containers included in the business container group may be the same or different. The embodiment of the present application does not specifically limit the type and number of operator containers included in the target computing device.

[0127] In the embodiment of the present application, the cloud computing device distributes the tasks to be processed to the operator container in the target computing device, and the operator container processes the tasks to be processed to obtain the processing results of the tasks to be processed. The operators in the operator container that are specifically involved in the operation can be accurately controlled to provide processing capabilities that match the tasks to be processed, and while effectively completing the task processing, the lack or waste of processing capabilities can be avoided.

[0128] In order to fully utilize the processing power of the cloud computing platform, the cloud computing platform can dynamically adjust the processing power of the operator container (or operator) included in the target computing device to achieve adaptive task processing capabilities. Based on this, the task information of the above-mentioned task to be processed can also include the data volume of the task to be processed. The cloud computing platform can perform expansion or reduction operations on the operator container included in the target computing device based on the data volume of the task to be processed.

[0129] It should be understood that the data volume of the task to be processed is the data volume required to process the task to be processed.

[0130] It is understandable that the size of the data volume of the task to be processed can be 1 kilobyte (kb), and the size of the data volume of the task to be processed can also be 1 megabyte (mb). The embodiment of the present application does not specifically limit the size of the data volume of the task to be processed.

[0131] In one case, when the data volume of the task to be processed is greater than or equal to the data volume threshold, it means that the data volume of the task to be processed is large, and the target computing device (specifically, the operator container included in the target computing device) may not be able to meet the processing process of the task to be processed. At this time, the cloud computing platform can perform a capacity expansion operation on the operator container included in the target computing device to improve the processing capability of the target computing device.

[0132] In another case, when the data volume of the task to be processed is less than the data volume threshold, it means that the data volume of the task to be processed is small, and the target computing device (specifically, the operator container included in the target computing device) can easily complete the processing process of the task to be processed. At this time, the cloud computing platform can perform a capacity reduction operation on the operator container included in the target computing device to save the processing resources of the target computing device.

[0133] For example, Fig.10 As shown, the cloud computing platform includes a scheduler, seven processing tasks (including task 1, task 2, task 3, task 4, task 5, task 6 and task 7) and four business container groups (including business container group 1, business container group 2, business container group 3 and business container group 4). Business container group 1 includes four operator containers, namely operator container 1, operator container 2, operator container 3 and operator container 4.

[0134] Based on the task information of each of the seven processing tasks and the processing capacity information of each of the four business container groups, the scheduler assigns Task 1, Task 2, and Task 3 to business container group 1 for processing, and assigns Task 4 and Task 5 to business container group 2 for processing, and assigns Task 6 and Task 7 to business container group 3 and business container group 4 for processing, respectively.

[0135] Assume that Task 1, Task 2, and Task 3 constitute the above-mentioned pending tasks, and the data volume of the pending tasks is greater than or equal to the data volume threshold. At this time, the cloud computing platform performs a capacity expansion operation on the operator containers included in the business container group 1, and the final business container group 1 includes 8 operator containers, namely, operator container 1, operator container 2, operator container 3, operator container 4, operator container 5, operator container 6, operator container 7, and operator container 8.

[0136] Step S103: The intelligent hierarchical cluster generates an access data table of the target object based on the processing results of the tasks to be processed.

[0137] The access data table is used to manage the storage area of ​​the target object.

[0138] In an optional implementation, the intelligent hierarchical cluster may send the access data table of the target object to the configuration interface, so that the configuration interface displays the access data table.

[0139] It should be understood that after the configuration interface displays the access data table of the target object, the user can make some decisions or operations on the data of the target object based on the access data table.

[0140] In one case, the user may apply for more storage resources or storage devices for the data of the target object.

[0141] In another case, the user can identify hot data (or high-value data) and cold data (or low-value data) from the data of the target object, and determine that the hot data is stored in a storage medium with higher performance and the cold data is stored in a storage medium with lower performance.

[0142] Optionally, the access data table of the target object may include a combination of one or more of the following data: an identifier of the target object, an identifier of a user accessing the target object, an identifier of the area where the user is located, and an operation performed by the user on the target object.

[0143] For example, in Fig.11 The configuration interface shown includes an access data table of the target object, which includes the target object's identifier (ie, 00001), the target object's access user, the region where the target object's access user is located, and the operations performed by the target object's access user on the target object.

[0144] Specifically, there are three users accessing the target object, including user 1, user 2, and user 3. The three users are located in Beijing, Shanghai, and Shenzhen, respectively. The operations performed by user 1 on the target object include creation and modification, the operations performed by user 2 on the target object include modification and deletion, and the operations performed by user 3 on the target object include creation, modification, and deletion.

[0145] In an embodiment of the present application, after the cloud computing platform obtains the task information of the task to be processed and the processing capacity information of the business container group deployed in the cloud computing platform, it can allocate the task to be processed to the target computing device for processing based on the task information and the processing capacity information. Since the target computing device is an intelligent hierarchical cluster deployed in the business container group or the cloud computing platform, the data of the target object can be processed by the business pod side or the intelligent hierarchical cluster, which can avoid only one cluster or system processing a large amount of data uniformly, thereby improving the effectiveness of data processing. And the intelligent hierarchical cluster can generate an access data table for the target object based on the processing results of the task to be processed. Since the access data table is used to manage the storage area of ​​the target object, the user can apply for more storage resources for the data of the target object based on the access data table, or identify hot data based on the access data table and apply to store the hot data in a storage medium with higher performance, which can provide customized service needs for users.

[0146] In combination with the description of the above embodiment, the cloud computing platform is also deployed with a data collection system. The data collection system can actively collect data of the target object from the business container group, thereby realizing the intelligent hierarchical processing process of the data of the target object. Based on this, the task processing method provided in the embodiment of the present application also includes steps A to C.

[0147] Step A: The cloud computing platform obtains the collection performance information of the data collection system.

[0148] It should be understood that the acquisition performance information is used to ensure the acquisition capability of the data acquisition system.

[0149] Optionally, the acquisition performance information of the data acquisition system includes a combination of one or more of the following information: network bandwidth occupied by the data acquisition system, CPU resources occupied by the data acquisition system, and memory resources occupied by the data acquisition system.

[0150] Step B: The cloud computing platform instructs the data collection system to collect data of the target object from the business container group based on the collection performance information of the data collection system.

[0151] Optionally, when multiple business container groups are deployed in the cloud computing platform, the cloud computing platform may collect data of target objects from the multiple business container groups in batches based on collection performance information of the data collection system.

[0152] For example, Fig.12 As shown, a scheduler, a data acquisition system and four business container groups (including business container group 1, business container group 2, business container group 3 and business container group 4) are deployed in the cloud computing platform, and the business data of the target object includes seven data (namely data 1, data 2, data 3, data 4, data 5, data 6 and data 7).

[0153] Based on the collection capability of the data collection system, the scheduler instructs the data collection system to collect the 7 data from the 4 business container groups in 3 batches.

[0154] Specifically, the data collection system collects data 1, data 2, and data 3 from business container group 1, collects data 4 and data 5 from business container group 2, and collects data 6 and data 7 from business container group 3 and business container group 4, respectively. And data 1, data 2, and data 3 are the first batch of data collected by the data collection system, data 4, data 5, and data 6 are the second batch of data collected by the data collection system, and data 7 is the third batch of data collected by the data collection system.

[0155] Step C: The data acquisition system sends the data of the target object to the intelligent grading system.

[0156] It is understandable that after acquiring the data of the target object sent by the data acquisition system, the intelligent grading system can obtain the processing result of the task to be processed based on the data of the target object.

[0157] In the embodiment of the present application, the cloud computing platform can instruct the data collection system to actively collect the data of the target object from the business container group based on the collection performance information of the data collection system, and can effectively control the data collection order to ensure that the data collection system does not exceed its own collection capacity. In addition, the data collection system can send the data of the target object to the intelligent grading system, so that the intelligent grading system can obtain the processing results of the task to be processed based on the data of the target object, which can ensure that the data collected by the data collection system will not exceed the maximum processing capacity of the intelligent grading cluster, reducing the processing pressure of the intelligent grading cluster.

[0158] In an optional implementation, the data volume of the above-mentioned tasks to be processed may be large, so the business container group alone may not be able to timely and effectively process all the contents of the tasks to be processed. In this case, the cloud computing platform can allocate part of the contents of the tasks to be processed to the business container group for processing, and allocate other contents to the intelligent hierarchical cluster for processing. Based on this, the task processing method provided in the embodiment of the present application can also include steps D-E.

[0159] Step D: The cloud computing platform splits the task to be processed to obtain multiple subtasks to be processed.

[0160] It should be understood that the multiple subtasks to be processed are subtasks included in the task to be processed.

[0161] In an optional implementation, the task information of the task to be processed may also include the data volume of the task to be processed. The cloud computing platform may split the task to be processed based on the data volume of the task to be processed, that is, the task information of each of the multiple subtasks may be obtained.

[0162] In one case, the task information of the task to be processed may also include the data volume of the task to be processed. The cloud computing platform may evenly split the data volume of the task to be processed, that is, the data volume of each of the multiple subtasks to be processed is the same.

[0163] In another case, the cloud computing platform may also perform non-uniform splitting of the data volume of the task to be processed, that is, the data volume of each subtask to be processed is different.

[0164] Step E: Based on the processing capability information of the business container group and the task information of the multiple subtasks to be processed, the cloud computing platform allocates at least one subtask to the business container group for processing, and allocates other subtasks to the intelligent hierarchical cluster for processing.

[0165] It should be understood that the other subtasks are subtasks other than the at least one subtask among the multiple subtasks to be processed.

[0166] Specifically, for any of the multiple subtasks, the cloud computing platform can determine whether the processing capability information of the business container group matches the task information of the subtask successfully. If the match is successful, the cloud computing platform can assign the subtask to the business container group for processing. If the match fails, the cloud computing platform can assign the subtask to the intelligent hierarchical cluster for processing.

[0167] Optionally, the cloud computing platform can obtain the data volume of the multiple subtasks to be processed after splitting the data volume of the task to be processed. The cloud computing platform can then query the task data relationship based on the data volume of the multiple subtasks to be processed, and determine the data volume of the multiple subtasks to be processed in the task data relationship. Finally, the cloud computing platform determines the task information corresponding to the data volume of the multiple subtasks to be processed in the task data relationship as the task information of the multiple subtasks to be processed. The task data relationship includes multiple data volumes and task information corresponding to each of the multiple data volumes.

[0168] In the embodiment of the present application, the cloud computing platform can instruct the business container group and the intelligent hierarchical cluster to collaboratively complete the processing process of the task to be processed, specifically, the business container group processes at least one subtask, and the intelligent hierarchical cluster processes other subtasks. It can fully utilize the processing capabilities of the intelligent hierarchical cluster and the business pod side, greatly saving the processing cost of the cloud computing platform, avoiding the processing of all data by a single cluster (or system), and improving the efficiency of data processing.

[0169] For example, Fig.13 As shown, a scheduler, an intelligent hierarchical cluster and three business container groups (including business container group 1, business container group 2 and business container group 3) are deployed in the cloud computing platform. The tasks to be processed include seven subtasks, namely Task 1, Task 2, Task 3, Task 4, Task 5, Task 6 and Task 7.

[0170] Based on the processing capacity information of the three business container groups and the task information of the seven subtasks, the scheduler assigns Task 1, Task 2, and Task 3 to the intelligent hierarchical cluster for processing, and assigns Task 4 and Task 5 to business container group 1 for processing, and assigns Task 6 and Task 7 to business container group 2 and business container group 3 for processing, respectively.

[0171] It is understandable that in order to implement the functions in the above embodiments, the cloud computing platform includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0172] Combined with the above Figures 1 to 13 , describes in detail the task processing method provided by this embodiment, and will now be combined with Fig.14 , describes the task processing device provided by this embodiment. Fig.14 As shown, an embodiment of the present application provides a task processing device 30 running on a cloud computing platform, the cloud computing platform is deployed with a container management system, the container management system is deployed with an intelligent hierarchical cluster and one or more business container groups, the cloud computing platform also provides a configuration interface, the task processing device includes: an acquisition module 301 and a first processing module 302, the intelligent hierarchical cluster includes a second processing module 303.

[0173] The acquisition module 301 is used to acquire the task information of the task to be processed and the processing capacity information of the business container group according to the task configuration operation of the user in the configuration interface, wherein the task configuration operation indicates to perform intelligent hierarchical processing on the data of the target object.

[0174] A first processing module 302 is used to allocate the task to be processed to a target computing device for processing based on the task information of the task to be processed and the processing capability information of the business container group, where the target computing device is the business container group or the intelligent hierarchical cluster;

[0175] The second processing module 303 is used to generate an access data table of the target object based on the processing result of the task to be processed, and the access data table is used to manage the storage area of ​​the target object.

[0176] Optionally, the first processing module 302 is further configured to match the processing capability information of the service container group with the task information of the task to be processed.

[0177] The first processing module 302 is further configured to determine whether the processing capability information of the service container group matches the task information of the task to be processed successfully, and to allocate the task to be processed to the service container group for processing.

[0178] The first processing module 302 is further configured to determine that the processing capability information of the business container group fails to match the task information of the task to be processed, and allocate the task to be processed to the intelligent hierarchical cluster for processing.

[0179] Optionally, the first processing module 302 is further configured to split the task to be processed into a plurality of subtasks to be processed.

[0180] The first processing module 302 is further configured to allocate at least one subtask to the business container group for processing and allocate other subtasks to the intelligent hierarchical cluster for processing based on the processing capability information of the business container group and the task information of the multiple subtasks to be processed.

[0181] Optionally, the cloud computing platform is also deployed with a data acquisition system, the task processing device 30 further includes an indication module 304 , and the data acquisition system includes a sending module 305 .

[0182] The acquisition module 301 is also used to acquire acquisition performance information of the data acquisition system.

[0183] The instruction module 304 is used to instruct the data collection system to collect the data of the target object from the business container group based on the collection performance information of the data collection system.

[0184] The sending module 305 is used to send the data of the target object to the intelligent grading cluster.

[0185] Optionally, the service container group includes a third processing module 306 .

[0186] The third processing module 306 is used to process the task to be processed and obtain a processing result of the task to be processed.

[0187] Optionally, the data acquisition system further includes an acquisition module 307 .

[0188] The third processing module 306 is used to process the task to be processed and obtain an intermediate result of the task to be processed.

[0189] The collection module 307 is used to collect the intermediate result of the task to be processed from the business container group.

[0190] The sending module 305 is used to send the intermediate result of the task to be processed to the intelligent grading cluster.

[0191] The second processing module 303 is used to obtain the processing result of the task to be processed according to the intermediate result of the task to be processed.

[0192] Optionally, the target computing device includes an operator container, and the operator container is deployed in the target computing device in the form of a container image.

[0193] The first processing module 302 is specifically used to assign the task to be processed to the operator container for processing, and obtain the processing result of the task to be processed.

[0194] Optionally, the task information of the task to be processed includes the data volume of the task to be processed.

[0195] The first processing module 302 is further configured to perform a capacity expansion operation or a capacity reduction operation on the operator container included in the target computing device based on the data volume of the task to be processed.

[0196] Among them, the acquisition module 301, the processing module (including the first processing module 302, the second processing module 303 and the third processing module 306), the indication module 304, the sending module 305 and the acquisition module 307 can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the first processing module 302 is introduced below by taking the first processing module 302 as an example. Similarly, the implementation of the acquisition module 301, the second processing module 303, the indication module 304, the sending module 305, the third processing module 306 and the acquisition module 307 can refer to the implementation of the determination module.

[0197] As an example of a software functional unit, the first processing module 302 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the first processing module 302 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0198] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0199] As an example of a hardware functional unit, the first processing module 302 may include at least one computing device, such as a server, etc. Alternatively, the first processing module 302 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0200] The multiple computing devices included in the first processing module 302 can be distributed in the same region or in different regions. The multiple computing devices included in the processing module 302 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first processing module 302 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0201] It should be noted that, in other embodiments, the first processing module 302 can be used to execute any step in the task processing method, the acquisition module 301 can be used to execute any step in the task processing method, the second processing module 303 can be used to execute any step in the task processing method, the indication module 304 can be used to execute any step in the task processing method, the sending module 305 can be used to execute any step in the task processing method, the third processing module 306 can be used to execute any step in the task processing method, and the acquisition module 307 can also be used to execute any step in the task processing method. The steps that the acquisition module 301, the processing module (including the first processing module 302, the second processing module 303 and the third processing module 306), the indication module 304, the sending module 305 and the acquisition module 307 are responsible for implementing can be specified as needed, and the acquisition module 301, the processing module (including the first processing module 302, the second processing module 303 and the third processing module 306), the indication module 304, the sending module 305 and the acquisition module 307 respectively implement different steps in the task processing method to realize the full functions of the task processing device.

[0202] The present application also provides a computing device 400. Fig.15As shown, computing device 400 includes: bus 402, processor 404, memory 406 and communication interface 408. Processor 404, memory 406 and communication interface 408 communicate through bus 402. Computing device 400 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 400.

[0203] The bus 402 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 The bus 402 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 402 may include a path for transmitting information between various components of the computing device 400 (eg, the memory 406, the processor 404, and the communication interface 408).

[0204] The processor 404 may include any one or more processors such as a CPU, a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0205] The memory 406 may include a volatile memory, such as a random access memory (RAM). The processor 404 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0206] The memory 406 stores executable program codes, and the processor 404 executes the executable program codes to respectively implement the functions of the acquisition module, the processing module, the indication module, the sending module, and the acquisition module, thereby implementing the task processing method. That is, the memory 406 stores instructions for executing the task processing method.

[0207] The communication interface 408 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 400 and other devices or a communication network.

[0208] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0209] like Fig.16 As shown, the computing device cluster includes at least one computing device 400. The memory 406 in one or more computing devices 400 in the computing device cluster may store the same instructions for executing the task processing method.

[0210] In some possible implementations, the memory 406 of one or more computing devices 400 in the computing device cluster may also store partial instructions for executing the task processing method. In other words, the combination of one or more computing devices 400 may jointly execute instructions for executing the task processing method.

[0211] It should be noted that the memory 406 in different computing devices 400 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the task processing device. That is, the instructions stored in the memory 406 in different computing devices 400 can implement the functions of one or more modules among the acquisition module, the processing module, the indication module, the sending module and the collection module.

[0212] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.17 A possible implementation is shown. Fig.17 As shown, two computing devices 400A and 400B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 406 in the computing device 400A stores instructions for executing the functions of the indication module, the sending module, and the acquisition module. At the same time, the memory 406 in the computing device 400B stores instructions for executing the functions of the acquisition module and the processing module.

[0213] Fig.17 The connection mode between the computing device clusters shown may be considered to be a process of obtaining a large amount of information and assigning the tasks to be processed to the target computing device for processing. Therefore, it is considered that the functions implemented by the acquisition module and the processing module are handed over to the computing device 400B for execution.

[0214] It should be understood that Fig.17The functions of the computing device 400A shown in FIG. 4 may also be completed by multiple computing devices 400. Similarly, the functions of the computing device 400B may also be completed by multiple computing devices 400.

[0215] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.16 and Fig.17 The connection mode of the computing device cluster is different in that the memory 406 in one or more computing devices 400 in the computing device cluster may store the same instructions for executing the task processing method.

[0216] In some possible implementations, the memory 406 of one or more computing devices 400 in the computing device cluster may also store partial instructions for executing the task processing method. In other words, the combination of one or more computing devices 400 may jointly execute instructions for executing the task processing method.

[0217] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the task processing method.

[0218] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the task processing method.

[0219] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A task processing method running on a cloud computing platform, characterized in that: The cloud computing platform is deployed with a container management system, in which an intelligent hierarchical cluster and one or more business container groups are deployed, and the cloud computing platform also provides a configuration interface. The method includes: The cloud computing platform obtains task information of the task to be processed and processing capacity information of the business container group according to the task configuration operation of the user in the configuration interface, wherein the task configuration operation indicates intelligent hierarchical processing of data of the target object; The cloud computing platform allocates the task to be processed to a target computing device for processing based on the task information of the task to be processed and the processing capability information of the business container group, wherein the target computing device is the business container group or the intelligent hierarchical cluster; The intelligent hierarchical cluster generates an access data table of the target object based on the processing result of the task to be processed, and the access data table is used to manage the storage area of ​​the target object.

2. The method according to claim 1, characterized in that The cloud computing platform allocates the task to be processed to a target computing device for processing based on the task information of the task to be processed and the processing capability information of the business container group, including: The cloud computing platform matches the processing capability information of the business container group with the task information of the task to be processed; The cloud computing platform determines that the processing capability information of the business container group successfully matches the task information of the task to be processed, and allocates the task to be processed to the business container group for processing; The cloud computing platform determines that the processing capability information of the business container group fails to match the task information of the task to be processed, and allocates the task to be processed to the intelligent hierarchical cluster for processing.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The cloud computing platform splits the task to be processed to obtain a plurality of subtasks to be processed; The cloud computing platform allocates at least one subtask to the business container group for processing based on the processing capability information of the business container group and the task information of the multiple subtasks to be processed, and allocates other subtasks to the intelligent hierarchical cluster for processing.

4. The method according to any one of claims 1 to 3, characterized in that: The cloud computing platform is also deployed with a data acquisition system, and the method further includes: The cloud computing platform obtains the collection performance information of the data collection system; The cloud computing platform instructs the data acquisition system to acquire the data of the target object from the business container group based on the acquisition performance information of the data acquisition system; The data acquisition system sends the data of the target object to the intelligent grading cluster.

5. The method according to any one of claims 2 to 4, characterized in that: After the cloud computing platform assigns the to-be-processed tasks to the business container group for processing, the method includes: The business container group processes the task to be processed to obtain a processing result of the task to be processed.

6. The method according to any one of claims 2 to 4, characterized in that: After the cloud computing platform assigns the to-be-processed tasks to the business container group for processing, the method includes: The business container group processes the task to be processed to obtain an intermediate result of the task to be processed; The data collection system collects the intermediate results of the tasks to be processed from the business container group, and sends the intermediate results of the tasks to be processed to the intelligent grading cluster; The intelligent grading cluster obtains a processing result of the task to be processed according to the intermediate result of the task to be processed.

7. The method according to any one of claims 1 to 6, characterized in that: The target computing device includes an operator container, and the operator container is deployed in the target computing device in the form of a container image. The cloud computing platform allocates the task to be processed to the target computing device for processing, including: The cloud computing platform distributes the tasks to be processed to the operator container for processing, and obtains processing results of the tasks to be processed.

8. The method according to any one of claims 1 to 7, characterized in that: The task information of the task to be processed includes the data volume of the task to be processed, and the method further includes: The cloud computing platform performs an expansion operation or a reduction operation on the operator container included in the target computing device based on the data volume of the task to be processed.

9. A task processing device running on a cloud computing platform, characterized in that: The cloud computing platform is deployed with a container management system, in which an intelligent hierarchical cluster and one or more business container groups are deployed, the cloud computing platform also provides a configuration interface, the device includes an acquisition module and a first processing module, and the intelligent hierarchical cluster includes a second processing module; The acquisition module is used to acquire task information of the task to be processed and processing capacity information of the business container group according to the task configuration operation of the user in the configuration interface, wherein the task configuration operation indicates intelligent hierarchical processing of the data of the target object; The first processing module is used to allocate the task to be processed to a target computing device for processing based on the task information of the task to be processed and the processing capability information of the business container group, wherein the target computing device is the business container group or the intelligent hierarchical cluster; The second processing module is used to generate an access data table of the target object based on the processing result of the task to be processed, and the access data table is used to manage the storage area of ​​the target object.

10. The device according to claim 9, characterized in that The first processing module is further used to match the processing capability information of the business container group with the task information of the task to be processed; The first processing module is further configured to determine whether the processing capability information of the business container group successfully matches the task information of the task to be processed, and to allocate the task to be processed to the business container group for processing; The first processing module is further configured to determine that the processing capability information of the business container group fails to match the task information of the task to be processed, and allocate the task to be processed to the intelligent hierarchical cluster for processing.

11. The device according to claim 9 or 10, characterized in that The first processing module is further used to split the task to be processed to obtain multiple subtasks to be processed; The first processing module is further configured to allocate at least one subtask to the business container group for processing and allocate other subtasks to the intelligent hierarchical cluster for processing based on the processing capability information of the business container group and the task information of the multiple subtasks to be processed.

12. The device according to any one of claims 9 to 11, characterized in that The cloud computing platform is also deployed with a data acquisition system, the device also includes an indication module, and the data acquisition system includes a sending module; The acquisition module is further used to acquire acquisition performance information of the data acquisition system; The instruction module is used to instruct the data acquisition system to collect the data of the target object from the business container group based on the acquisition performance information of the data acquisition system; The sending module is used to send the data of the target object to the intelligent grading cluster.

13. The device according to any one of claims 10 to 12, characterized in that The service container group includes a third processing module; The third processing module is used to process the task to be processed to obtain a processing result of the task to be processed.

14. The device according to any one of claims 10 to 12, characterized in that The data acquisition system includes an acquisition module and a sending module, and the business container group includes a third processing module; The third processing module is used to process the task to be processed to obtain an intermediate result of the task to be processed; The collection module is used to collect the intermediate results of the tasks to be processed from the business container group; The sending module is used to send the intermediate result of the task to be processed to the intelligent grading cluster; The second processing module is used to obtain the processing result of the task to be processed according to the intermediate result of the task to be processed.

15. The device according to any one of claims 9 to 14, characterized in that The target computing device includes an operator container, and the operator container is deployed in the target computing device in the form of a container image; The first processing module is specifically used to assign the task to be processed to the operator container for processing, and obtain the processing result of the task to be processed.

16. The device according to any one of claims 9 to 15, characterized in that The task information of the task to be processed includes the data volume of the task to be processed; The first processing module is further used to perform an expansion operation or a reduction operation on the operator container included in the target computing device based on the data volume of the task to be processed.

17. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the task processing method according to any one of claims 1 to 8.

18. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the task processing method according to any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the task processing method according to any one of claims 1 to 8.

Images