Data processing method, device, storage medium and computer terminal

Abstract

The present invention discloses a data processing method, apparatus, storage medium, and computer terminal. Among them, the method includes: in response to a data transmission instruction received by a target platform, obtaining a target identifier and target data; determining a target cluster from at least one cluster in the target platform based on the target identifier, where the target platform is used to provide at least one cluster, each cluster corresponds to a different business line respectively, and the target cluster is used to manage at least one resource pool; determining a target resource pool from the target cluster based on the target identifier, where the target resource pool includes at least one server; determining a target server from the target resource pool based on the target data, and transmitting the target data to the target server, where the target server is used to run a target service using the target data. The present invention solves the technical problem in the related art that the utilization rate of the server is relatively low, resulting in a relatively high resource occupancy cost for service operation.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular, to a data processing method, apparatus, storage medium, and computer terminal. Background Art

[0002] At present, improving the number of services deployed on a single server through co-location, thereby improving the utilization rate of machine resources and reducing the energy consumption of the data center, is one of the urgent problems to be solved in the Internet field. At present, the main method for co-location management during the same period is to directly co-locate services based on containers. However, since containers only provide resource isolation capabilities, whether the isolation takes effect depends on the switch configuration at the time of container startup, and it can only limit the container itself and cannot limit other containers; moreover, the total number of containers that can be started on a single server and the resources requested by the containers all rely on manual management, resulting in a low utilization rate of the server.

[0003] In response to the above problems, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of the present invention provide a data processing method, apparatus, storage medium, and computer terminal to at least solve the technical problem in the related art that the utilization rate of the server is low, resulting in a high resource occupancy cost for service operation.

[0005] According to one aspect of the embodiments of the present invention, a data processing method is provided, including: in response to a received data transmission instruction, obtaining a target identifier and target data, where the target identifier is used to represent the business line and attributes of the target service, and the target data is used to represent the resource attributes required by the target service; determining a target cluster in a target platform based on the target identifier, where the target platform is used to provide at least one cluster, each cluster corresponding to a different business line, and the target cluster is used to manage at least one resource pool; determining a target resource pool from the target cluster based on the target identifier, where the target resource pool includes at least one server; determining a target server from the target resource pool based on the target data, and transmitting the target data to the target server, where the target server is used to run the target service using the target data.

[0006] Optionally, determining a target server from the target resource pool based on the target data includes: performing tagging processing on the target data to generate a first tag; obtaining a second tag corresponding to each server in the target resource pool; and determining the target server from the target resource pool based on the first tag and the second tag.

[0007] Optionally, determining a target server from a target resource pool based on a first tag and a second tag includes: comparing the first tag with the second tag corresponding to each server to obtain a comparison result, where the comparison result is used to describe the similarity between the first tag and the second tag; and determining a target server from the target resource pool based on the comparison result, where the similarity between the second tag of the target server and the first tag is greater than a preset similarity.

[0008] Optionally, determining a target server from a target resource pool based on target data includes: obtaining the current remaining resources of the target resource pool and the resource occupancy of the target data; updating the target resource pool based on the current remaining resources and the resource occupancy to obtain an updated target resource pool; and determining a target server from the updated target resource pool based on the target data.

[0009] Optionally, updating the target resource pool based on the current remaining resources and the resource occupancy to obtain an updated target resource pool includes: comparing the current remaining resources and the resource occupancy to generate a comparison result, where the comparison result is used to indicate whether the resource occupancy is greater than the current remaining resources; and in response to the resource occupancy being greater than the current remaining resources, using the target platform to transfer at least one server from other resource pools to the target resource pool through the target cluster to obtain an updated target resource pool, where the other resource pools are other resource pools except the target resource pool among the at least one resource pools.

[0010] Optionally, after determining a target server from a target resource pool based on target data and transmitting the target data to the target server, the method further includes: in response to receiving a transmission success instruction, updating the current remaining resources of the target resource pool based on the target data through the target cluster.

[0011] Optionally, the method further includes: in response to receiving a transmission success instruction, determining a server to be removed from at least one server based on the current remaining resources of the target resource pool and the current resource occupancy of the target resource pool; and transferring the server to be removed to other resource pools.

[0012] According to another aspect of the embodiments of the present invention, there is also provided a data processing device, including: an acquisition module, configured to acquire a target identifier and target data in response to a received data transmission instruction, where the target identifier is used to represent the business line to which the target service belongs and the attributes of the target service, and the target data is used to represent the resource attributes required by the target service; a first determination module, configured to determine a target cluster in a target platform based on the target identifier, where the target platform is used to provide at least one cluster, and each cluster corresponds to a different business line respectively, and the target cluster is used to manage at least one resource pool; a second determination module, configured to determine a target resource pool from the target cluster based on the target identifier, where the target resource pool includes at least one server; a transmission module, configured to determine a target server from the target resource pool based on the target data, and transmit the target data to the target server, where the target server is used to run the target service by using the target data.

[0013] According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, where the computer-readable storage medium includes a stored program, and when the program runs, it controls the device where the computer-readable storage medium is located to execute the above data processing method.

[0014] According to another aspect of the embodiments of the present invention, there is also provided a computer terminal, including: a processor and a memory, where the processor is used to run a program, and when the program runs, it executes the above data processing method.

[0015] Through the above steps, first, in response to a received data transmission instruction, a target identifier and target data are acquired, where the target identifier is used to represent the business line to which the target service belongs and the attributes of the target service, and the target data is used to represent the resource attributes required by the target service. Based on the target identifier, a target cluster is determined in the target platform, where the target platform is used to provide at least one cluster, and each cluster corresponds to a different business line respectively, and the target cluster is used to manage at least one resource pool. Based on the target identifier, a target resource pool is determined from the target cluster, where the target resource pool includes at least one server; finally, based on the target data, a target server is determined from the target resource pool, and the target data is transmitted to the target server, where the target server is used to run the target service by using the target data, realizing reasonable scheduling of the target server in combination with business requirements, being able to effectively utilize server resources, avoiding causing, and thus solving the technical problem in the related art that the utilization rate of the server is relatively low, resulting in a relatively high resource occupancy cost for service operation. Description of the Drawings

[0016] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and the illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0017] Figure 1 is a flowchart of a data processing method according to an embodiment of the present invention;

[0018] Figure 2 is a schematic diagram of resource pooling management according to an embodiment of the present invention;

[0019] Figure 3 is a structural block diagram of service resource management according to an embodiment of the present invention;

[0020] Figure 4 is a flowchart of another data processing method according to an embodiment of the present invention;

[0021] Figure 5 is a schematic diagram of a data processing device according to an embodiment of the present invention. Detailed implementation manners

[0022] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0024] First, some nouns or terms that appear in the process of describing the embodiments of the present application are applicable to the following explanations:

[0025] Program: Can run on a computer and has specific functions;

[0026] Service: Includes one or more programs, which can be accessed by users inside or outside the company and provides one or several functions to users;

[0027] Server: The computer device on which the program runs, which can provide hardware resources such as CPU, memory, disk, network, GPU, etc. for the program to use;

[0028] Online service: A service used to provide real-time interaction to users, which needs to quickly respond to users' operation requests and return results (such as when a user scans a code to pay, it is necessary to immediately return whether the user has paid successfully and the deducted amount);

[0029] Offline service: A service that provides non-real-time interaction, and only needs to return results within a relatively long time range (such as the daily income of current account financial management and the deduction service of automatic renewal, as long as it is completed on the same day);

[0030] Mysql service: An open-source database service that provides functions for regular storage and query of user data (such as storing the user's account balance and modifying the account balance when the user pays);

[0031] Redis service: An open-source in-memory caching service that provides functions for high-speed storage and query of user data (such as activities like product seckill, red envelope snatching, and coupon snatching);

[0032] Kafka service: An open-source message queue service that provides functions for asynchronous large-volume writing and distribution of user messages (such as log collection and multiple parties reading the same data simultaneously);

[0033] Load: The hardware resources such as CPU, memory, and disk used when the service runs, which will change with the traffic;

[0034] Container: Based on the capabilities of the Linux kernel, it virtualizes a small-scale running environment. By packaging the image and starting the container, a service can be created, and the resources used by the service within the container can be limited and isolated (including CPU, memory, I / O, and network, etc.). The current mainstream container project in the industry is docker.

[0035] k8s: Full name is kubernetes, which is an orchestration and management tool for portable containers born for container services, providing solutions for container management such as service deployment, service monitoring, application scaling, and fault handling.

[0036] Improving the number of services deployed on a single server through co-location, thereby enhancing the utilization rate of machine resources and reducing the energy consumption of the data center, is one of the urgent problems to be solved in the Internet field. The basis of co-location is resource isolation. Containers are the most widely used resource isolation technology at present, and Docker is the most popular container product in the industry. Through the resource isolation ability of containers, the resources on a single server can be flexibly divided into multiple containers. Inside each container, only the pre-allocated amount of resources can be seen and used; the service load inside a container will be restricted by the container within the preset resource range, and no matter how the load changes, it will not affect the normal operation of services in other containers. However, as the scale of co-located containers increases, the difficulty of container management also increases exponentially. For example, in a server cluster of 10,000 units, the corresponding deployed container scale can reach between 100,000 and 1 million. Container technology can solve the problem of resource isolation, but it cannot solve the problems of management efficiency and reliability of co-located containers. The Kubernetes (K8S) technology was born to solve this problem and has become the most popular open-source container orchestration product in the co-location field and is widely used in the industry. In summary, the technology stack combination of Docker + K8S has formed the de facto standard for container orchestration in the current co-location field.

[0037] On the other hand, due to the particularity of financial business in terms of legal compliance, for different business directions such as payment, insurance, credit, funds, wealth management, and fintech, it is necessary to isolate the service programs and data of different businesses. Programs between different businesses cannot be co-located, and data cannot be directly interconnected; in addition, the financial industry's requirements for the security level of the data center and the reliability requirements for services and data are higher than the average level of the Internet industry.

[0038] The current container co-location management methods are mainly as follows:

[0039] (1) Co-locating similar services directly based on containers:

[0040] It is mainly based on the standard container solution provided by Docker. Its own services are directly deployed and run through containers. By increasing the number of service containers running on a single machine, the average utilization rate of machine resources is improved. Since containers only provide resource isolation capabilities, whether the isolation takes effect depends on the switch configuration at container startup, and it can only limit the container itself and cannot limit other containers; moreover, the total number of containers that can be started on a single server and the total resources requested by the containers all rely on human management. Therefore, the solution of directly co-locating containers determines that a single machine can only be used for a single service, such as online, MySQL, etc. That is, there can only be one type of service on a single server, such as only the MySQL service or only the online service.

[0041] However, the existing problems are that similar services have the same resource demand preferences. Co-location will lead to bottlenecks in hot resources, and it is difficult to effectively improve the co-location density. For example, MySQL has a large demand for disks. After multiple MySQL services are co-located, the disks will become the bottleneck, while the CPU and memory are relatively idle, resulting in a waste of resources. Second, for services without obvious hot resources, simply relying on containers to increase the co-location density will lead to a decline in management efficiency, and a container management platform or tool needs to be developed accordingly.

[0042] (2) Co-location of similar services based on containers + k8s:

[0043] Based on container technology, this method introduces k8s technology to manage the servers and containers used in a cluster manner. Through the interfaces provided by k8s, the addition and deletion of servers, and the startup and shutdown of containers can be achieved. The number of resources required by a single container and the total amount of resources that a single machine can provide are both managed and maintained by k8s. When a container starts up, k8s automatically selects machines and mounts containers according to the resources required by the container and the real-time available resources of all machines in the current cluster, improving resource utilization and container management efficiency while also providing better service stability.

[0044] After introducing containers + k8s, the container management efficiency problem is alleviated. However, due to the strictest compliance requirements in the financial industry, the servers of different business lines must be physically isolated and deployed, resulting in the need to further split similar services by business line, facing the following two problems: First, the same as in (1), the bottleneck of hot resources within similar services makes it difficult to effectively improve the co-location density. Second, due to compliance requirements, similar services need to be further isolated and split according to the business internally, resulting in the k8s cluster needing to be split into more and smaller sub-clusters, greatly increasing the complexity of cross-cluster management.

[0045] (3) Co-location of different services based on containers + k8s:

[0046] This method can be regarded as a further extension of the previous method. Utilizing the efficient container management capabilities provided by k8s, different types of services are co-located on the same server. By taking advantage of the different resource usage preferences of different services, resources are used more balancedly, further improving resource utilization.

[0047] In this method, there is complementarity in resource preferences among different services. If different services are deployed in the same set of k8s clusters, the co-location density can indeed be improved. However, the management methods of different services are different, and the personnel responsible for service management are also different. It is difficult to converge the k8s management permissions. Multiple service providers have the permission to modify k8s, and there is a risk of interference and even conflict between operations. It is very likely that due to the unexpected operation of one party, a low-priority container incorrectly requests a high-priority resource capacity, and the real high-priority container is wrongly killed due to resource competition; or a certain type of service occupies too many resources, resulting in other services being unable to normally apply for resources, and the reliability of the cluster is difficult to guarantee.

[0048] To solve the above problems, the present application provides a data processing method. From the perspective of the container co-location management solution, relying on k8s as the underlying support for container co-location, it is uniformly encapsulated on the outer layer to provide a resource orchestration and management platform, which converges the resource requirements of different service providers. Each service provider only needs to focus on the resource requirements and provide the characteristic information of the resources through a standard interface, such as: priority, quantity, server preference, business category, etc. The platform converts the resource characteristic information and distributes it to different k8s clusters for resource scheduling and service deployment. The entire conversion and scheduling deployment process is transparent to the service. While improving the resource utilization rate through the co-location of multiple services, the stability and controllability of the co-located resources are ensured through the convergence entry, the usage efficiency of the server is improved, and the resource management cost of each service provider can be reduced.

[0049] Embodiment 1

[0050] According to an embodiment of the present invention, an embodiment of a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0051] Figure 1 It is a method for data processing according to an embodiment of the present invention, as Figure 1 shown, the method includes the following steps:

[0052] Step S102, in response to the received data transmission instruction, obtain the target identifier and target data.

[0053] Among them, the target identifier is used to represent the business line to which the target service belongs and the attributes of the target service, and the target data is used to represent the resource attributes required by the target service.

[0054] The above data transmission instruction may be an instruction sent by the target service when it needs server resources, and the above data transmission instruction may carry the target identifier and target data.

[0055] The above-mentioned target identifier can be used to describe the business line to which the target service belongs and the type of the target service. It should be noted that each business line can include multiple types of services, but each business line is isolated from each other.

[0056] The above-mentioned target data can be resource attributes required by the target service, such as the resource usage, resource level, and environmental dependencies of the target service.

[0057] In an alternative embodiment, a data transmission instruction can be received, and the target identifier and target data carried by the data transmission instruction can be obtained, so as to determine a target resource pool capable of allocating resources from multiple clusters managed by the target platform, and select a server capable of running the target service from the target resource pool.

[0058] The above-mentioned target services can be online services, offline services, mysql services, kafla services, redias services, etc.

[0059] Step S104, determine a target cluster in the target platform based on the target identifier.

[0060] Among them, the target platform is used to provide at least one cluster, each cluster corresponds to a different business line respectively, and the target cluster is used to manage at least one resource pool.

[0061] At least one cluster provided by the above-mentioned target platform can be a k8s cluster. Among them, the target platform can be a hybrid resource orchestration and scheduling management platform. The above-mentioned target cluster can be a k8s cluster corresponding to the target identifier.

[0062] In an alternative embodiment, multiple different resource pools can be managed. Each type of service can apply for an exclusive resource pool, and the idle resources in the resource pool can be provided for other services to use in a hybrid manner, or it can also apply to use the hybrid resources provided by other services.

[0063] The above-mentioned target platform manages the resource pool by recording the resource usage data and idle resource transfer data of all services on different resource pools and different servers. When it is found that the available resources of a resource pool are insufficient, the target platform will apply to the asset management party for a new server to join the resource pool; similarly, when the resources of the resource pool are too idle, the target platform will return the idle machine to the asset management party with the permission of the service party to help reduce costs.

[0064] In another alternative embodiment, a target cluster corresponding to the business line of the target service can be determined in the target platform according to the target identifier, so that the target platform can orchestrate and schedule the resource pools in the target cluster to run the target service.

[0065] Such asFigure 2 The figure shows a schematic diagram of resource pooling management. Among them, each business line contains multiple resource pools, which can include Business A, Business B, and Business C. The resource pools included in each business can be the same or different. Specifically, they can be online resource pools, offline resource pools, open-source database resource pools, open-source memory cache service resource pools, open-source message queue service resource pools, activity resource pools, standby resource pools, etc.

[0066] Step S106: Determine the target resource pool from the target cluster based on the target identifier.

[0067] Among them, the target resource pool includes at least one server.

[0068] In an alternative embodiment, the target resource pool can be determined from at least one resource pool in the target cluster according to the attributes of the target service in the target identifier. Specifically, the target resource pool corresponding to the type of the target service can be determined from at least one resource pool in the target cluster according to the type of the target service, so as to run the target service through the servers in the target resource pool.

[0069] Since the target identifier records that different services have different attributes and levels, such as service priority, service resource preference, etc., and there are mutually exclusive or complementary characteristics among these attributes. For services with certain characteristics, there are also limitations on the amount of resource allocation in the resource pool. Therefore, only some resource pools can meet the resource requirements of the target service. The target cluster corresponding to the target service business line can be selected first according to the target identifier, and then the target resource pool corresponding to the attributes of the target service can be selected according to the target identifier. Ensure that the target data of the target service is scheduled in the target resource pool, and use the servers in the target resource pool to run the target service through the target data.

[0070] Step S108: Determine the target server from the target resource pool based on the target data, and transmit the target data to the target server.

[0071] Among them, the target server is used to run the target service using the target data.

[0072] The above-mentioned target server can be one or more, and the number of target servers is determined based on the resource occupancy of the target data.

[0073] In an alternative embodiment, after determining the target resource pool, it is possible to determine whether the remaining resources in the target resource pool are greater than the resource occupancy of the target data. When the remaining resources in the target resource pool are sufficient for the target data to be used, the target server can be directly determined from the target resource pool, and the target data can be transmitted to the target server for operation. When the remaining resources in the target resource pool are not sufficient for the target data to be used, some servers can be retrieved from other resource pools and placed in the target resource pool, and then the target server can be determined from the target resource pool, so that the target server can use the target data to run the target service.

[0074] In the traditional process, when a business user wants to use a certain service, they need to apply to the service management party; after receiving the application, the service management party needs to deploy the service on the selected resources; if the existing resources are insufficient, they need to apply to the asset layer for new servers to make up for the lack of resources. As Figure 3 Shown is a structural block diagram of a service resource management, which can provide standardized resource applications to interface with the resource requirements of the service, and can schedule the resource requirements to a suitable k8s cluster. Through the container orchestration and deployment capabilities provided by k8s, the scheduling and deployment of containers can be completed and delivered to the service management party for use. When there is a shortage or surplus of resources in the k8s cluster, the hybrid resource orchestration and scheduling management platform uniformly interfaces with the asset management party to apply for new resources for supplementation or return idle resources. Through the hybrid resource orchestration and scheduling management platform, unified management of hybrid resources can be achieved, and each service management party does not need to pay attention to the management details of the underlying resources, which can overall improve the reliability and efficiency of the hybrid deployment and reduce the resource cost.

[0075] The application scenario of the above content can be when the overall resource utilization rate of the server is not high, and the resource utilization rate is improved by mixing and deploying multiple different types of services on the same server. When meeting financial compliance requirements, due to strict business isolation, and each individual business still contains all types of service types, the resource scale of different services is smaller, and the management complexity of resource hybrid deployment is higher. It is necessary to use a unified hybrid resource orchestration and scheduling management platform, based on container and k8s technologies, to reduce the difficulty of resource management, improve the reliability and efficiency of the hybrid deployment, and further improve the resource utilization rate. Specifically, it can be based on the general solutions of containers and k8s. Through the platform, all service management parties are docked, and the resource requirements of the services are converged to the platform for unified docking and use in a standard interface manner, avoiding each service separately managing and maintaining the k8s cluster resources, and monitoring the usage of each cluster resource to ensure the reliability of the hybrid deployment; at the same time, the requirements of the k8s cluster for adding and deleting servers are uniformly docked with the asset management party through the platform; for the hybrid deployment between different services, the resource characteristic data registered by each service is standardized through the interface, and the platform decides to send the service to a suitable k8s cluster.

[0076] Through the above steps, first, in response to the received data transmission instruction, the target identifier and target data are obtained. The target identifier is used to represent the business line to which the target service belongs and the attributes of the target service, and the target data is used to represent the resource attributes required by the target service. Based on the target identifier, the target cluster is determined in the target platform. The target platform is used to provide at least one cluster, and each cluster corresponds to a different business line. The target cluster is used to manage at least one resource pool. Based on the target identifier, the target resource pool is determined from the target cluster. The target resource pool includes at least one server. Finally, based on the target data, the target server is determined from the target resource pool, and the target data is transmitted to the target server. The target server is used to run the target service using the target data, realizing the reasonable scheduling of the target server in combination with business requirements, effectively utilizing the server resources, avoiding causing, and thus solving the technical problem that the utilization rate of the server in the related technology is relatively low, resulting in a relatively high resource occupancy cost for service operation.

[0077] Optionally, determining the target server from the target resource pool based on the target data includes: performing tagging processing on the target data to generate a first tag; obtaining a second tag corresponding to each server in the target resource pool; and determining the target server from the target resource pool based on the first tag and the second tag.

[0078] The above target data can be metadata. Among them, the metadata can be of the resource usage type, resource level type, environment dependency type, etc. The resources applied through the entry can be uniformly managed at the metadata level.

[0079] In an alternative embodiment, when any service needs resources, it applies through a unified resource application entry. The data specification for resource application can be defined, the common features of the resources of different services can be extracted, and the personalized feature requirements of the services can be taken into account.

[0080] The above first tag and second tag can be container tags.

[0081] In an alternative embodiment, tagging processing can be performed on the target data to generate a first tag, so that the target data can be scheduled and used on the target server in the form of a container according to the first tag. It should be noted that when the target platform manages the servers in the resource pool, a second tag can be attached to each server according to the remaining resources of the server, so that the target server with sufficient resources can be screened through the first tag.

[0082] Optionally, determining a target tag from a target resource pool based on a first tag and a second tag includes: comparing the first tag with the second tag corresponding to each resource pool to determine a comparison result, where the comparison result is used to describe the similarity between the first tag and the second tag; and determining a target server from the target resource pool based on the comparison result, where the similarity between the second tag of the target server and the first tag is greater than a preset similarity.

[0083] In an alternative embodiment, it is possible to determine whether target data can run on a server based on the similarity between the first tag and the second tag. If the similarity between the first tag and the second tag is greater than a preset similarity, it indicates that the resource occupancy of the target data matches the remaining resources of the server. At this time, the server can be determined as the target server to identify a server with better performance to run the target service through the target data.

[0084] Optionally, determining a target server from a target resource pool based on target data includes: obtaining the current remaining resources of the target resource pool and the resource occupancy of the target data; updating the target resource pool based on the current remaining resources and the resource occupancy to obtain an updated target resource pool; and determining a target server from the updated target resource pool based on the target data.

[0085] In an alternative embodiment, when determining a target server from a target resource pool according to target data, it is also possible to obtain the current remaining resources of all servers in the target resource pool and the resource occupancy of the target data, and when the current remaining resources of all servers in the target resource pool are less than the resource occupancy of the target data, transfer other servers to the target resource pool to increase the current remaining resources of the target resource pool so that the target resource pool can provide the resources required for the target service.

[0086] In another alternative embodiment, after transferring a server to the target resource pool, the current remaining resources of the target resource pool can be updated. When the current remaining resources are greater than the resource occupancy, a target server can be selected from the updated target resource pool so as to select a server that can run the target service through the target data.

[0087] Optionally, updating the target resource pool based on the current remaining resources and the resource occupancy to obtain an updated target resource pool includes: comparing the current remaining resources and the resource occupancy to generate a comparison result, where the comparison result is used to indicate whether the resource occupancy is greater than the current remaining resources; and in response to the resource occupancy being greater than the current remaining resources, using the target platform to transfer at least one server from other resource pools to the target resource pool through the target cluster to obtain an updated target resource pool, where the other resource pools are the other resource pools in the at least one resource pool except the target resource pool.

[0088] In an alternative embodiment, the current remaining amount of resources and the amount of resources occupied can be compared to generate a comparison result. When the amount of resources occupied is greater than the current remaining amount of resources, it indicates that the resources in the resource pool are insufficient, and it is difficult for the server to run the target service with the target data. At this time, the target platform can transfer one or more servers from other resource pools to the target resource pool to increase the current remaining amount of resources in the target resource pool. Specifically, the current remaining amount of resources in other resource pools can be polled, and one or more servers can be selected from the resource pool with the largest current remaining amount of resources and transferred to the target server, which can reasonably utilize the idle resources and avoid the target service being difficult to run due to insufficient resources in the resource pool.

[0089] Specifically, a k8s cluster can manage multiple different resource pools. Each service can apply for an exclusive resource pool and provide the idle resources in the resource pool for other services to use in a mixed deployment manner; or it can also apply to use the mixed deployment resources provided by other services. The target platform manages the resource pool by recording the resource usage data and idle resource transfer data of all services on different resource pools and different servers. When it is found that the available resources in a resource pool are insufficient, the platform will apply to the asset management party for new servers to be added to the resource pool; similarly, when the resources in the resource pool are too idle, the platform will return the idle machines to the asset management party with the permission of the service party to help reduce costs.

[0090] Optionally, after determining the target server from the target resource pool based on the target data and transmitting the target data to the target server, the method further includes: in response to the received transmission success instruction, updating the current remaining amount of resources in the target resource pool based on the target data through the target cluster.

[0091] In an alternative embodiment, after transmitting the target data to the target server, a transmission success instruction can be sent to the target platform, and the target platform can update the current remaining amount of resources in the target resource pool based on the target data through the target cluster, so that subsequent resource scheduling can be performed according to the latest information when resources are called.

[0092] Optionally, the method further includes: in response to the received transmission success instruction, determining a server to be removed from at least one server based on the current remaining amount of resources in the target resource pool and the current amount of resources occupied in the target resource pool; transferring the server to be removed to other resource pools.

[0093] The above-mentioned server to be removed can be an idle server, and the above-mentioned server to be removed can also be a server transferred from other resource pools.

[0094] In an optional embodiment, after receiving a successful transmission instruction, a server to be removed can be determined from at least one server based on the current remaining resources of the target resource pool and the current resource occupancy of the target resource pool, and the server to be removed can be moved to other resource pools to improve the utilization efficiency of the server and reduce the cost of resource management.

[0095] The following combination Figure 4 A preferred embodiment of the present application is described in detail. The method can be executed by a computer terminal or a server. Figure 4 As shown, the method includes the following steps:

[0096] Step S401: Determine a target cluster according to the service attributes recorded in the target identifier.

[0097] The target platform uses a unified resource application portal to ensure that all resource operations are initiated solely through the platform, preventing unknown and illegal resource occupation. Through resource pool management, it accurately monitors resource usage information for each resource pool and server in each Kubernetes cluster. Based on this information, the platform can determine the appropriate orchestration and scheduling solution based on specific strategies whenever a new resource request arrives.

[0098] By judging the business attributes, it is possible to prevent the target service from being dispatched to different business resource pools.

[0099] Step S402: judging service attributes and resource levels according to target data, and determining a target resource pool.

[0100] Optionally, different services in a resource pool may have different attributes and levels, such as service priority and resource preference. These attributes may be mutually exclusive or complementary, and resource pools may also limit the amount of resources allocated to services with certain characteristics. Therefore, only some resource pools can satisfy new resource requests. In this step, the corresponding Kubernetes clusters and resource pools are selected based on the attributes of the newly requested resources, ensuring that newly requested resources are only scheduled to these resource pools.

[0101] Step S403: The target data is transmitted to the target server in the corresponding target resource pool in the form of a container tag.

[0102] Optionally, by screening and matching the container labels and resource pool machine labels, the container can be automatically scheduled and deployed to the target server with sufficient resources based on the amount of resources requested by the target service and the amount of available resources on the machine.

[0103] In step S404, after the platform detects that the container is successfully scheduled and deployed, it records the latest resource pool usage data and returns the container information to the service provider to complete the resource delivery.

[0104] In this solution, based on the container + k8s technology, we introduce an independent hybrid resource orchestration and scheduling management platform. By unifying and converging the operations of k8s through the platform, the cost of service resource management for all parties is reduced. Through resource pooling management, the general automation management ability of different service resource pools is achieved. A standard interface for resource application is provided to standardize and record the resource metadata information of different service applications. Through characteristic metadata such as the business attributes, service attributes, resource usage, and environmental dependencies of resources, the accurate and efficient scheduling and delivery of resources required by services are realized.

[0105] Through the above steps, the management concept of container + k8s can be combined with the unified resource management platform and integrated into the hybrid resource orchestration and scheduling architecture, effectively solving problems such as containerized hybrid management between different services, improvement of resource utilization rate, and reliability of hybrid resources. The entire method has good practical application capabilities. For issues such as the reliability and management cost of extracting different categories of service hybrids, by integrating capabilities such as containers and k8s, through container orchestration and scheduling, the unification and transparency of different services can be achieved, reducing the cost of service hybrids and hybrid resource management, and improving the reliability of the k8s cluster. On the other hand, through resource pooling management, the status of resource hybrids can be comprehensively grasped, and the resource requirements of services can be reasonably scheduled to the most suitable k8s cluster and resource pool according to different attributes such as business, service, and resource characteristics, effectively improving the efficiency and accuracy of resource scheduling.

[0106] It should be noted that in addition to k8s, other container orchestration technologies such as swarm (container cluster management system) and Mesos (resource unified management and scheduling platform) can also be selected.

[0107] Embodiment 2

[0108] According to an embodiment of the present invention, a data processing device is provided. This device can execute the motor control method in the above embodiment. The specific implementation manner and preferred application scenario are the same as those in the above embodiment and will not be elaborated here.

[0109] Figure 5 is a schematic diagram of a data processing device according to an embodiment of the present invention. As Figure 5 shown, the device includes:

[0110] An acquisition module 52, configured to acquire a target identifier and target data in response to a received data transmission instruction, where the target identifier is used to represent the business line to which the target service belongs and the attributes of the target service, and the target data is used to represent the resource attributes required by the target service;

[0111] The first determination module 54 is configured to determine a target cluster in a target platform based on a target identifier, where the target platform is used to provide at least one cluster, each cluster corresponds to a different business line respectively, and the target cluster is used to manage at least one resource pool;

[0112] The second determination module 56 is configured to determine a target resource pool from the target cluster based on the target identifier, where the target resource pool includes at least one server;

[0113] The transmission module 58 is configured to determine a target server from the target resource pool based on target data and transmit the target data to the target server, where the target server is used to run a target service using the target data.

[0114] Optionally, the transmission module is further configured to perform tagging processing on the target data to generate a first tag; obtain a second tag corresponding to each server in the target resource pool; and determine the target server from the target resource pool based on the first tag and the second tag.

[0115] Optionally, the transmission module is further configured to compare the first tag with the second tag corresponding to each server to determine a comparison result, where the comparison result is used to describe the similarity between the first tag and the second tag; and determine the target server from the target resource pool based on the comparison result, where the similarity between the second tag of the target server and the first tag is greater than a preset similarity.

[0116] Optionally, the transmission module is further configured to obtain the current remaining resources of the target resource pool and the resource occupancy of the target data; update the target resource pool based on the current remaining resources and the resource occupancy to obtain an updated target resource pool; and determine the target server from the updated target resource pool based on the target data.

[0117] Optionally, the transmission module is further configured to compare the current remaining resources and the resource occupancy to generate a comparison result, where the comparison result is used to indicate whether the resource occupancy is greater than the current remaining resources; and in response to the resource occupancy being greater than the current remaining resources, use the target platform to transfer at least one server from other resource pools to the target resource pool through the target cluster to obtain an updated target resource pool, where the other resource pools are the other resource pools except the target resource pool among the at least one resource pools.

[0118] Optionally, the apparatus is further configured to, in response to a received transmission success instruction, determine a server to be removed from at least one server based on the current remaining resources of the target resource pool and the current resource occupancy of the target resource pool; and transfer the server to be removed to other resource pools.

[0119] Embodiment 3

[0120] According to an embodiment of the present invention, there is also provided a computer-readable storage medium. The computer-readable storage medium includes a stored program. When the program runs, it controls the device where the computer-readable storage medium is located to execute the data transmission method in Embodiment 1 above.

[0121] Embodiment 4

[0122] According to an embodiment of the present invention, there is also provided a processor. The processor is used to run a program. When the program runs, it executes the data transmission method in Embodiment 1 above.

[0123] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0124] In the above embodiments of the present invention, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0125] In the several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces. The indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0126] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0127] In addition, the functional units in each embodiment of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0128] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a resource pool, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0129] The foregoing are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A data processing method, characterized in that, Including: Upon receiving a data transmission instruction, obtain a target identifier and target data, where the target identifier is used to represent the business line to which the target service belongs and the attributes of the target service, and the target data is used to represent the resource attributes required by the target service; Based on the target identifier, determine a target cluster in a target platform, where the target platform is used to provide at least one cluster, each cluster corresponding to a different business line respectively, and the target cluster is used to manage at least one resource pool; Based on the target identifier, determine a target resource pool from the target cluster, where the target resource pool includes at least one server; Based on the target data, determine a target server from the target resource pool and transmit the target data to the target server, where the target server is used to run the target service using the target data; Based on the target data, determining a target server from the target resource pool includes: Perform a tagging process on the target data to generate a first tag; Obtain a second tag corresponding to each server in the target resource pool; Compare the first tag with the second tag corresponding to each server to determine a comparison result, where the comparison result is used to describe the similarity between the first tag and the second tag; Based on the comparison result, determine the target server from the target resource pool, where the similarity between the second tag of the target server and the first tag is greater than a preset similarity.

2. The method according to claim 1, characterized in that Based on the target data, determining a target server from the target resource pool includes: Obtain the current remaining resources of the target resource pool and the resource occupancy of the target data; Update the target resource pool based on the current remaining resources and the resource occupancy to obtain an updated target resource pool; Based on the target data, determine the target server from the updated target resource pool.

3. The method according to claim 2, characterized in that, Updating the target resource pool based on the current remaining resources and the resource occupancy to obtain an updated target resource pool includes: Compare the current remaining resources and the resource occupancy to generate a comparison result, where the comparison result is used to indicate whether the resource occupancy is greater than the current remaining resources; In response to the resource occupancy being greater than the current remaining resources, use the target platform to transfer at least one server from other resource pools to the target resource pool through the target cluster to obtain the updated target resource pool, where the other resource pools are the other resource pools except the target resource pool among the at least one resource pools.

4. The method according to claim 3, characterized in that After determining a target server from the target resource pool based on the target data and transmitting the target data to the target server, the method further includes: In response to receiving a transmission success instruction, update the current remaining resources of the target resource pool based on the target data through the target cluster.

5. The method according to claim 4, characterized in that The method further includes: In response to the received transmission success instruction, determine the servers to be removed from the at least one server based on the current remaining resources and the current resource occupancy of the target resource pool; Transfer the servers to be removed to the other resource pool.

6. A data processing device, characterized in that, Comprising: An acquisition module, configured to acquire a target identifier and target data in response to a received data transmission instruction, wherein the target identifier is used to represent the business line to which the target service belongs and the attributes of the target service, and the target data is used to represent the resource attributes required by the target service; A first determination module, configured to determine a target cluster in a target platform based on the target identifier, wherein the target platform is used to provide at least one cluster, each cluster corresponding to a different business line, and the target cluster is used to manage at least one resource pool; A second determination module, configured to determine a target resource pool from the target cluster based on the target identifier, wherein the target resource pool includes at least one server; A transmission module, configured to determine a target server from the target resource pool based on the target data and transmit the target data to the target server, wherein the target server is used to run the target service using the target data; The transmission module further includes: being configured to perform tagging processing on the target data to generate a first tag; acquiring a second tag corresponding to each server in the target resource pool; being configured to determine the target server from the target resource pool based on the first tag and the second tag; being configured to compare the first tag and the second tag corresponding to each server to determine a comparison result, wherein the comparison result is used to describe the similarity between the first tag and the second tag; being configured to determine the target server from the target resource pool based on the comparison result, wherein the similarity between the second tag of the target server and the first tag is greater than a preset similarity.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein when the program runs, it controls the device where the computer-readable storage medium is located to execute the data processing method according to any one of claims 1 to 5.

8. A computer terminal, characterized in that, Comprising: A processor and a memory, the processor being configured to run a program, wherein when the program runs, it executes the data processing method according to any one of claims 1 to 5.

Images