Data processing method and device, equipment, storage medium and program product

Abstract

The invention discloses a data processing method and device, equipment, a storage medium and a program product, relates to the technical field of data processing, and is used for reasonably determining an object storage cluster for processing a data processing request. The method comprises the following steps: receiving a data processing request from a client; determining at least one storage cluster from the plurality of storage clusters based on a position parameter set and / or a performance parameter of each storage cluster in the plurality of storage clusters, the position parameter set comprising a position parameter of the client and a position parameter of each storage cluster, the data processing capability of at least one storage cluster is higher than that of other storage clusters in the plurality of storage clusters; and determining the at least one storage cluster as a processing cluster of the data processing request. The method and the device are applied to a scene of scheduling the storage cluster.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment, storage medium and program product. Background Art

[0002] Object storage is a way of storing data with objects as the basic storage unit. It uses the unique identifier of the object as an index and stores the object as a whole. Object storage adopts a distributed architecture, which combines multiple storage nodes into an object storage cluster (also called a cluster, storage cluster or data node cluster, etc.), which jointly undertakes the task of data storage and access. With the rapid development of big data and cloud computing technologies, enterprises and organizations have established a large number of object storage clusters.

[0003] Different object storage clusters are usually located in different geographical locations, and the data processing capabilities of different object storage clusters are also different. Therefore, when receiving a data processing request (also called a data storage and processing request) from a client, how to reasonably determine the object storage cluster that processes the data processing request becomes an urgent problem to be solved. Summary of the invention

[0004] The present application provides a data processing method, apparatus, device, storage medium and program product, which are used to solve the technical problem of being unable to reasonably determine an object storage cluster that processes a data processing request when receiving a data processing request from a client.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a data processing method is provided, the method comprising: receiving a data processing request from a client; determining at least one storage cluster from multiple storage clusters based on a location parameter set and / or performance parameters of each storage cluster in the multiple storage clusters, the location parameter set comprising location parameters of the client and location parameters of each storage cluster, the data processing capability of the at least one storage cluster being higher than the data processing capability of other storage clusters in the multiple storage clusters; and determining at least one storage cluster as a processing cluster for the data processing request.

[0007] In a possible implementation, there are multiple clients; there are multiple data processing requests; based on a set of location parameters and / or performance parameters of each storage cluster in the multiple storage clusters, at least one storage cluster is determined from the multiple storage clusters, including: based on the location parameters of each client in the multiple clients and the location parameters of each storage cluster, determining the distance between each client and each storage cluster; based on the distance between each client and each storage cluster and / or the performance parameters of each storage cluster, determining at least one storage cluster from the multiple storage clusters, wherein one client corresponds to M storage clusters, one storage cluster is used to process N data processing requests, and M and N are positive integers.

[0008] In a possible implementation, based on the distance between each client and each storage cluster and / or the performance parameters of each storage cluster, at least one storage cluster is determined from multiple storage clusters, including: obtaining a first probability distribution and a second probability distribution, the first probability distribution is used to characterize the number of data processing requests sent by each client, and the second probability distribution is used to characterize the number of data processing requests processed by each storage cluster; based on the distance between each client and each storage cluster and / or the performance parameters of each storage cluster, determining a capability matrix; the capability matrix is ​​used to characterize the ability of each storage cluster to process the data processing request of each client; based on the first probability distribution, the second probability distribution and the capability matrix, obtaining a relationship matrix; the relationship matrix is ​​used to characterize the corresponding relationship between the client and the storage cluster; based on the relationship matrix, determining at least one storage cluster from multiple storage clusters.

[0009] In one possible implementation, a capability matrix is ​​determined based on the distance between each client and each storage cluster and / or the performance parameters of each storage cluster, including: determining the sum of a target distance and a target performance parameter as the capability matrix; the target distance includes the distance between each client and each storage cluster; the target performance parameter includes the performance parameter of each storage cluster; or, determining the target distance as the capability matrix; or, determining the target performance parameter as the capability matrix.

[0010] In a possible implementation, the statistical parameter includes a probability distribution function; the probability distribution function is used to characterize the probability of time consumption of running the application to be evaluated based on preset computing resources.

[0011] In a possible implementation, there are multiple clients; there are multiple data processing requests; the method also includes: obtaining a first probability distribution and a second probability distribution, the first probability distribution is used to characterize the number of data processing requests sent by each of the multiple clients, and the second probability distribution is used to characterize the number of data processing requests corresponding to each storage cluster in the multiple storage clusters; based on the first probability distribution and the second probability distribution, a relationship matrix is ​​obtained; the relationship matrix is ​​used to characterize the corresponding relationship between the client and the storage cluster; based on the relationship matrix, at least one storage cluster is determined from the multiple storage clusters.

[0012] In a possible implementation manner, the performance parameter includes at least one of the following: latency, disk utilization, memory utilization, bandwidth utilization, and processor utilization.

[0013] In a possible implementation, multiple storage clusters correspond to one domain name.

[0014] In a second aspect, a data processing device is provided, which includes: a transmission unit and a processing unit; the transmission unit is used to receive a data processing request from a client; the processing unit is used to determine at least one storage cluster from multiple storage clusters based on a location parameter set and / or performance parameters of each storage cluster in the multiple storage clusters, the location parameter set includes location parameters of the client and location parameters of each storage cluster, and the data processing capability of at least one storage cluster is higher than the data processing capability of other storage clusters in the multiple storage clusters; the processing unit is also used to determine at least one storage cluster as a processing cluster for the data processing request.

[0015] In a possible implementation, there are multiple clients; there are multiple data processing requests; the processing unit is further used to determine the distance between each client and each storage cluster based on the location parameters of each client in the multiple clients and the location parameters of each storage cluster; the processing unit is further used to determine at least one storage cluster from the multiple storage clusters based on the distance between each client and each storage cluster and / or the performance parameters of each storage cluster, wherein one client corresponds to M storage clusters, one storage cluster is used to process N data processing requests, and M and N are positive integers.

[0016] In a possible implementation, the transmission unit is further used to obtain a first probability distribution and a second probability distribution, the first probability distribution is used to characterize the number of data processing requests sent by each client, and the second probability distribution is used to characterize the number of data processing requests processed by each storage cluster; the processing unit is further used to determine a capability matrix based on the distance between each client and each storage cluster and / or the performance parameters of each storage cluster; the capability matrix is ​​used to characterize the ability of each storage cluster to process the data processing request of each client; the processing unit is further used to obtain a relationship matrix based on the first probability distribution, the second probability distribution and the capability matrix; the relationship matrix is ​​used to characterize the corresponding relationship between the client and the storage cluster; the processing unit is further used to determine at least one storage cluster from multiple storage clusters based on the relationship matrix.

[0017] In one possible implementation, the processing unit is further used to determine the sum of the target distance and the target performance parameter as a capability matrix; the target distance includes the distance between each client and each storage cluster; the target performance parameter includes the performance parameter of each storage cluster; the processing unit is further used to determine the target distance as the capability matrix; the processing unit is further used to determine the target performance parameter as the capability matrix.

[0018] In a possible implementation, there are multiple clients; there are multiple data processing requests; the transmission unit is further used to obtain a first probability distribution and a second probability distribution, the first probability distribution is used to characterize the number of data processing requests sent by each of the multiple clients, and the second probability distribution is used to characterize the number of data processing requests corresponding to each storage cluster in the multiple storage clusters; the processing unit is further used to obtain a relationship matrix based on the first probability distribution and the second probability distribution; the relationship matrix is ​​used to characterize the corresponding relationship between the client and the storage cluster; the processing unit is further used to determine at least one storage cluster from the multiple storage clusters based on the relationship matrix.

[0019] In a possible implementation manner, the performance parameter includes at least one of the following: latency, disk utilization, memory utilization, bandwidth utilization, and processor utilization.

[0020] In a possible implementation, multiple storage clusters correspond to one domain name.

[0021] In a third aspect, an electronic device comprises: a processor and a memory; wherein the memory is used to store one or more programs, and the one or more programs include computer execution instructions. When the electronic device is running, the processor executes the computer execution instructions stored in the memory to enable the electronic device to perform a data processing method as in the first aspect.

[0022] In a fourth aspect, a computer-readable storage medium storing one or more programs is provided. The one or more programs include instructions. When the instructions are executed by a computer, the computer executes a data processing method as described in the first aspect.

[0023] In a fifth aspect, a computer program product is provided. When the computer instructions are executed on an electronic device, the electronic device executes a data processing method as in the first aspect.

[0024] The present application provides a data processing method, apparatus, device, storage medium and program product, which are applied in the scenario of data processing. When it is necessary to determine the storage cluster for processing a data processing request, a data processing request from a client can be received, and at least one storage cluster can be determined from multiple storage clusters based on the location parameters of the client and the location parameters of each storage cluster in multiple storage clusters and / or the performance parameters of each storage cluster. Further, at least one storage cluster is determined as the processing cluster for the data processing request to process the data processing request based on at least one storage cluster. Among them, the data processing capacity of at least one storage cluster is higher than the data processing capacity of other storage clusters in multiple storage clusters. That is, the closer the location of the storage cluster is to the location of the client, the shorter the time it takes to transmit the data processing request of the client to the storage cluster for processing, that is, the higher the data processing capacity of the storage cluster for processing the client. And, the better the performance parameters of the storage cluster, the better the effect of the storage cluster in processing the data processing request (including short time consumption, high accuracy, low loss rate, etc.), that is, the higher the data processing capacity of the storage cluster for processing the client. Thus, the object storage cluster for processing the data processing request can be reasonably determined.

[0025] Through the above method, at least one storage cluster with the highest data processing capability for the data processing request of the client can be determined from multiple storage clusters, so as to process the data processing request of the client based on these clusters. Thus, the technical problem of being unable to reasonably determine the object storage cluster that processes the data processing request when receiving the data processing request from the client is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of a data processing method provided in an embodiment of the present application Figure 1 ;

[0028] Figure 3 A schematic diagram of a data processing method provided in an embodiment of the present application Figure 2 ;

[0029] Figure 4 A schematic diagram of the location distribution of a client and a storage cluster provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of a data processing method provided in an embodiment of the present application Figure 3 ;

[0031] Figure 6 A schematic diagram of a data processing method provided in an embodiment of the present application Figure 4 ;

[0032] Figure 7 A schematic diagram of a data processing method provided in an embodiment of the present application Figure 5 ;

[0033] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0034] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0036] In the description of this application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" and "plurality" refer to two or more. The words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be different.

[0037] With the rapid development of big data and cloud computing technologies, enterprises and organizations have established a large number of infrastructures for storing and processing data related to big data and cloud computing technologies. Due to the limitations of technology iteration and initial planning, a large number of small and medium-sized storage clusters have been formed. These storage clusters are often distributed in different geographical locations. Although each small and medium-sized storage cluster can meet some specific needs, overall, these small and medium-sized storage clusters have problems such as low resource utilization and insufficient funds for subsequent expansion. How to effectively integrate and utilize these scattered small and medium-sized storage clusters has become a research topic that needs to be solved urgently.

[0038] In response to the above-mentioned multi-cluster scheduling problem, the relevant technology adopts an open source platform for managing containerized applications (kubernets, K8s) and a containerized multi-cluster scheduling method to solve the multi-cluster scheduling problem. For example, the multi-cluster scheduling problem is solved based on an open source platform for joint management of containerized applications (kubernetes federation, KubeFed) or lightweight cross-cluster communication (Liqo). Specifically, KubeFed can define a series of custom federated resources through custom resource definition (Custom Resource Definition, CRD) + controller (Operator), and realize cross-cluster resource coordination capabilities through a custom controller (controller). Liqo is a more dynamic solution. Its design allows clusters to automatically discover each other and share resources when necessary. It can better dynamically migrate loads to achieve disaster recovery and balance.

[0039] However, the K8s containerized multi-cluster scheduling method is only a solution for containerized deployment applications. For non-containerized deployment applications, some additional tools and mechanisms are needed to migrate non-containerized deployment applications to containerization, which requires more configuration and customization. In addition, in the actual deployment of storage services, manufacturers will not adopt containerized deployment solutions due to storage performance considerations. Therefore, the containerized scheduling solution cannot be applied to multi-cluster management of object storage services, and a new solution needs to be proposed.

[0040] In order to solve the above-mentioned multi-cluster scheduling problem, effectively integrate and utilize the existing scattered small and medium-sized cluster resources, on the basis of introducing a unified domain name in a proxy-like layer manner, this application provides a data processing method, which can realize multi-cluster scheduling through a sinkhorn algorithm that is effective for multi-object storage cluster scheduling, and can adapt to the current large-scale commercial non-containerized deployment scheme. The algorithm can comprehensively consider the performance parameters of each storage cluster (such as resource conditions, load) and the distance between each storage cluster and the client of the data processing request (also known as a new storage bucket (bucket) request) and other factors to construct a cost matrix (i.e., the capacity matrix in this application, which can also be called a cost matrix). For batch bucket creation requests, the final scheduling position of each bucket (i.e., the storage cluster corresponding to each bucket creation request) can be calculated through multiple rounds of algorithm iterations, and the location and quantity recommendations of multi-bucket creation clusters are obtained, so as to improve the balance of capacity utilization of multi-storage clusters, successfully alleviate the pressure of some overloaded clusters, and extend the service life of these key nodes. At the same time, by considering the performance parameters of each storage cluster, the comprehensive performance of the multi-cluster is improved, the request processing speed on the user side is improved, and the overall user experience becomes more friendly. In addition, it can predict and avoid capacity risks in advance, fully utilize and balance multi-cluster resources, reduce operation and maintenance costs, and increase corporate net profits.

[0041] A data processing method provided in an embodiment of the present application may be applicable to a data processing system. Figure 1 FIG. 1 shows a schematic diagram of the structure of a data processing system. Figure 1 As shown, the data processing system 10 includes: a client 11, an electronic device 12 and a storage cluster 13. The client 11, the electronic device 12 and the storage cluster 13 can be connected by wire or by wireless, which is not limited in the embodiment of the present invention. The client 11 can include: an API, an SDK, a gadget, a browser, and a desktop tool. The number of clients 11 can be multiple, and the number of storage clusters 13 can be multiple.

[0042] The client 11 can send a data processing request to the scheduling layer of the electronic device 12 corresponding to the unified domain name address (also referred to as domain name) of the plurality of storage clusters 13 through an application programming interface (API), a software development kit (SDK), a gadget, a browser, a desktop tool, etc. The number of data processing requests sent by a client 11 can be multiple.

[0043] When receiving the data processing request, the scheduling layer of the electronic device 12 determines at least one storage cluster 13 from the multiple storage clusters 13 based on the location parameter of the client 11 and the location parameter of each storage cluster 13 in the multiple storage clusters 13 and / or the performance parameter of each storage cluster 13, and determines the at least one storage cluster 13 as the processing cluster for the data processing request. The at least one storage cluster 13 is used to process the data processing request of the client 11.

[0044] It should be noted that if Figure 1 As shown, the multiple storage clusters 13 currently in operation are all independently operated, and each storage cluster 13 has its own domain name address. This application can unify the domain name addresses of multiple storage clusters 13 to achieve unified entrances to multiple storage clusters 13. This change does not involve changes in the usage posture of the client 11. The client 11 only needs to adjust the domain name address to be accessed (that is, adjust the domain name address to be accessed to a unified domain name address). Based on the demand for unified scheduling, it is necessary to add a scheduling layer to the upper layer of multiple storage clusters, and complete task allocation through the scheduling layer according to the scheduling algorithm proposed in this application, so as to achieve data processing requests as much as possible to be distributed to storage clusters with abundant resources and good performance while taking into account latency, and to achieve capacity balance and pressure balance among multiple storage clusters.

[0045] The client 11 may be a device with wireless transceiver functions, such as a mobile phone, a tablet computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a desktop computer, also known as a desktop computer or a desktop computer. The electronic device 12 is a desktop computer, a server, or a server group composed of multiple servers. The storage cluster 13 may be a server cluster composed of multiple servers. Alternatively, the server cluster may also be a distributed cluster. Alternatively, the server may be a cloud server. The embodiments of the present application do not limit the specific implementation of the client 11, the electronic device 12, and the storage cluster 13.

[0046] The following describes a data processing method provided by an embodiment of the present application in conjunction with the accompanying drawings. Figure 2 As shown, a data processing method provided in an embodiment of the present application is applied to an electronic device, and the method includes S201-S203:

[0047] S201: Receive a data processing request from a client.

[0048] In a possible implementation, multiple storage clusters correspond to one domain name.

[0049] Optionally, the client can send the data processing request to the scheduling layer of the electronic device corresponding to one domain name (i.e., a unified domain name) corresponding to multiple storage clusters, thereby avoiding the situation in the related technology where a storage cluster has only one domain name, resulting in the inability to select a cluster to process the request from multiple storage clusters. It can realize the construction of a unified entrance for multiple storage clusters, making it convenient to select a cluster to process the request from multiple storage clusters.

[0050] Furthermore, the scheduling layer of the electronic device can receive a data processing request from a client. The client can be a device that needs to process, store and / or retrieve relevant data, and the data processing request can be a data processing request issued by the client when it needs to process, store and / or retrieve relevant data. The data processing request is used to request the storage cluster to process, store and / or retrieve the relevant data. The data processing request can also be called a create bucket request, because before the storage cluster processes, stores and / or retrieves the data, the scheduling layer of the electronic device needs to create a storage bucket corresponding to the storage cluster for the data processing request, so as to store the relevant data of the data processing request through the storage bucket.

[0051] Furthermore, when the central control module of the scheduling layer of the electronic device receives a data processing request from a client, it can parse out the bucket name of the data processing request, and call the metadata module to query the bucket related information to confirm whether the bucket of the data processing request already exists. If the bucket of the data processing request already exists, the metadata module returns the cluster record (i.e., the storage cluster corresponding to the bucket of the data processing request) to the central control module. The existence of the bucket of the data processing request indicates that the storage cluster that processes the data request has been determined for the data request, and no subsequent steps are required.

[0052] S202: Determine at least one storage cluster from multiple storage clusters based on a location parameter set and / or a performance parameter of each storage cluster in the multiple storage clusters.

[0053] The location parameter set includes the location parameter of the client and the location parameter of each storage cluster, and the data processing capability of at least one storage cluster is higher than the data processing capability of other storage clusters in the multiple storage clusters.

[0054] Optionally, if the storage bucket of the data processing request does not exist, the scheduling layer of the electronic device obtains the location parameters of the client and the location parameters of each storage cluster in the multiple storage clusters and the performance parameters of each storage cluster. Furthermore, the scheduling layer of the electronic device determines at least one storage cluster from the multiple storage clusters based on the location parameters of the client and the location parameters of each storage cluster in the multiple storage clusters and the performance parameters of each storage cluster. The location parameter may be a geographic location, such as longitude and latitude coordinates. That is, the closer the location of the storage cluster is to the location of the client, the shorter the time it takes to transmit the data processing request of the client to the storage cluster for processing, and the higher the data processing capacity of the storage cluster for processing the client, and vice versa.

[0055] In a possible implementation, the performance parameter includes at least one of the following: latency, disk utilization, memory utilization, bandwidth utilization, and processor (also referred to as central processing unit (CPU)) utilization.

[0056] It should be noted that the smaller the performance parameter, the better the performance of the storage cluster. The smaller the latency of the storage cluster, the smaller the network latency of the storage cluster, the shorter the time it takes for the storage cluster to process data processing requests, and the higher the data processing capacity of the storage cluster to process the client, and vice versa. The smaller the disk utilization, memory utilization, bandwidth utilization, and / or CPU utilization of the storage cluster, the more idle the storage cluster is, the shorter the time it takes for the storage cluster to process data processing requests, the higher the accuracy, and the lower the loss rate, and the higher the data processing capacity of the storage cluster to process the client, and vice versa.

[0057] S203: Determine at least one storage cluster as a processing cluster for the data processing request.

[0058] Optionally, the scheduling layer of the electronic device may determine at least one storage cluster as a processing cluster for the data processing request. In this way, at least one storage cluster may process the data processing request of the electronic device to complete data processing, storage and / or retrieval of relevant data of the client.

[0059] In one design, Figure 3 As shown, in a data processing method provided in an embodiment of the present application, there are multiple clients; there are multiple data processing requests; the method in the above step S201 specifically includes S301-S302:

[0060] S301 : Determine a distance between each client and each storage cluster based on a location parameter of each client among a plurality of clients and a location parameter of each storage cluster.

[0061] Optionally, the multiple data processing requests sent by the multiple clients may be the same batch of data processing requests sent at the same time point. The scheduling layer of the electronic device may determine the distance between each client and each storage cluster based on the location parameter of each client among the multiple clients and the location parameter of each storage cluster. The distance between the client and the storage cluster may be obtained by subtracting the location parameter of a storage cluster from the location parameter of the client.

[0062] For example, Figure 4 As shown, the number of clients is 3, namely, client 1, client 2, and client 3. The number of data processing requests of client 1 is 2, the number of data processing requests of client 2 is 3, and the number of data processing requests of client 3 is 4. The number of multiple storage clusters is 4, namely, storage cluster A, storage cluster B, storage cluster C, and storage cluster D.

[0063] S302: Determine at least one storage cluster from multiple storage clusters based on the distance between each client and each storage cluster and / or the performance parameter of each storage cluster.

[0064] Among them, one client corresponds to M storage clusters, and one storage cluster is used to process N data processing requests, where M and N are positive integers.

[0065] Optionally, the scheduling layer of the electronic device can determine at least one storage cluster from multiple storage clusters based on the distance between each client and each storage cluster and / or the performance parameters of each storage cluster, that is, determine the storage cluster corresponding to each data processing request of each client. The closer the location of the storage cluster is to the client, the shorter the time it takes to transmit the data processing request of the client to the storage cluster for processing, and the higher the data processing capacity of the storage cluster for processing the client, and vice versa.

[0066] Exemplarily, the storage cluster that processes two data processing requests from client 1 is storage cluster A, the storage cluster that processes three data processing requests from client 2 is storage cluster B, the storage cluster that processes three of the four data processing requests from client 3 is storage cluster C, and the storage cluster that processes another one of the four data processing requests from client 3 is storage cluster D.

[0067] This application can allocate each data processing request of each client to a suitable storage cluster for processing according to the distance between each client and each storage cluster and / or the performance parameters of each storage cluster. That is, by comprehensively considering the performance parameters of each storage cluster and the distance between each storage cluster and the bucket creation request and other factors, the final scheduling position of each bucket (that is, the storage cluster corresponding to each bucket creation request) can be obtained for batch bucket creation requests, and the location and quantity recommendations of multiple bucket creation clusters can be obtained. The balance of capacity utilization of multiple storage clusters is improved, the pressure of some overloaded clusters is successfully alleviated, and the service life of these key nodes is extended. At the same time, by considering the performance parameters of each storage cluster, the comprehensive performance of multiple clusters is improved, the request processing speed on the user side is improved, and the overall user experience becomes more friendly. In addition, it can predict and avoid capacity risks in advance, make full use of and balance the resources of multiple clusters, reduce operation and maintenance costs, and increase corporate net profits.

[0068] In one design, Figure 5 As shown, a data processing method provided in an embodiment of the present application, the method in the above step S302 includes: S401-S404:

[0069] S401. Obtain a first probability distribution and a second probability distribution.

[0070] The first probability distribution is used to characterize the number of data processing requests sent by each client, and the second probability distribution is used to characterize the number of data processing requests processed by each storage cluster.

[0071] Optionally, the electronic device may determine at least one storage cluster from the plurality of storage clusters using an entropy regularized sinkhorn algorithm based on the distance between each client and each storage cluster and / or a performance parameter of each storage cluster.

[0072] The specific execution steps of determining at least one storage cluster from multiple storage clusters using the sinkhorn algorithm may be as shown in S1-S4:

[0073] S1. Model the discrete probability distribution α (i.e., the first probability distribution in this application) of data processing requests sent by multiple clients as {x1…,x n}, where n is the number of clients, and the discrete probability distribution β of the multiple storage clusters (i.e., the second probability distribution in this application) is modeled as {y1…,y m}, where m is the number of storage clusters. The target problem can be modeled as solving a mapping T with the minimum scheduling cost: {x1…,x n}→{y1…,y mThis problem can be simplified as "multiple x → For the matching problem of y", the discrete probability distribution vector α∈R n and β∈R m , given an initial cost matrix (also called capability matrix, cost matrix) C∈R n×m , the transfer matrix (also called the relationship matrix) is defined as P. Solving the minimum cost matching matrix P for mapping from α distribution to β distribution can be quantified into the following optimization problem:

[0074]

[0075] In Formula 1, i represents the i-th client, and j represents the j-th storage cluster.

[0076] Exemplarily, n is 3, m is 4, x1 in the first probability distribution is 2 (i.e., the number of data processing requests sent by client 1 is 2), x2 is 3 (i.e., the number of data processing requests sent by client 2 is 3), and x3 is 4 (i.e., the number of data processing requests sent by client 3 is 4). In the second probability distribution, y1 is 2 (i.e., the number of data processing requests processed by storage cluster A is 2), y2 is 3 (i.e., the number of data processing requests processed by storage cluster B is 3), y3 is 4 (i.e., the number of data processing requests processed by storage cluster C is 4), and y4 is 5 (i.e., the number of data processing requests processed by storage cluster D is 5).

[0077] S402: Determine a capability matrix based on the distance between each client and each storage cluster and / or the performance parameter of each storage cluster.

[0078] The capability matrix is ​​used to characterize the capability of each storage cluster to process the data processing request of each client.

[0079] Optionally, the electronic device may determine the capability matrix based on the distance between each client and each storage cluster and / or the performance parameter of each storage cluster.

[0080] In one design, Figure 6 As shown, a data processing method provided in an embodiment of the present application, the method in the above step S402 includes: S501, or S502, or S503:

[0081] S501. Determine the sum of the target distance and the target performance parameter as a capability matrix.

[0082] The target distance includes the distance between each client and each storage cluster; and the target performance parameter includes the performance parameter of each storage cluster.

[0083] It should be noted that, unlike traditional transportation problems, the reasonable scheduling of storage clusters cannot only consider the geographical distance factor, but also the network delay caused by spatial distance, cluster performance, cluster utilization, cluster memory utilization, cluster bandwidth utilization, cluster CPU utilization indicators, etc. How to effectively organize these performance parameters and distances, and use these performance parameters and distances to build a multi-factor capability matrix is ​​the key to the algorithm.

[0084] Optionally, in order to comprehensively utilize the scheduling cost parameters of the above multiple storage clusters, the present application proposes a multi-factor cost matrix (i.e., capability matrix) algorithm. The steps of constructing the multi-factor capability matrix are as follows:

[0085] S2. Multiple performance parameters of storage cluster A can be defined as:

[0086] A=[a1,a2,a3,a4,a5,a6]Formula 3;

[0087] Among them, a i ,i=1,2,3,4,5,6 represent the cluster vectors of latency, disk utilization, memory utilization, bandwidth utilization, and CPU utilization of each storage cluster, respectively.

[0088] To facilitate subsequent calculations, discrete normalization is used to sort each cluster vector in turn, and the following is obtained:

[0089]

[0090] Among them, a min is the minimum performance parameter among the performance parameters of each storage cluster, a max It is the maximum performance parameter among the performance parameters of each storage cluster.

[0091] Next, we use weighted average to sort out the multiple parameters, and the multi-factor cost matrix c from the i-th user end to the j-th storage cluster is i,j It can be expressed as:

[0092]

[0093] in, represents the parameter vector of the jth storage cluster, w represents the weight vector of the corresponding storage cluster’s corresponding parameters, and d i,j represents the distance from the i-th client to the j-th storage cluster after discrete normalization, w d represents the weight value of the distance. And w and w d Satisfy formula six:

[0094]

[0095] After the above operations and solutions, the solution of the multi-factor cost matrix under the storage scheduling of multi-storage clusters is completed.

[0096] S502: Determine the target distance as a capability matrix.

[0097] Alternatively, if only geographical distance is considered when scheduling storage clusters, the capacity matrix can be d in Formula 5 i,j .

[0098] S503: Determine the target performance parameters as a capability matrix.

[0099] Optionally, if only performance parameters are considered when scheduling the storage cluster, the capacity matrix can be

[0100] S403: Obtain a relationship matrix based on the first probability distribution, the second probability distribution and the capability matrix.

[0101] The relationship matrix is ​​used to represent the corresponding relationship between the client and the storage cluster.

[0102] Optionally, based on the first probability distribution, the second probability distribution and the capability matrix, the scheduling layer of the electronic device can obtain a relationship matrix. The specific steps are as follows:

[0103] S3, the initial capacity matrix (C∈R n×m ) is replaced by the capability matrix (i.e., the multi-factor cost matrix c) obtained in step S501, S502, or S503 above. i,j d i,j ,or ).

[0104] S4. Introduce a regularization term D(P) in Formula 1 and change it to find an approximate solution to the problem. Its expression is shown in Formula 7:

[0105]

[0106] Therefore, the problem after regularization is shown in Formula 8:

[0107]

[0108] Among them, ε>0 represents the coefficient of the regularization term, which is a small constant. Introducing the dual variables λ0 and λ1, and the Lagrangian function L in Formula 8, it can be expressed as Formula 9:

[0109] L(P,λ0,λ1)=<C,P> +εD(P)+(λ0,P1-α)+(λ1,P T 1-β) Formula 9;

[0110] For a fixed dual variable, the minimum of L is obtained when the gradient with respect to P is zero. For each element P in Equation 9 i,j Taking partial derivatives, we can get formula 10:

[0111]

[0112] Then the transmission matrix (ie, the relationship matrix in this application) can be expressed as Formula 11:

[0113] P i,j =u i K i,j v j Formula XI;

[0114] Among them, u i is the K i,j is the v j is the

[0115] The transmission matrix after adjusting Formula 11 can be expressed as Formula 12:

[0116] P = diag (u) K diag (v) Formula 12;

[0117] Where, diag(u) is the u in formula 11. i , K is K in formula 11 i,j , diag(v) is v in formula 11 j .

[0118] S404: Determine at least one storage cluster from multiple storage clusters based on the relationship matrix.

[0119] Optionally, the scheduling layer of the electronic device can determine the storage cluster corresponding to each data processing request in a batch of data processing requests (i.e., multiple data processing requests sent by each of the multiple clients in this application) based on the transmission matrix, forward each data processing request to the corresponding storage cluster for processing, and create a new storage bucket for the data processing request sent by each client.

[0120] For example, Figure 4As shown in the figure, it is assumed that the performance of storage cluster A, storage cluster B, storage cluster C, and storage cluster D are ranked from high to low as follows: storage cluster A, storage cluster B, storage cluster C, and storage cluster D. The distances between client 1 and storage cluster A, storage cluster B, storage cluster C, and storage cluster D are ranked from small to large as follows: storage cluster A, storage cluster B, storage cluster C, and storage cluster D. The distances between client 2 and storage cluster A, storage cluster B, storage cluster C, and storage cluster D are ranked from small to large as follows: storage cluster B, storage cluster C, storage cluster A, and storage cluster D. The distances between client 3 and storage cluster A, storage cluster B, storage cluster C, and storage cluster D are ranked from small to large as follows: storage cluster C, storage cluster D, storage cluster B, and storage cluster A. Then, based on the relationship matrix including the multiple factor cost matrix, it can be obtained that the storage cluster that processes the two data processing requests of client 1 is storage cluster A, the storage cluster that processes the three data processing requests of client 2 is storage cluster B, the storage cluster that processes three of the four data processing requests of client 3 is storage cluster C, and the storage cluster that processes the other one of the four data processing requests of client 3 is storage cluster D.

[0121] This application organizes the performance parameters of the storage cluster and the distance between the storage cluster and the client, and uses these performance parameters and distances to construct a multi-factor cost matrix. Through the multi-factor cost matrix, data processing requests are dispersed as much as possible to storage clusters with abundant resources and good performance while taking latency into consideration, thereby achieving capacity balance and pressure balance among multiple storage clusters.

[0122] In one design, Figure 7 As shown, in an embodiment of the present application, a data processing method is provided, the number of clients is multiple; the number of data processing requests is multiple; the method in the above step S302 includes: S601-S603:

[0123] S601. Obtain a first probability distribution and a second probability distribution.

[0124] The first probability distribution is used to characterize the number of data processing requests sent by each of the multiple clients, and the second probability distribution is used to characterize the number of data processing requests processed by each of the multiple storage clusters.

[0125] S602: Obtain a relationship matrix based on the first probability distribution and the second probability distribution.

[0126] The relationship matrix is ​​used to represent the corresponding relationship between the client and the storage cluster.

[0127] Optionally, the electronic device may determine at least one storage cluster from a plurality of storage clusters using a sinkhorn algorithm based on the first probability distribution, the second probability distribution and the initial capability matrix.

[0128] The specific execution steps of determining at least one storage cluster from multiple storage clusters using the sinkhorn algorithm may be as shown in S1 and S4 in the above embodiment, and will not be repeated here.

[0129] S603: Determine at least one storage cluster from multiple storage clusters based on the relationship matrix.

[0130] Optionally, the scheduling layer of the electronic device can determine the storage cluster corresponding to each of the multiple data processing requests sent by each of the multiple clients based on the transmission matrix, forward each data processing request to the corresponding storage cluster for processing, and create a new storage bucket for the data processing request sent by each client.

[0131] It should be noted that if the cost matrix in the sinkhorn algorithm is the initial cost matrix, then the sinkhorn algorithm can only obtain the corresponding allocation results based on the number of data processing requests sent by each client and the number of data processing requests processed by each storage cluster, without considering the distance between the client and the storage cluster, and the performance parameters of the storage cluster.

[0132] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0133] The embodiment of the present application can divide a data processing method into functional modules according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0134] Figure 8A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 8 As shown, a data processing device 100 is used to reasonably determine an object storage cluster for processing a data processing request, for example, for executing Figure 2 A data processing method is shown. The data processing device 100 includes: a transmission unit 1001 and a processing unit 1002.

[0135] The transmission unit 1001 is used to receive a data processing request from a client.

[0136] The processing unit 1002 is used to determine at least one storage cluster from multiple storage clusters based on a location parameter set and / or performance parameters of each storage cluster in the multiple storage clusters, the location parameter set includes location parameters of the client and location parameters of each storage cluster, and the data processing capacity of the at least one storage cluster is higher than the data processing capacity of other storage clusters in the multiple storage clusters.

[0137] The processing unit 1002 is further configured to determine at least one storage cluster as a processing cluster for the data processing request.

[0138] In a possible implementation, there are multiple clients; there are multiple data processing requests; the processing unit 1002 is further used to determine the distance between each client and each storage cluster based on the location parameters of each client in the multiple clients and the location parameters of each storage cluster; the processing unit 1002 is further used to determine at least one storage cluster from the multiple storage clusters based on the distance between each client and each storage cluster and / or the performance parameters of each storage cluster, wherein one client corresponds to M storage clusters, one storage cluster is used to process N data processing requests, and M and N are positive integers.

[0139] In a possible implementation, the transmission unit 1001 is also used to obtain a first probability distribution and a second probability distribution, the first probability distribution is used to characterize the number of data processing requests sent by each client, and the second probability distribution is used to characterize the number of data processing requests corresponding to each storage cluster; the processing unit 1002 is also used to determine a capability matrix based on the distance between each client and each storage cluster and / or the performance parameters of each storage cluster; the capability matrix is ​​used to characterize the ability of each storage cluster to process the data processing request of each client; the processing unit 1002 is also used to obtain a relationship matrix based on the first probability distribution, the second probability distribution and the capability matrix; the relationship matrix is ​​used to characterize the corresponding relationship between the client and the storage cluster; the processing unit 1002 is also used to determine at least one storage cluster from multiple storage clusters based on the relationship matrix.

[0140] In one possible implementation, processing unit 1002 is also used to determine the sum of the target distance and the target performance parameter as a capability matrix; the target distance includes the distance between each client and each storage cluster; the target performance parameter includes the performance parameter of each storage cluster; processing unit 1002 is also used to determine the target distance as a capability matrix; processing unit 1002 is also used to determine the target performance parameter as a capability matrix.

[0141] In a possible implementation, there are multiple clients; there are multiple data processing requests; the transmission unit 1001 is also used to obtain a first probability distribution and a second probability distribution, the first probability distribution is used to characterize the number of data processing requests sent by each of the multiple clients, and the second probability distribution is used to characterize the number of data processing requests corresponding to each storage cluster in the multiple storage clusters; the processing unit 1002 is also used to obtain a relationship matrix based on the first probability distribution and the second probability distribution; the relationship matrix is ​​used to characterize the corresponding relationship between the client and the storage cluster; the processing unit 1002 is also used to determine at least one storage cluster from the multiple storage clusters based on the relationship matrix.

[0142] In a possible implementation manner, the performance parameter includes at least one of the following: latency, disk utilization, memory utilization, bandwidth utilization, and processor utilization.

[0143] In a possible implementation, multiple storage clusters correspond to one domain name.

[0144] In the case of implementing the functions of the above-mentioned integrated modules in the form of hardware, the embodiment of the present application provides a possible structural diagram of the electronic device involved in the above-mentioned embodiment. Fig. 9 As shown, an electronic device 120 is used to reasonably determine an object storage cluster for processing a data processing request, such as for executing Figure 2 The electronic device 120 includes a processor 1201, a memory 1202 and a bus 1203. The processor 1201 and the memory 1202 may be connected via the bus 1203.

[0145] The processor 1201 is the control center of the communication device, which can be a processor or a general term for multiple processing elements. For example, the processor 1201 can be a general-purpose central processing unit (CPU) or other general-purpose processors. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0146] As an embodiment, the processor 1201 may include one or more CPUs, such as Fig. 9CPU 0 and CPU 1 are shown in .

[0147] The memory 1202 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0148] As a possible implementation, the memory 1202 may exist independently of the processor 1201, and the memory 1202 may be connected to the processor 1201 via a bus 1203 to store instructions or program codes. When the processor 1201 calls and executes the instructions or program codes stored in the memory 1202, a data processing method provided in an embodiment of the present application can be implemented.

[0149] In another possible implementation, the memory 1202 may also be integrated with the processor 1201 .

[0150] The bus 1203 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0151] It should be pointed out that Fig. 9 The structure shown does not constitute a limitation on the electronic device 120. Fig. 9 In addition to the components shown, the electronic device 120 may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0152] As an example, combining Figure 8 The functions implemented by the transmission unit 1001 and the processing unit 1002 in the data processing device 100 are similar to those Fig. 9 The function of processor 1201 in is the same.

[0153] Optional, such as Fig. 9 As shown, the electronic device 120 provided in the embodiment of the present application may further include a communication interface 1204 .

[0154] The communication interface 1204 is used to connect with other devices through a communication network. The communication network may be Ethernet, wireless access network, wireless local area network (WLAN), etc. The communication interface 1204 may include a receiving unit for receiving data and a sending unit for sending data.

[0155] In one design, in the electronic device provided in the embodiment of the present application, the communication interface can also be integrated into the processor.

[0156] Through the description of the above implementation methods, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above functional units is used as an example. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the device can be divided into different functional units to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0157] An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When a computer executes the instructions, the computer executes each step in the method flow shown in the above method embodiment.

[0158] An embodiment of the present application provides a computer program product. When a computer instruction is executed on an electronic device, the electronic device executes a data processing method in the above method embodiment.

[0159] Among them, the computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk. Random access memory (random access memory, RAM), read-only memory (read-only memory, ROM), erasable programmable read only memory (erasable programmable read only memory, EPROM), registers, hard disks, optical fibers, portable compact disc read-only memory (compact disc read-only memory, CD-ROM), optical storage devices, magnetic storage devices, or the above in a suitable combination, or any other form of computer-readable storage medium known in the art.

[0160] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC).

[0161] In the embodiments of the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program, which may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0162] Since the electronic device, computer-readable storage medium, and computer program product in the embodiments of the present application can be applied to the above method, the technical effects that can be obtained can also refer to the above method embodiments, and the embodiments of the present application will not be repeated here.

[0163] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Receive data processing requests from clients; Determine at least one storage cluster from the multiple storage clusters based on a location parameter set and / or a performance parameter of each storage cluster in the multiple storage clusters, the location parameter set including the location parameter of the client and the location parameter of each storage cluster, the data processing capability of the at least one storage cluster being higher than the data processing capability of other storage clusters in the multiple storage clusters; The at least one storage cluster is determined as a processing cluster for the data processing request.

2. The method according to claim 1, characterized in that The number of the clients is multiple; the number of the data processing requests is multiple; and the determining at least one storage cluster from the multiple storage clusters based on the location parameter set and / or the performance parameter of each storage cluster in the multiple storage clusters includes: Determine the distance between each client and each storage cluster based on the location parameter of each client among the multiple clients and the location parameter of each storage cluster; Based on the distance between each client and each storage cluster and / or the performance parameters of each storage cluster, at least one storage cluster is determined from the multiple storage clusters, wherein one client corresponds to M storage clusters, one storage cluster is used to process N data processing requests, and M and N are positive integers.

3. The method according to claim 2, characterized in that The determining the at least one storage cluster from the plurality of storage clusters based on the distance between each client and each storage cluster and / or the performance parameter of each storage cluster comprises: Obtain a first probability distribution and a second probability distribution, wherein the first probability distribution is used to characterize the number of data processing requests sent by each client, and the second probability distribution is used to characterize the number of data processing requests corresponding to each storage cluster; Determine a capability matrix based on the distance between each client and each storage cluster and / or the performance parameter of each storage cluster; the capability matrix is ​​used to characterize the capability of each storage cluster to process the data processing request of each client; Based on the first probability distribution, the second probability distribution and the capability matrix, a relationship matrix is ​​obtained; the relationship matrix is ​​used to represent the corresponding relationship between the client and the storage cluster; Based on the relationship matrix, the at least one storage cluster is determined from the plurality of storage clusters.

4. The method according to claim 3, characterized in that The determining of the capability matrix based on the distance between each client and each storage cluster and / or the performance parameter of each storage cluster includes: The sum of the target distance and the target performance parameter is determined as the capability matrix; the target distance includes the distance between each client and each storage cluster; the target performance parameter includes the performance parameter of each storage cluster; or, determining the target distance as the capability matrix; or, The target performance parameters are determined as the capability matrix.

5. The method according to claim 1, characterized in that The number of the clients is multiple; the number of the data processing requests is multiple; the method further includes: Obtain a first probability distribution and a second probability distribution, wherein the first probability distribution is used to characterize the number of data processing requests sent by each of the multiple clients, and the second probability distribution is used to characterize the number of data processing requests processed by each storage cluster in the multiple storage clusters; Based on the first probability distribution and the second probability distribution, a relationship matrix is ​​obtained; the relationship matrix is ​​used to represent the corresponding relationship between the client and the storage cluster; Based on the relationship matrix, the at least one storage cluster is determined from the plurality of storage clusters.

6. The method according to claim 1, characterized in that The performance parameter includes at least one of the following: latency, disk utilization, memory utilization, bandwidth utilization, and processor utilization.

7. The method according to claim 1, characterized in that The multiple storage clusters correspond to one domain name.

8. A data processing device, characterized in that: The data processing device comprises: a transmission unit and a processing unit; The transmission unit is used to receive a data processing request from a client; The processing unit is configured to determine at least one storage cluster from the multiple storage clusters based on a location parameter set and / or a performance parameter of each storage cluster in the multiple storage clusters, the location parameter set including the location parameter of the client and the location parameter of each storage cluster, the data processing capability of the at least one storage cluster being higher than the data processing capability of other storage clusters in the multiple storage clusters; The processing unit is further configured to determine the at least one storage cluster as a processing cluster for the data processing request.

9. An electronic device, characterized in that: include: Processor and memory; The memory is used to store one or more programs, and the one or more programs include computer-executable instructions. When the electronic device is running, the processor executes the computer-executable instructions stored in the memory to enable the electronic device to perform the method described in any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1-7.

11. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 7.

Images