Data processing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to the technical field of novel information, and discloses a data processing method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: acquiring a data operation request of an object storage gateway; wherein the data operation request comprises at least one of a data addition request, a data deletion request, a data modification request and a data query request; determining a first load state corresponding to the data operation request according to the data operation request; and according to a target scheduling strategy corresponding to the first load state, performing a target data operation corresponding to the processing data operation request on the target data of the main metadata center.

Description

Technical Field

[0001] The present invention relates to the field of novel information technology, and in particular to a data processing method, device, computer equipment, storage medium and program product. Background Art

[0002] The business characteristics of object storage systems, especially the large number of concurrent accesses and low latency requirements that need to be handled, directly determine that metadata management must have high concurrent processing capabilities and low latency response characteristics. In order to effectively improve performance, object storage gateways generally use long connections. This strategy optimizes the communication efficiency between clients and servers by reducing the overhead of establishing and disconnecting connections.

[0003] However, since persistent connections allow clients to maintain continuous communication with specific servers, this may cause some servers (or computing nodes) to be overloaded due to processing too many requests, while other servers may be relatively idle, which will not only reduce the overall performance of the system, but may also cause overloaded nodes to crash due to resource exhaustion, thereby affecting the stability and reliability of the system. Summary of the invention

[0004] In view of this, the present invention provides a data processing method, apparatus, computer equipment, storage medium and program product.

[0005] In a first aspect, the present invention provides a data processing method, which includes: obtaining a data operation request from an object storage gateway; wherein the data operation request includes: at least one of a request to add data, a request to delete data, a request to modify data, and a request to query data; according to the data operation request, determining a first load state corresponding to the data operation request; and performing a target data operation corresponding to the data operation request on the target data of a primary metadata center according to a target scheduling strategy corresponding to the first load state.

[0006] The data processing method provided in this embodiment can directly reflect the processing pressure of the node because the number and type of data operation requests (such as adding, deleting, modifying, and querying) can directly reflect the processing pressure of the node. By obtaining the data operation requests, the system can monitor the load status of each node (or server) in real time or approximately in real time. At the same time, according to different load states, the target data operation corresponding to the data operation request can be processed on the target data of the main metadata center through the target scheduling strategy corresponding to the load state, which can effectively balance the load of each node and avoid the situation where some nodes are overloaded while other nodes are idle.

[0007] In one possible implementation, target data operations corresponding to data operation requests are processed on target data of the primary metadata center according to a target scheduling strategy corresponding to the first load state, including: when the first load state is the first target load state, establishing a short connection between the object storage gateway and the primary metadata center; and executing the target data operations corresponding to the data operation requests through the short connection.

[0008] The data processing method provided in this embodiment performs target data operations through short connections in a first load state, and closes the connection immediately after the target data operation is completed, thereby reducing the connection management burden on the server side and effectively balancing the load of each node.

[0009] In one possible implementation, when the number of data operation requests is multiple, the target data operations corresponding to the data operation requests are processed on the target data of the main metadata center according to the target scheduling strategy corresponding to the first load state, including: when the first load state is the second target load state, determining the first target connection corresponding to each data operation request from the connection pool; disconnecting the first target connection, and reconnecting the first target connection, so that some data operation requests execute the target data operations corresponding to each data operation request in the partial data operation requests through the connections of other nodes; and executing the target data operations corresponding to another part of the data operation requests through the first target connection.

[0010] The data processing method provided in this embodiment uses the connection pool technology to pre-create and maintain a certain number of database connections, which are reused when needed, avoiding resource consumption and performance overhead caused by frequent creation and destruction of connections. At the same time, by disconnecting and reconnecting the first target connection, necessary cleanup or reset of the connection can be performed without releasing the connection resources, and the introduction of a load balancing mechanism during the reconnection process can dynamically allocate requests to different metadata center computing nodes, thereby achieving load balancing and avoiding overload of a single node.

[0011] In one possible implementation, target data operations corresponding to data operation requests are processed on target data of the main metadata center according to the target scheduling strategy corresponding to the first load state, including: when the first load state is the third target load state, determining a second target connection corresponding to the data operation request from the connection pool; executing the target data operation corresponding to the data operation request through the second target connection; wherein the second target connection is a long connection.

[0012] The data processing method provided in this embodiment uses a long connection, which means that the connection is not closed immediately after the data transmission is completed, but remains open for subsequent data transmission. This avoids the overhead of re-establishing the connection for each data transmission, thereby improving the efficiency of data transmission.

[0013] In one possible implementation, the target data operation corresponding to the data operation request is processed on the target data of the primary metadata center according to the target scheduling strategy corresponding to the first load state, including: detecting whether the first current load of the primary metadata center exceeds the preset load; if the first current load of the primary metadata center does not exceed the preset load; determining the load difference between the first current load and the preset load; detecting whether the load corresponding to the first load state exceeds the load difference; if the load corresponding to the first load state exceeds the load difference; forwarding the data operation request to the backup metadata center, and executing the target data operation corresponding to the data operation request through the third target connection of the backup metadata center.

[0014] The data processing method provided in this embodiment can detect potential overload risks in a timely manner by continuously detecting the first current load of the primary metadata center and further analyzing the load difference when the load does not exceed the preset load. Once the load corresponding to the first load state exceeds the load difference, the system can quickly forward the data operation request to the backup metadata center, thereby avoiding the collapse of the primary metadata center due to overload and improving the stability of the entire system. In addition, when the load of the primary metadata center is light, it can focus on processing high-priority or urgent data operation requests; and when the load of the primary metadata center is close to or exceeds the preset load, some requests can be forwarded to the backup metadata center to achieve optimal resource allocation and load balancing.

[0015] In one possible implementation, after forwarding the data operation request to the backup metadata center and executing the target data operation corresponding to the data operation request through the third target connection of the backup metadata center, it also includes: in the case of obtaining other data operation requests, detecting whether the second current load of the main metadata center exceeds the preset load; if the second current load of the main metadata center does not exceed the preset load, determining the load difference between the second current load and the preset load; according to the other data operation requests, determining the second load state corresponding to the other data operation requests; detecting whether the load corresponding to the second load state exceeds the load difference between the second current load and the preset load; if the load corresponding to the second load state does not exceed the load difference between the second current load and the preset load, forwarding the other data operation requests to the main metadata center, and processing the target data of the main metadata center according to the target scheduling policy corresponding to the second load state.

[0016] The data processing method provided in this embodiment can detect the first current load of the primary metadata center continuously and further analyze the load difference when the load does not exceed the preset load, so that the system can timely discover the potential overload risk. Once the load corresponding to the first load state exceeds the load difference, the system can quickly forward the data operation request to the backup metadata center, thereby avoiding the collapse of the primary metadata center due to overload and improving the stability of the entire system.

[0017] In addition, if the load corresponding to the second load state does not exceed the load difference between the second current load and the preset load, other data operation requests are forwarded to the main metadata center for processing. Prioritizing the use of the main metadata center can ensure the efficiency and real-time nature of data operations because the main metadata center usually has higher processing capabilities and more optimized resource allocation.

[0018] In a second aspect, the present invention provides a data processing device, which includes: an acquisition module, used to obtain a data operation request of an object storage gateway; wherein the data operation request includes: at least one of a request to add data, a request to delete data, a request to modify data, and a request to query data; a first determination module, used to determine a first load state corresponding to the data operation request based on the data operation request; a processing module, used to process the target data corresponding to the data operation request on the target data of the main metadata center according to a target scheduling strategy corresponding to the first load state.

[0019] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the data processing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0020] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data processing method of the first aspect or any corresponding embodiment thereof.

[0021] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the data processing method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 is a schematic diagram of the architecture of an object storage gateway and a metadata center provided according to an embodiment of the present invention;

[0024] Figure 2 is a schematic diagram of a scheduling strategy provided according to an embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of interaction between a metadata query traffic scheduling platform and a main metadata center provided according to an embodiment of the present invention;

[0026] Figure 4 It is a schematic diagram of the interaction between the metadata query traffic scheduling platform, the primary metadata center and the backup metadata center provided by an embodiment of the present invention;

[0027] Figure 5 is a flow chart of a data processing method provided according to an embodiment of the present invention;

[0028] Figure 6 is a schematic diagram of a data processing architecture provided according to an embodiment of the present invention;

[0029] Figure 7 A structural block diagram of a data processing device according to an embodiment of the present invention;

[0030] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0032] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the architecture of the object storage gateway and metadata center provided according to an embodiment of the present invention. Figure 2 , Figure 2is a schematic diagram of a scheduling strategy provided according to an embodiment of the present invention.

[0033] The object storage gateway sends the addition, deletion, and modification operations of the metadata center to the LVS of the metadata center. When the computing node needs to process the data corresponding to the addition, deletion, and modification operations, the computing node will send a data access request to the storage node. After receiving the data access request, the storage node will perform the corresponding operation according to the content corresponding to the data access request (such as data reading, writing, or modification), and return the result to the computing node. Among them, the data processing system dynamically allocates and schedules the resources of the computing nodes and storage nodes according to the system load to ensure the efficient operation of the system.

[0034] As an example, the object storage gateway sends the metadata center's addition, deletion, modification and query to the metadata center's LVS, selects the minimum connection algorithm (lc), and ensures that the number of connections for each computing node is the same. The connections are divided into three types: the first is high-load query requests, for which short connections are used for management; the second is medium-load query requests, for which connection pools + connection timed disconnection strategies are used; the third is low-load query requests, for which connection pools + long connection strategies are used.

[0035] Using long connections can reduce network overhead and improve communication efficiency. However, if a connection happens to carry a medium-load query request (multiple consecutive medium-load queries can complete the request), it will cause an unbalanced load on the computing nodes of the entire metadata center. Therefore, for medium-load query requests, a connection pool + timed disconnection and reconnection strategy is adopted. During the reconnection process, there is a certain probability that the load will be transferred to other metadata center computing nodes.

[0036] Please refer to Figure 3 , Figure 3 Schematic diagram of the interaction between the metadata query traffic scheduling platform and the main metadata center provided by an embodiment of the present invention. Figure 4 , Figure 4 It is a schematic diagram of the interaction between the metadata query traffic scheduling platform, the primary metadata center and the backup metadata center provided according to an embodiment of the present invention.

[0037] Combination Figure 3 and Figure 4It can be seen that this embodiment adjusts the query load of the main metadata center through a negative feedback device. By designing a negative feedback device to adjust the query load of the main metadata center, when the load of the computing node of the main metadata center exceeds the set threshold, part of the query requests are gradually switched to the backup pool object storage metadata gateway for negative feedback adjustment until the load of the computing node of the main metadata center is maintained within the preset range of the expected value. When the load of the computing node of the main metadata center is low, the system will switch the query traffic switched to the backup pool object storage metadata gateway back to the main metadata center through the metadata query traffic scheduling platform. Since the backup metadata center is shared by multiple resource pools, the computing and storage resources of the backup metadata center can be fully utilized in this way. The controller is used to calculate the mapping relationship between the load difference and the adjustment amount. The role of the metadata query traffic scheduling platform is to forward query requests to different metadata center computing nodes, thereby avoiding overload of the main pool metadata center.

[0038] Metadata is replicated across regions to maintain near-real-time synchronization between the backup metadata center and the primary metadata center. Since the public network data transmission channel does not require an additional whitelist of database backup ports, security risks are reduced. At the same time, since metadata backup uses asynchronous replication, when network instability causes backup exceptions, it will not affect the normal business processes of the primary resource pool.

[0039] The object storage metadata gateway only carries out the duties of adding, deleting, modifying and checking metadata, and does not perform actual data operations. It is equivalent to a streamlined gateway, through which the metadata of the backup pool is maintained. When the primary resource pool successfully processes a request, it will issue an event notification. The metadata cross-region replication module will send an HTTP request to the backup pool object storage metadata gateway based on the event type. The backup pool object storage metadata gateway will parse the request and convert it into the corresponding metadata operation to update the backup metadata center. In order to improve resource utilization, the primary metadata center and the backup metadata center can perform resource planning in a many-to-one manner.

[0040] According to an embodiment of the present invention, a data processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] In this embodiment, a data processing method is provided, which can be used in computer equipment, such as computers, servers, etc. Figure 5 is a schematic diagram of a data processing flow according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps:

[0042] Step S501, obtaining a data operation request of an object storage gateway; wherein the data operation request includes: at least one of a request to add data, a request to delete data, a request to modify data, and a request to query data.

[0043] The object storage gateway is used to represent the middleware or service between the client (such as application, user, etc.) and the object storage service, and is used to process data operation requests from the client and forward these requests to the back-end object storage system; the object storage gateway provides the S3 protocol to the outside world, and multiple storage clusters can be managed simultaneously through an object storage gateway. Data operation requests are used to represent requests for operations on data stored in the object storage system, including adding data (such as uploading files), deleting data (such as deleting files), modifying data (such as updating file content or metadata), and querying data (such as downloading files or obtaining file metadata). Specifically, the object storage gateway listens for requests from the client, and these requests are sent to the object storage gateway via network protocols (such as HTTP / HTTPS). After receiving these requests, the object storage gateway will parse the request content, including the data object to be operated, the operation type (add, delete, modify, query), etc.

[0044] Step S502: Determine a first load state corresponding to the data operation request according to the data operation request.

[0045] The first load state is used to characterize the load of the object storage gateway or its backend object storage system in processing data operation requests, including the current state of indicators such as CPU usage, memory usage, network bandwidth, and disk I / O. Specifically, the object storage gateway evaluates the current system load to determine how to optimally process the received data operation request. Specifically, the first load state can be determined by monitoring the real-time performance indicators of the system and calculating a load state value based on these indicators.

[0046] The target scheduling strategy can be used to represent one or more strategies for processing data operation requests selected according to the load status of the system, such as load balancing, data migration, fault recovery, etc.

[0047] As an example, a mapping table may be established based on the relationship between the scheduling strategy and the load state. After the first load state is determined, the target scheduling strategy corresponding to the first load state may be determined by querying the mapping table.

[0048] Step S503: performing a target data operation corresponding to the data operation request on the target data of the main metadata center according to the target scheduling strategy corresponding to the first load state.

[0049] The main metadata center can represent the main and core metadata center among all metadata centers. The target data can represent the data corresponding to the data operation request of the object storage gateway. For example, if a deletion operation is performed on data K, the data operation request is a deletion operation request, and the data corresponding to the deletion operation request is data K. According to different first load states, different task scheduling strategies can be selected to process the target data of the main metadata center and the target data operation corresponding to the data operation request.

[0050] As an example, if the first load state is a high load state, a short connection task scheduling strategy may be adopted, that is, after the data operation request corresponding to the high load state is processed, the short connection is released.

[0051] As an example, if the first load state is a low load state, a long connection task scheduling strategy can be adopted, that is, multiple connections are preset in the connection pool, and after receiving a data operation request, the connection corresponding to the data operation request is matched to process the target data operation corresponding to the data operation request.

[0052] The data processing method provided in this embodiment can directly reflect the processing pressure of the node because the number and type of data operation requests (such as adding, deleting, modifying, and querying) can directly reflect the processing pressure of the node. By obtaining the data operation requests, the system can monitor the load status of each node (or server) in real time or approximately in real time. At the same time, according to different load states, the target data operation corresponding to the data operation request can be processed on the target data of the main metadata center through the target scheduling strategy corresponding to the load state, which can effectively balance the load of each node and avoid the situation where some nodes are overloaded while other nodes are idle.

[0053] In a possible implementation, the above step S503 includes:

[0054] Step a1: when the first load state is the first target load state, a short connection is established between the object storage gateway and the primary metadata center.

[0055] The first target load state may be a high load state, that is, the data operation request corresponding to the first load state needs to process a large amount of data, which requires more system resources. A short connection may represent a fast, direct, and short-duration communication connection for efficiently transmitting data or performing specific tasks.

[0056] Specifically, when the first load state is the first target load state, a short connection is established between the object storage gateway and the primary metadata center.

[0057] As an example, determine the communication requirements between the object storage gateway and the main metadata center based on the data operation request, including data transfer volume, transmission frequency, data type, etc. Evaluate the object storage gateway and the main metadata center to ensure that they have the hardware and software conditions required to establish short connections. Determine the network topology, determine the routing path of the short connection, bandwidth allocation, etc.

[0058] Configure relevant network parameters on the object storage gateway, such as IP address, port number, protocol type, etc. Configure the gateway's security policy as needed, such as firewall rules, access control list (ACL), etc. Configure the corresponding receiving end on the primary metadata center to ensure that it can receive data transmission from the gateway. Configure the network parameters of the data center to ensure that they match the network parameters of the gateway. Configure the necessary authentication and authorization mechanisms to ensure that the communication between the gateway and the data center is secure. Specifically, this may include the use of security credentials such as keys, certificates, and tokens for identity authentication and authorization. The gateway initiates a connection request to the data center, including sending a connection request message and waiting for the data center's response. After receiving the connection request, the data center verifies and returns a response message. A handshake protocol may be required between the gateway and the data center to ensure that both parties can communicate normally. The handshake protocol usually includes steps such as exchanging security parameters, verifying identity, and negotiating encryption algorithms. After successfully completing the handshake protocol, a short connection is established between the gateway and the data center.

[0059] Step a2: Execute the target data operation corresponding to the data operation request through the short connection.

[0060] The object storage gateway receives data operation requests from the client. The data operation request contains the type of data operation to be performed (such as read, write, update or delete, etc.), and may also include related data identifiers (such as file name, data ID, etc.). The object storage gateway parses the received data operation request to determine the specific operation to be performed and the required data. Through the established short connection, the object storage gateway sends data operation requests to the primary metadata center or receives data operation requests from the primary metadata center. In the primary metadata center, the corresponding data operation is performed according to the received data operation request and data.

[0061] The data processing method provided in this embodiment performs the target data operation through a short connection in the first load state, and closes the connection immediately after the target data operation is completed, thereby reducing the connection management burden on the server side, thereby effectively balancing the load of each node.

[0062] In a possible implementation, when there are multiple data operation requests, step S503 includes:

[0063] Step b1: when the first load state is the second target load state, determining a first target connection corresponding to each data operation request from the connection pool.

[0064] The first load state may be continuously monitored, and specifically may include key indicators such as CPU usage, memory usage, network bandwidth, I / O operations, etc. When the key indicators such as CPU usage, memory usage, network bandwidth, I / O operations, etc. reach or exceed the key indicators corresponding to the second target load state, the first target connection corresponding to each data operation request is determined from the connection pool.

[0065] As an example, check the existing connection pool status, including the number of available connections, connection usage, connection response time, etc., classify multiple received data operation requests, such as read requests, write requests, update requests, etc., and assign a first target connection to each data operation request based on the connection pool status and the classification and priority of the request.

[0066] In one possible implementation, once a connection is allocated, the system will continuously monitor the status and performance of the connection. If it is found that the connection performance is degraded or a failure occurs, the system will attempt to reallocate the connection or take other recovery measures.

[0067] Step b2, disconnecting the first target connection, and reconnecting the first target connection, so that the partial data operation request executes the target data operation corresponding to each data operation request in the partial data operation request through the connection of other nodes.

[0068] Step b3: Execute the target data operation corresponding to the other part of the data operation request through the first target connection.

[0069] It is monitored that the first load state reaches the second target load state, which indicates that the current load is high and connections may need to be redistributed to optimize performance. According to the current load and connection usage, a portion of the first target connections are disconnected. The first target connection may be a connection on a node with a high current load or a connection with a long response time. The first target connection is reconnected so that part of the first target connection can be allocated to new connections on other nodes.

[0070] As an example, a distributed database system contains multiple nodes, each of which is responsible for processing a portion of data operation requests. These nodes are connected through a high-speed network and are all connected to a common load balancer. The task of the load balancer is to distribute the data operation requests entering the system to the appropriate nodes for execution. The load status of each node is continuously monitored, including key indicators such as CPU usage, memory usage, and disk I / O. When the load of a node reaches a preset threshold (i.e., the second target load status), the system believes that the node may be about to be overloaded and measures need to be taken to optimize performance. Assuming that the current load of node A is very high and has reached the second target load status, the system decides to disconnect some of the first target connections associated with node A. These connections may be connections that are currently processing a large number of data operation requests. The load balancer sends a signal to node A to request it to disconnect these connections. Before disconnecting, node A will ensure that the ongoing operations can be safely terminated or transferred to other connections (for example, through transaction rollback or data persistence). After disconnecting, the load balancer redistributes these connection requests to other nodes with lower loads (such as node B and node C). These nodes will establish new connections to handle these requests.

[0071] The data processing method provided in this embodiment uses the connection pool technology to pre-create and maintain a certain number of database connections, which are reused when needed, avoiding resource consumption and performance overhead caused by frequent creation and destruction of connections. At the same time, by disconnecting and reconnecting the first target connection, necessary cleanup or reset of the connection can be performed without releasing the connection resources, and the introduction of a load balancing mechanism during the reconnection process can dynamically allocate requests to different metadata center computing nodes, thereby achieving load balancing and avoiding overload of a single node.

[0072] In a possible implementation, the above step S503 includes:

[0073] Step c1: when the first load state is the third target load state, determining a second target connection corresponding to the data operation request from the connection pool.

[0074] The third target load state is used to represent a low load state, that is, the connection corresponding to the data operation request can continue to exist, so the second target connection corresponding to the data operation request can be determined from the connection pool. The second target connection is a long connection.

[0075] Step c2: Execute the target data operation corresponding to the data operation request through the second target connection; wherein the second target connection is a long connection.

[0076] A data operation request is received from the client, which contains detailed information about the target data operation to be performed. Based on the characteristics of the request (such as data type, operation type, priority, etc.) and the current state of the connection pool, the server selects a suitable long connection from the connection pool as the second target connection. The server sends the data operation request to the target database or data storage system through the second target connection and waits for the execution result. The target database or data storage system performs the corresponding data operation (such as read, write, update or delete) according to the request. Once the data operation is completed, the target database or data storage system returns the execution result to the server through the second target connection. The server then processes the result and returns it to the client.

[0077] The data processing method provided in this embodiment uses a long connection, which means that the connection is not closed immediately after the data transmission is completed, but remains open for subsequent data transmission. This avoids the overhead of re-establishing the connection for each data transmission, thereby improving the efficiency of data transmission.

[0078] In a possible implementation, the above step S503 includes:

[0079] Step d1, detecting whether a first current load of the main metadata center exceeds a preset load.

[0080] Step d2: if the first current load of the main metadata center does not exceed the preset load, determine the load difference between the first current load and the preset load.

[0081] Step d3, detecting whether the load corresponding to the first load state exceeds the load difference.

[0082] Step d4: if the load corresponding to the first load state exceeds the load difference.

[0083] The preset load can be used to characterize the maximum load bearing capacity of the computing nodes of the main metadata center. Among them, when the load of the computing nodes of the main metadata center exceeds the preset load, it may cause the main metadata center to crash. Specifically, after receiving a data operation request, it can first be determined whether the first current load of the main metadata center exceeds the preset load. That is, before receiving the data operation request, the main metadata center has performed some other tasks, and it is necessary to first determine whether the first current load of the main metadata center exceeds the preset load to avoid the load of the computing nodes of the main metadata center exceeding the preset load.

[0084] The load difference may be the difference between the first current load and the preset load, reflecting the relative distance between the current load of the main metadata center and the preset threshold. If the first current load of the main metadata center does not exceed the preset load, it is necessary to detect whether the load corresponding to the first load state exceeds the load difference; wherein, when the load corresponding to the first load state exceeds the load difference, the computing node representing the main metadata center executes the data operation request corresponding to the first load state, which will cause coefficient collapse.

[0085] Step d5: forwarding the data operation request to the standby metadata center, and executing the target data operation corresponding to the data operation request through the third target connection of the standby metadata center.

[0086] The data operation request is forwarded to the standby metadata center. The metadata cross-region replication module can send an HTTP request to the standby pool object storage metadata gateway according to the event type. The standby pool object storage metadata gateway will parse the request and convert it into the corresponding metadata operation to update the standby metadata center.

[0087] The data processing method provided in this embodiment can detect potential overload risks in a timely manner by continuously detecting the first current load of the primary metadata center and further analyzing the load difference when the load does not exceed the preset load. Once the load corresponding to the first load state exceeds the load difference, the system can quickly forward the data operation request to the backup metadata center, thereby avoiding the collapse of the primary metadata center due to overload and improving the stability of the entire system. In addition, when the load of the primary metadata center is light, it can focus on processing high-priority or urgent data operation requests; and when the load of the primary metadata center is close to or exceeds the preset load, some requests can be forwarded to the backup metadata center to achieve optimal resource allocation and load balancing.

[0088] In a possible implementation, after step d6, the method further includes:

[0089] Step e1, in the case of obtaining other data operation requests, detecting whether the second current load of the main metadata center exceeds the preset load.

[0090] When the above-mentioned data operation requests are executed through the central node of the backup metadata center, the client can still issue new data operation requests (other data operation requests). When other data operation requests are issued, it is first necessary to detect whether the second current load of the primary metadata center exceeds the preset load, that is, whether the primary metadata center has executed some requests and reduced the load.

[0091] Step e2: if the second current load of the main metadata center does not exceed the preset load, determine the load difference between the second current load and the preset load.

[0092] Step e3: determining a second load state corresponding to the other data operation request according to the other data operation request.

[0093] Step e4, detecting whether the load corresponding to the second load state exceeds the load difference between the second current load and the preset load.

[0094] Step e5, if the load corresponding to the second load state does not exceed the load difference between the second current load and the preset load, other data operation requests are forwarded to the main metadata center, and the target data of the main metadata center is processed according to the target scheduling strategy corresponding to the second load state.

[0095] If the second current load of the main metadata center does not exceed the preset load, it is necessary to determine the second load state corresponding to the other data operation requests based on other data operation requests, and then detect whether the load corresponding to the second load state exceeds the load difference between the second current load and the preset load, that is, to determine whether the main metadata center can execute other data operation requests. If the load corresponding to the second load state does not exceed the load difference between the second current load and the preset load, it indicates that the main metadata center executes other data operation requests. Then, the query traffic switched to the backup pool object storage metadata gateway can be switched back to the main metadata center through the metadata query traffic scheduling platform, and the target data of the main metadata center can be processed by the main metadata center according to the target scheduling policy corresponding to the second load state.

[0096] Please refer to Figure 6 , Figure 6 is a schematic diagram of a data processing architecture provided according to an embodiment of the present invention.

[0097] Figure 6 In the system, the primary metadata center and the backup metadata center can plan resources in a many-to-one manner, that is, one backup metadata center can correspond to multiple primary metadata centers. The object storage gateway of each resource pool interacts with the metadata center of this resource pool. At the same time, the same metadata is maintained in the backup metadata center through metadata cross-region replication. The metadata of the backup metadata center has two functions. The first is to carry additional high-load query requests when the load of the primary metadata center is high; the second function is that when the primary metadata center crashes, the object storage gateway can directly cut the metadata traffic to the backup metadata center to ensure the availability of the system.

[0098] The data processing method provided in this embodiment can detect the first current load of the primary metadata center continuously and further analyze the load difference when the load does not exceed the preset load, so that the system can timely discover the potential overload risk. Once the load corresponding to the first load state exceeds the load difference, the system can quickly forward the data operation request to the backup metadata center, thereby avoiding the collapse of the primary metadata center due to overload and improving the stability of the entire system.

[0099] In addition, if the load corresponding to the second load state does not exceed the load difference between the second current load and the preset load, other data operation requests are forwarded to the main metadata center for processing. Prioritizing the use of the main metadata center can ensure the efficiency and real-time nature of data operations because the main metadata center usually has higher processing capabilities and more optimized resource allocation.

[0100] In the present embodiment, a data processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0101] This embodiment provides a data processing device, such as Figure 7 As shown, it includes: an acquisition module 701, which is used to obtain a data operation request of an object storage gateway; wherein the data operation request includes: at least one of a request to add data, a request to delete data, a request to modify data, and a request to query data; a first determination module 702, which is used to determine a first load state corresponding to the data operation request according to the data operation request; and a processing module 703, which is used to process the target data corresponding to the data operation request on the target data of the main metadata center according to a target scheduling strategy corresponding to the first load state.

[0102] In one possible implementation, the processing module 703 includes: an establishment unit, used to establish a short connection between the object storage gateway and the primary metadata center when the first load state is a first target load state; and a first execution unit, used to execute the target data operation corresponding to the data operation request through the short connection.

[0103] In one possible implementation, when the number of data operation requests is multiple, the above-mentioned processing module 603 includes: a first determination unit, used to determine the first target connection corresponding to each data operation request from the connection pool when the first load state is the second target load state; a disconnection unit, used to disconnect the first target connection, and reconnect the first target connection so that some data operation requests can execute the target data operation corresponding to each data operation request in the partial data operation requests through the connection of other nodes; a second execution unit, used to execute the target data operation corresponding to another partial data operation request through the first target connection.

[0104] In one possible implementation, the processing module 703 includes: a second determination unit, used to determine a second target connection corresponding to a data operation request from a connection pool when the first load state is a third target load state; a third execution unit, used to execute a target data operation corresponding to the data operation request through the second target connection; wherein the second target connection is a long connection.

[0105] In one possible implementation, the processing module 703 includes: a first detection unit, used to detect whether the first current load of the main metadata center exceeds the preset load; a third determination unit, used to determine the load difference between the first current load and the preset load if the first current load of the main metadata center does not exceed the preset load; a second detection unit, used to detect whether the load corresponding to the first load state exceeds the load difference; and a fourth execution unit, used to forward the data operation request to the backup metadata center if the load corresponding to the first load state exceeds the load difference, and execute the target data operation corresponding to the data operation request through the third target connection of the backup metadata center.

[0106] In one possible implementation, the device further includes: a first detection module, used to detect whether the second current load of the main metadata center exceeds the preset load when obtaining other data operation requests; a second determination module, used to determine the load difference between the second current load and the preset load if the second current load of the main metadata center does not exceed the preset load, and determine the second load state corresponding to the other data operation request according to the other data operation request; the second detection module, used to detect whether the load corresponding to the second load state exceeds the load difference between the second current load and the preset load; and an execution module, used to forward the other data operation requests to the main metadata center if the load corresponding to the second load state does not exceed the load difference between the second current load and the preset load, and process the target data of the main metadata center according to the target scheduling strategy corresponding to the second load state.

[0107] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0108] The data processing device in this embodiment is presented in the form of a functional unit, where the functional unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0109] The embodiment of the present invention also provides a computer device having the above Figure 7 The data processing device shown.

[0110] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0111] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0112] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0113] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0114] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0115] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0116] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0117] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0118] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a data operation request of an object storage gateway; wherein the data operation request includes: at least one of a request to add data, a request to delete data, a request to modify data, and a request to query data; Determining, according to the data operation request, a first load state corresponding to the data operation request; The target data operation corresponding to the data operation request is performed on the target data of the main metadata center according to the target scheduling policy corresponding to the first load state.

2. The data processing method according to claim 1, characterized in that: The performing a target data operation corresponding to the data operation request on the target data of the primary metadata center according to the target scheduling strategy corresponding to the first load state includes: When the first load state is a first target load state, establishing a short connection between the object storage gateway and the primary metadata center; The target data operation corresponding to the data operation request is executed through the short connection.

3. The data processing method according to claim 1, characterized in that: When the number of the data operation requests is multiple, performing the target data operation corresponding to the data operation request on the target data of the primary metadata center according to the target scheduling strategy corresponding to the first load state includes: When the first load state is the second target load state, determining a first target connection corresponding to each data operation request from a connection pool; Disconnecting the first target connection, and reconnecting the first target connection, so that the partial data operation request executes the target data operation corresponding to each data operation request in the partial data operation request through the connection of other nodes; Target data operations corresponding to another part of the data operation requests are performed through the first target connection.

4. The data processing method according to claim 1, characterized in that: The performing a target data operation corresponding to the data operation request on the target data of the primary metadata center according to the target scheduling strategy corresponding to the first load state includes: When the first load state is a third target load state, determining a second target connection corresponding to the data operation request from a connection pool; The target data operation corresponding to the data operation request is performed through the second target connection; wherein the second target connection is a long connection.

5. The data processing method according to claim 1, characterized in that: The performing a target data operation corresponding to the data operation request on the target data of the primary metadata center according to the target scheduling strategy corresponding to the first load state includes: Detecting whether a first current load of the main metadata center exceeds a preset load; If the first current load of the master metadata center does not exceed the preset load, determining a load difference between the first current load and the preset load; detecting whether the load corresponding to the first load state exceeds the load difference; If the load corresponding to the first load state exceeds the load difference, the data operation request is forwarded to the standby metadata center, and the target data operation corresponding to the data operation request is executed through the third target connection of the standby metadata center.

6. The data processing method according to claim 5, characterized in that: After forwarding the data operation request to the standby metadata center and performing the target data operation corresponding to the data operation request through the third target connection of the standby metadata center, the method further includes: In case of obtaining other data operation requests, detecting whether a second current load of the primary metadata center exceeds a preset load; If the second current load of the main metadata center does not exceed the preset load, determining a load difference between the second current load and the preset load, and determining a second load state corresponding to the other data operation request according to the other data operation request; detecting whether the load corresponding to the second load state exceeds a load difference between the second current load and a preset load; If the load corresponding to the second load state does not exceed the load difference between the second current load and the preset load, the other data operation requests are forwarded to the main metadata center, and the target data of the main metadata center is processed according to the target scheduling policy corresponding to the second load state.

7. A data processing device, characterized in that: The device comprises: An acquisition module, used to acquire a data operation request of an object storage gateway; wherein the data operation request includes: at least one of a request to add data, a request to delete data, a request to modify data, and a request to query data; A first determining module, configured to determine, according to the data operation request, a first load state corresponding to the data operation request; A processing module is used to perform a target data operation corresponding to the processing data operation request on the target data of the main metadata center according to the target scheduling strategy corresponding to the first load state.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data processing method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data processing method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the data processing method according to any one of claims 1 to 6.

Images