Data processing method and device, electronic equipment and storage medium

Abstract

This disclosure provides a data processing method, apparatus, electronic device, and storage medium, which can be used in the field of data processing technology or the financial field. The method includes: determining a storage node corresponding to an operation request obtained from a client, the storage node including a master node and M slave nodes, where M is a positive integer; processing the operation request and waiting instruction request based on the real-time availability status of the M slave nodes, generating a merge request, wherein the real-time availability status of the M slave nodes is obtained in real time through a monitoring component communicating with a registry center; establishing a dynamic connection relationship between an agent and the storage node based on the merge request, wherein the dynamic connection relationship is terminated when preset conditions are met; and sending the merge request to the storage node based on the dynamic connection relationship, wherein the agent receives and forwards a response result generated based on the execution result of the storage node, wherein the execution result is generated by the master node and slave nodes of the storage node according to the merge request.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and specifically to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of the internet, to ensure high availability during data operations, data synchronization is necessary. In data structure servers, the master and slave nodes use asynchronous replication for data synchronization. When hardware failures or network interruptions occur in the storage layer, the master node triggers a high-availability switch, and a slave node becomes the master, replacing the faulty one. However, in current technologies, data between master and slave nodes is replicated asynchronously. During asynchronous replication, it's impossible to guarantee complete data synchronization between master and slave nodes. Consequently, in scenarios like flash sales or limited-time offers, if a master-slave switch occurs before complete synchronization, it can lead to lost operation requests and data errors. Summary of the Invention

[0003] In view of this, the present disclosure provides a data processing method, apparatus, electronic device and storage medium, which at least partially solves the problem of data loss caused by master-slave node switching, can ensure data consistency after master-slave node switching, and improve the high availability of storage nodes.

[0004] The first aspect of this disclosure provides a data processing method, comprising: determining a storage node corresponding to an operation request obtained from a client, the storage node including a master node and M slave nodes, where M is a positive integer; processing the operation request and a waiting instruction request based on the real-time availability status of the M slave nodes to generate a merge request, wherein the real-time availability status of the M slave nodes is obtained in real time through a monitoring component communicating with a registry center; establishing a dynamic connection between an agent and the storage node based on the merge request, wherein the dynamic connection is released when a preset condition is met; sending the merge request to the storage node based on the dynamic connection, wherein the agent receives and forwards a response result generated based on the execution result of the storage node, wherein the execution result is generated by the master node and slave nodes of the storage node according to the merge request.

[0005] In some exemplary embodiments of this disclosure, the method further includes: before processing the operation request and the waiting instruction request based on the real-time availability status of the M slave nodes to generate a merge request, obtaining from the registry the availability status of each of the M slave nodes as obtained in real time by the monitoring component.

[0006] In some exemplary embodiments of this disclosure, the method further includes: obtaining an operation request from the client before determining the storage node corresponding to the operation request based on the operation request obtained from the client; and storing client information and request data associated with the operation request into a target queue after receiving the operation request obtained from the client.

[0007] In some exemplary embodiments of this disclosure, determining the storage node corresponding to the operation request based on the operation request obtained from the client includes: parsing the operation request obtained from the client according to a preset algorithm to determine target data; and determining the storage node corresponding to the operation request based on the target data.

[0008] In some exemplary embodiments of this disclosure, the step of processing the operation request and the waiting instruction request based on the real-time availability status of the M slave nodes and generating a merge request includes: counting the number of available slave nodes based on the availability status of each of the M slave nodes; and processing the operation request and the waiting instruction request to generate a merge request when the number of available slave nodes is greater than a set threshold.

[0009] In some exemplary embodiments of this disclosure, the preset conditions include: the time taken to establish the dynamic connection relationship exceeds a set time; and the slave node generates an execution result based on the merge request.

[0010] In some exemplary embodiments of this disclosure, the method further includes: deleting the request queue associated with the dynamic connection relationship when the dynamic connection relationship is terminated, and releasing the stored memory information associated with the dynamic connection relationship.

[0011] In some exemplary embodiments of this disclosure, the proxy receiving and forwarding a response result generated based on the execution result of the storage node includes: the proxy receiving a first execution result, which is generated by the master node according to the operation request in the merge request; the proxy receiving a second execution result, which is generated by the slave node synchronizing the data of the master node to M slave nodes according to the wait instruction request in the merge request; and the proxy generating a response result based on the received first and second execution results and forwarding it to the client.

[0012] A second aspect of this disclosure provides a data processing apparatus, comprising: a determining module configured to determine a storage node corresponding to an operation request obtained from a client, the storage node including a master node and M slave nodes, where M is a positive integer; a generating module configured to process the operation request and a waiting instruction request based on the real-time availability status of the M slave nodes, generating a merging request, the real-time availability status of the M slave nodes being obtained in real time through a monitoring component communicating with a registry center; a processing module configured to establish a dynamic connection relationship between an agent and the storage node based on the merging request, the dynamic connection relationship being terminated when a preset condition is met; and a receiving and forwarding module configured to send the merging request to the storage node based on the dynamic connection relationship, the agent receiving and forwarding a response result generated based on the execution result of the storage node, the execution result being generated by the master node and slave nodes of the storage node based on the merging request.

[0013] In some exemplary embodiments of this disclosure, the data processing apparatus further includes an acquisition module, which is configured to acquire, from the registry center, the availability status of each of the M slave nodes as acquired in real time by the monitoring component, before processing the operation request and the waiting instruction request based on the real-time availability status of the M slave nodes to generate a merge request.

[0014] In some exemplary embodiments of this disclosure, the data processing apparatus further includes a storage module configured to: obtain an operation request from a client before determining a storage node corresponding to the operation request based on the operation request obtained from the client; and, after receiving the operation request obtained from the client, store client information and request data associated with the operation request in a target queue.

[0015] In some exemplary embodiments of this disclosure, the determining module includes a determining submodule, which is configured to: parse the operation request obtained from the client according to a preset algorithm to determine target data; and determine the storage node corresponding to the operation request according to the target data.

[0016] In some exemplary embodiments of this disclosure, the generation module includes a generation submodule, which is configured to: count the number of available slave nodes based on the available status of each of the M slave nodes; and when the number of available slave nodes is greater than a set threshold, process the operation request and the waiting instruction request to generate a merge request.

[0017] In some exemplary embodiments of this disclosure, the data processing apparatus further includes a deletion and release module, which is configured to: delete the request queue associated with the dynamic connection relationship and release the stored memory information associated with the dynamic connection relationship when the dynamic connection relationship is terminated.

[0018] In some exemplary embodiments of this disclosure, the receiving and forwarding module further includes a receiving and forwarding submodule, which is configured as follows: the proxy receives a first execution result, which is generated by the master node according to the operation request in the merge request; the proxy receives a second execution result, which is generated by the slave node synchronizing the data of the master node to M slave nodes according to the wait instruction request in the merge request; the proxy generates a response result based on the received first and second execution results and forwards it to the client.

[0019] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a storage device for storing executable instructions that, when executed by the processor, implement the method described above.

[0020] A fourth aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon that, when executed by a processor, implement the method described above.

[0021] A fifth aspect of this disclosure provides a computer program product including a computer program that, when executed by a processor, implements the method described above.

[0022] According to embodiments of this disclosure, the data processing method can be applied to a proxy end. By processing operation requests and waiting instruction requests from clients based on the real-time availability status of the slave nodes, a merge request is generated. Based on the established dynamic connection relationship, the merge request is sent to the storage node, and a response result generated based on the execution result of the storage node is received. On one hand, since the merge request is generated based on the availability status of the slave nodes, the availability of slave nodes in the storage node can be guaranteed. This avoids the problem of failure or delay in operation requests from the master node and slave nodes due to the slave nodes being unavailable and unable to respond to waiting instruction requests. It can also reduce the network transmission resource consumption of operation requests and improve the speed of data processing. On the other hand, since the execution result is generated by the master node and slave nodes of the storage node based on the merge request, and each merge request corresponds to a dynamic connection relationship, the consistency of data between the master node and slave nodes can be guaranteed during operation. Furthermore, the use of dynamic connection relationships ensures that operation requests will not be blocked, improving the efficiency of data processing. Attached Figure Description

[0023] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0024] Figure 1 An exemplary system architecture to which data processing methods can be applied according to embodiments of this disclosure is illustrated;

[0025] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0026] Figure 3 The flowchart illustrating the data processing method according to an embodiment of the present disclosure in obtaining the availability status of a slave node is shown in the illustration.

[0027] Figure 4 A flowchart illustrating the data processing method according to an embodiment of the present disclosure in operation 400 is shown schematically.

[0028] Figure 5 A flowchart illustrating operation S210 of the data processing method according to an embodiment of the present disclosure is shown schematically.

[0029] Figure 6 The flowchart illustrating the data processing method according to an embodiment of the present disclosure in operation S220 is shown in the schematic diagram.

[0030] Figure 7 The flowchart illustrating the data processing method according to an embodiment of the present disclosure during the dynamic disconnection process is shown.

[0031] Figure 8 A flowchart illustrating the data processing method according to an embodiment of the present disclosure in operation 600 is shown schematically.

[0032] Figure 9 The illustration shows a schematic diagram of the execution process of a data processing method according to an embodiment of the present disclosure;

[0033] Figure 10 The illustration schematically shows the response result of the wait instruction request in the merge request of the data processing method according to an embodiment of the present disclosure;

[0034] Figure 11 The diagram illustrates the communication relationship between the agent, the registration center, and the monitoring component in the data processing method according to an embodiment of this disclosure.

[0035] Figure 12 A block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown schematically;

[0036] Figure 13 A block diagram of an electronic device for implementing a data processing method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0037] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0038] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0039] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0040] When using expressions such as "at least one of A, B, or C," it should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, features defined with "first" or "second" may explicitly or implicitly include one or more features.

[0041] In embodiments of this disclosure, the term "wait instruction request" refers to a request to wait for other instructions to complete. This command waits until all operation requests (write commands) from the master node of the currently connected storage node have been transmitted to a specified number of slave nodes. If data synchronization between the master and slave nodes is not completed within the timeout period, it returns 0. For example, a wait instruction request can refer to a wait request.

[0042] In the technical solution disclosed herein, the acquisition, storage, and application of user personal information comply with relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals. All operations involving the acquisition, storage, and application of user personal information in the technical solution disclosed herein have been authorized by the user.

[0043] In existing technologies, after receiving an operation request from a client, a storage node performs the operation on the master node and directly returns the response result based on the operation request to the client after the master node completes the operation. When the master node in the storage node becomes unavailable, a slave node is directly used as a backup master node to perform the function of the master node. However, due to the asynchronous replication strategy used between the master and slave nodes, there is a certain probability that the operation request will be lost when switching from the master node to the slave node. In addition, if the synchronization status of the slave node is waited for and a synchronization completion response is returned to the client after the slave node completes synchronization, the client will not be able to receive the response result if the slave node is damaged or unable to connect, which will also result in waiting time and cannot guarantee the availability of the slave node. Because of waiting for the synchronization completion response, connection resources are heavily occupied, reducing the efficiency of data transmission.

[0044] To address the aforementioned problems in related technologies, embodiments of this disclosure provide a data processing method, apparatus, electronic device, and storage medium that can be applied to a proxy terminal. This enables real-time acquisition of the availability status of slave nodes and, based on the availability status of slave nodes, sends a merge request containing operation requests and waiting instruction requests to the storage node. This allows the storage node to complete the data synchronization process between the master node and slave nodes, ensuring data consistency between the master node and slave nodes, while reducing the occupancy of connection resources and effectively improving data transmission efficiency. The data processing method disclosed herein includes: determining the storage node corresponding to the operation request based on the operation request obtained from the client, wherein the storage node includes a master node and M slave nodes, where M is a positive integer; processing the operation request and waiting instruction request based on the real-time availability status of the M slave nodes, generating a merge request, wherein the real-time availability status of the M slave nodes is obtained in real time through a monitoring component communicating with the registry center; establishing a dynamic connection relationship between the agent and the storage node based on the merge request, wherein the dynamic connection relationship is terminated when preset conditions are met; sending the merge request to the storage node based on the dynamic connection relationship, wherein the agent receives and forwards the response result generated based on the execution result of the storage node, wherein the execution result is generated by the master node and slave nodes of the storage node according to the merge request.

[0045] According to embodiments of this disclosure, the data processing method can be applied to a proxy end. By processing operation requests and waiting instruction requests from clients based on the real-time availability status of the slave nodes, a merge request is generated. Based on the established dynamic connection relationship, the merge request is sent to the storage node, and a response result generated based on the execution result of the storage node is received. On one hand, since the merge request is generated based on the availability status of the slave nodes, the availability of slave nodes in the storage node can be guaranteed. This avoids the problem of failure or delay in operation requests from the master node and slave nodes due to the slave nodes being unavailable and unable to respond to waiting instruction requests. It can also reduce the network transmission resource consumption of operation requests and improve the speed of data processing. On the other hand, since the execution result is generated by the master node and slave nodes of the storage node based on the merge request, and each merge request corresponds to a dynamic connection relationship, the consistency of data between the master node and slave nodes can be guaranteed during operation. Furthermore, the use of dynamic connection relationships ensures that operation requests will not be blocked, improving the efficiency of data processing.

[0046] Figure 1 An exemplary system architecture to which data processing methods can be applied according to embodiments of this disclosure is illustrated schematically. It should be noted that... Figure 1 The examples shown are merely examples of system architectures applicable to the embodiments of this disclosure, intended to help those skilled in the art understand the technical content of this disclosure. However, they do not imply that the embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. It should be noted that the data processing methods, apparatuses, electronic devices, and readable storage media provided in the embodiments of this disclosure can be used in the fields of data processing technology and finance, and can also be used in various fields other than finance. The data processing methods, apparatuses, electronic devices, and readable storage media provided in the embodiments of this disclosure do not limit the application fields.

[0047] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101 and 102, network 103, proxy server 104, and database server 105.

[0048] Network 103 serves as a medium for providing communication links between terminal devices 101 and 102, proxy server 104, and database server 105. Network 103 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0049] Users can use terminal devices 101 and 102 to interact with proxy server 104 and database server 105 via network 103 to receive or send data, instructions, etc. Various communication client applications can be installed on terminal devices 101 and 102, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only). Terminal devices 101 and 102 are used to retrieve data from database server 105 or send data requests to database server 105.

[0050] Terminal devices 101 and 102 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0051] Proxy server 104 can provide intermediary services between terminal devices 101, 102 and database server 105.

[0052] Database server 105 can be a server that provides various database services, such as a database server that provides data support for the websites browsed by users using terminal devices 101 and 102 (this is just an example). The database server can analyze data such as received user requests, and, according to the user's request, feed back data from the data service database server (such as web pages, information, or data obtained or generated according to the user's request) to the terminal device.

[0053] It should be noted that the data processing method provided in this embodiment can generally be executed by proxy server 104, and correspondingly, the data processing device provided in this embodiment can generally be located in proxy server 104. The data processing method provided in this embodiment can also be executed by other proxy servers different from proxy server 104 that are capable of communicating with terminal devices 101, 102 and / or database server 105. Correspondingly, the data processing device provided in this embodiment can also be located in other proxy servers different from terminal devices 101, 102 and / or database server 105.

[0054] It should be understood that Figure 1 The number of terminal devices, network and proxy servers, and database servers shown in the diagram is merely illustrative. Depending on implementation needs, any number of terminal devices, network and proxy servers, and database servers can be included.

[0055] Figure 2 A flowchart illustrating a data processing method according to an embodiment of this disclosure is shown schematically. The data processing method of this disclosure can be applied to a proxy terminal, such as a proxy module. Figure 2 As shown, the data processing method flow 200 of this embodiment includes operations S210 to S240.

[0056] In operation S210, based on the operation request obtained from the client, the storage node corresponding to the operation request is determined. The storage node includes a master node and M slave nodes, where M is a positive integer.

[0057] In the embodiments of this disclosure, the client sends a request message to the proxy, which includes an operation request, such as a write request, i.e., an operation request to write corresponding data to a storage node. After receiving the operation request from the client, the proxy processes the operation request, such as parsing it, to determine the storage node corresponding to the operation request.

[0058] For example, each operation request has a corresponding storage node. By determining the corresponding storage node, operations can be effectively performed on the master node and slave nodes within the storage node. This includes, for example, determining the availability status of the master node and slave nodes, performing operations on the master node and slave nodes according to the operation request, or obtaining the response results from the corresponding master node and slave nodes, etc.

[0059] In the embodiments of this disclosure, each storage node corresponds to a master node and one or more slave nodes that serve as backup nodes for the master node. When the master node fails (e.g., is damaged or unavailable), the slave node switches to become the master node and performs the corresponding functions in place of the master node, thereby ensuring the high availability of the system.

[0060] According to embodiments of this disclosure, the master node and M slave nodes of the storage node can switch over, ensuring high availability of the storage node when the client sends an operation request to the storage node and improving data consistency during data processing.

[0061] In operation S220, operation requests and waiting instruction requests are processed based on the real-time availability status of M slave nodes, and a merge request is generated. The real-time availability status of the M slave nodes is obtained in real time through a monitoring component that communicates with the registry center.

[0062] In the embodiments of this disclosure, the real-time availability status of M slave nodes of the storage node is obtained in real time through a monitoring component, and operation requests and waiting instruction requests are processed according to the real-time availability status of the M slave nodes.

[0063] For example, the real-time availability status of a slave node includes available and unavailable. When the number of slave nodes with a real-time availability status of available among M slave nodes is greater than or equal to 1, the operation request and the waiting instruction request are processed to generate a merge request.

[0064] If the number of available slave nodes in real time among the M slave nodes is less than 1, then the operation request and the waiting instruction request will not be processed, and the operation request will be sent directly to the master node of the storage node.

[0065] In the embodiments of this disclosure, a wait instruction request (e.g., a wait request) is used to synchronize the data written by the master node to the slave nodes. After the slave nodes have completed synchronization, the synchronization result is fed back to the agent, which then feeds back the final result to the client, allowing the client to perform further operations based on the feedback. When at least one slave node is available, a wait instruction request needs to be sent to the slave node to obtain the data synchronization result from the master node. When the number of available slave nodes is 0, it is not necessary to obtain the data synchronization result from the slave nodes; instead, the execution result of the master node of the storage node based on the operation request sent by the client is directly fed back to the client. This effectively improves the efficiency of data processing, eliminating the need for waiting.

[0066] In the embodiments of this disclosure, the wait instruction request is sent by the agent to the storage node to determine the synchronization result of the master node synchronizing data with the slave nodes in the storage node. For example, when the storage node includes at least one available slave node, the master node reports the result of the operation request to the agent based on the operation request, and the slave node synchronizes data from the master node to the slave node based on the wait instruction request. After the synchronization is completed, the slave node reports the synchronization result to the agent. This ensures data consistency between the slave node and the master node; that is, when the slave node is available, and after the slave node completes the data synchronization from the master node to the slave node based on the operation request, it reports the corresponding result to the client.

[0067] In operation S230, a dynamic connection is established between the agent and the storage node based on the merge request. The dynamic connection is released when preset conditions are met.

[0068] In the embodiments of this disclosure, when a merge request is generated, it indicates that the storage nodes include available slave nodes. When available nodes exist, it is necessary to obtain the synchronization results of the master node synchronizing data with the slave nodes. For each operation request, it is necessary to obtain the data synchronization results between the master node and the slave node corresponding to that operation request. To address this, this disclosure establishes a dynamic connection between the agent and the storage nodes. After the master node feeds back the execution result to the agent based on the client's operation request, it simultaneously feeds back the execution results of the slave nodes based on the operation request and the waiting instruction request to the agent through the same dynamic connection, i.e., it feeds back the data synchronization results between the slave nodes and the master node to the agent. After the agent obtains the execution result of the master node and the synchronization result of the slave node, it closes the dynamic connection, thereby reducing system resource consumption.

[0069] For example, the client's operation request could be a write request. When one or more of the M slave nodes are available, a merge request is generated based on the operation request (e.g., a write request) and the waiting instruction request. A dynamic connection is then established between the agent and the storage node; that is, a new connection is created for each merge request. By increasing the number of connections, concurrency is improved, achieving higher throughput. Since the new connection is dynamic, it is terminated when a preset condition is met.

[0070] In embodiments of this disclosure, the preset conditions include the time taken to establish a dynamic connection relationship exceeding a set time; and the generation of execution results from the slave node based on the merge request.

[0071] For example, the dynamic connection relationship can be periodically detected and monitored. If the time taken to establish the dynamic connection relationship exceeds the set time, it indicates that the dynamic connection between the agent and the storage node cannot be established within the set time. At this time, the establishment process of the dynamic connection relationship can be terminated, or the dynamic connection relationship can be terminated.

[0072] For example, after the slave node generates the execution result based on the merge request—that is, after the slave node synchronizes the master node's data to the slave node according to the wait instruction request in the merge request, and generates the execution result based on the synchronization result—the dynamic connection relationship is then terminated. By terminating the dynamic connection relationship, system resource consumption is reduced.

[0073] In operation S240, the merge request is sent to the storage node according to the dynamic connection relationship. The agent receives and forwards the response result generated based on the execution result of the storage node. The execution result is generated by the master node and slave node of the storage node according to the merge request.

[0074] In the embodiments of this disclosure, based on the established dynamic connection relationship, the merge request is sent to the storage node, and the agent receives the response result generated by the storage node based on the execution result of the merge request based on the dynamic connection relationship.

[0075] For example, after a merge request is sent to a storage node, the master node in the storage node generates an execution result based on the operation request in the merge request, and the slave node synchronizes the data in the master node to the slave node based on the wait instruction request, generating another execution result. The two execution results are processed to generate a response result, which is used to feed back to the client.

[0076] Figure 3 The flowchart illustrating the data processing method according to an embodiment of the present disclosure in obtaining the availability status of a slave node is shown.

[0077] like Figure 3As shown, the data processing method in this embodiment of the present disclosure further includes operation S300. In operation S300, before processing the operation request and the waiting instruction request based on the real-time availability status of the M slave nodes and generating the merge request, the availability status of each of the M slave nodes, which is obtained in real time by the monitoring component, is obtained from the registry center.

[0078] In the embodiments of this disclosure, a monitoring component monitors the availability status of slave nodes in the storage node and updates the availability status of slave nodes to the registry center in real time. Since the agent can obtain the availability status of each slave node from the registry center in real time, it can then decide whether to generate a merge request based on the availability status of the slave nodes. Furthermore, when receiving the response results, it can determine the number of execution results obtained by the slave nodes based on the merge request. For example, when there are N slave nodes (N is a positive integer less than or equal to M), if the slave nodes return N execution results, it can be considered that the process of data synchronization from the master node to the N slave nodes has been completed.

[0079] Figure 4 The flowchart of the data processing method according to an embodiment of the present disclosure in operation 400 is illustrated schematically.

[0080] like Figure 4 As shown, the data processing method of this embodiment further includes process 400, which is an operation before determining the storage node corresponding to the operation request based on the operation request obtained from the client. Process 400 includes operations S410 to S420.

[0081] In operating S410, an operation request is obtained from the client.

[0082] In embodiments of this disclosure, the operation request may be, for example, a write request, upon which the storage node writes data to the master node.

[0083] In operation S420, after receiving an operation request from the client, the client information and request data associated with the operation request are stored in the target queue.

[0084] According to embodiments of this disclosure, by storing client information associated with an operation request and request data in a target queue, it is convenient for the storage node to process the associated request data after receiving the operation request. Furthermore, upon receiving a response result, the response result is fed back to the client based on the client information.

[0085] In the embodiments of this disclosure, since the client's operation request and waiting instruction request are generated into a merged request, the agent will receive two response results from the storage node based on the merged request: a response result based on the operation request and a response result based on the waiting instruction request. According to the queue popping principle of the agent, after each response result is received, a client request object (e.g., requesting data) is popped from the queue. After generating the merged request, client request objects (e.g., requesting data) are repeatedly added to the queue.

[0086] Figure 5 The flowchart illustrating the data processing method according to an embodiment of the present disclosure in operation S210 is shown schematically.

[0087] like Figure 5 As shown, operation S210 includes operations S211 to S212.

[0088] In operation S211, the operation request obtained from the client is parsed according to a preset algorithm to determine the target data.

[0089] In the embodiments of this disclosure, after receiving an operation request from a client, the obtained operation request is parsed. For example, the preset algorithm may be a routing algorithm, which distinguishes data based on multiple features. By parsing the operation request, the key value in the operation request is obtained, and this key value is the target data.

[0090] In operation S212, the storage node corresponding to the operation request is determined based on the target data.

[0091] For example, based on the determined key value, the storage node corresponding to the operation request is calculated and determined. This storage node has a master node and M slave nodes.

[0092] Figure 6 The flowchart illustrating the data processing method according to an embodiment of the present disclosure in operation S220 is shown schematically.

[0093] like Figure 6 As shown, operation S220 includes operations S221 to S222.

[0094] In operation S221, the number of available slave nodes is counted based on the available status of each of the M slave nodes.

[0095] In the embodiments of this disclosure, the availability status of a slave node includes available and unavailable. When the availability status of a slave node is available, the number of slave nodes is counted. When the availability status of a slave node is unavailable, the number of slave nodes is not counted.

[0096] According to embodiments of this disclosure, by counting the number of available slave nodes, the reliability of storage nodes can be effectively improved. For example, when no slave node in a storage node is available, maintenance can be alerted to prevent data loss or service interruption when the master node fails. In embodiments of this disclosure, by counting the number of available slave nodes, the number of slave node execution results received when a storage node receives a merge request can also be determined. For example, if there are two available nodes, the number of slave node execution results received based on the merge request needs to be two. If the received number is less than two, it indicates that the slave node has not completed its execution and a waiting operation can be performed, thereby ensuring that each available slave node can execute the merge request.

[0097] In operation S222, when the number of available slave nodes is greater than a set threshold, operation requests and waiting instruction requests are processed to generate a merge request.

[0098] In embodiments of this disclosure, the threshold can be a positive integer greater than 0. For example, when the threshold is set to 0, if the number of available nodes is 1, the operation request and the waiting instruction request are processed to generate a merge request.

[0099] In other alternative embodiments, the threshold can be set to other suitable values.

[0100] Figure 7 The flowchart illustrating the data processing method according to an embodiment of the present disclosure during the dynamic disconnection process is shown.

[0101] like Figure 7 As shown, the data processing method of this embodiment further includes operation S500.

[0102] In operation S500, when a dynamic connection is terminated, the request queue associated with the dynamic connection is deleted, and the stored memory information associated with the dynamic connection is released.

[0103] According to embodiments of this disclosure, when a dynamic connection is established, as described above, client information and request data associated with the operation request are stored in a target queue, which may be a request queue associated with the dynamic connection. When the dynamic connection is terminated, it indicates that the data in the queue cannot be executed. At this time, the request queue associated with the dynamic connection is deleted, and the stored memory information associated with the dynamic connection is released. This effectively saves memory resources and avoids memory leaks caused by unresponsive storage layers.

[0104] Figure 8 The flowchart of the data processing method according to an embodiment of the present disclosure in operation 600 is illustrated schematically.

[0105] In embodiments of this disclosure, in operation S240, after sending the merge request to the storage node according to the dynamic connection relationship, the process further includes process 600: the agent receives and forwards the response result generated based on the execution result of the storage node. Process 600 includes operations S610 to S630.

[0106] In operation S610, the agent receives the first execution result, which is generated by the master node based on the operation request in the merge request.

[0107] In the embodiments of this disclosure, after the merge request is sent to the storage node, the master node in the storage node executes the operation request in the merge request and generates a first execution result. For example, if the first execution result is OK, it indicates that the master node has successfully executed the operation request. The agent receives the first execution result.

[0108] In operation S620, the agent receives the second execution result, which is generated by the slave node synchronizing the data of the master node to the M slave nodes according to the waiting instruction request in the merge request.

[0109] In the embodiments of this disclosure, after the master node generates the first execution result, the slave nodes in the storage nodes synchronize the master node's data to the M slave nodes according to the wait instruction request in the merge request, generating a second execution result. In this embodiment, the second execution result indicates that the master node's data has been synchronized to all of the M slave nodes. In an optional embodiment, the second execution result may indicate that the master node's data has been synchronized to at least one of the M slave nodes.

[0110] In operation S630, the agent generates a response result based on the received first and second execution results and forwards it to the client.

[0111] In the embodiments of this disclosure, after receiving the first and second execution results from the storage node, the agent generates a response result. This response result can represent the execution operation request of the master node in the storage node. In this embodiment, the response result also indicates that all data of the master node has been synchronized to the slave node. After receiving the response result, the client can perform other operations. At this time, the data of the master node and the slave node are consistent. When the master node fails and switches to the slave node, there will be no data loss, thus achieving data consistency between the master and slave nodes.

[0112] Figure 9 The illustration shows a schematic diagram of the execution process of a data processing method according to an embodiment of the present disclosure.

[0113] like Figure 9 As shown, the data processing method is executed through client 701, agent 702, and storage node 703.

[0114] Client 701 sends operation request 704 to agent 702. Operation request 704 may be, for example, a write operation. After receiving operation request 704 from client 701, agent 702 determines the storage node 703 corresponding to operation request 704. Storage node 703 includes a master node and M slave nodes. Agent 702 obtains the real-time availability status of the M slave nodes through a monitoring component that communicates with the registry center, and based on the real-time availability status of the M slave nodes, processes operation request 704 and waiting instruction request 705, generating merge request 706.

[0115] After generating merge request 706, a dynamic connection relationship is established between agent 702 and storage node 703 based on the merge request. The merge request 706 is then sent to the storage node through the dynamic connection relationship, and the execution result of storage node 703 based on merge request 706 is returned based on the dynamic connection relationship.

[0116] The master node in storage node 703 generates a first execution result 707 based on the operation request in the merge request, and the slave node in storage node 703 generates a second execution result 708 by synchronizing the data of the master node to the slave node based on the wait instruction request in the merge request.

[0117] The agent generates a response result 709 based on the received first execution result 707 and second execution result 708, and forwards the response result to the client 701.

[0118] According to embodiments of this disclosure, by acquiring the availability status of slave nodes in real time, the availability of slave nodes in the storage node can be guaranteed, avoiding the problem of failure or delay in operation requests of master and slave nodes due to the slave node's inability to respond to waiting instruction requests when the slave node is unavailable. By generating a merge request, the network transmission resource consumption of operation requests can be reduced, and the data processing speed can be improved. The agent generates a response request based on the received first and second execution results and forwards the response request to the client, ensuring the consistency of data between master and slave nodes, and adopting a dynamic connection relationship to ensure that operation requests are not blocked, thereby improving the efficiency of data processing.

[0119] Figure 10 The illustration schematically shows the response result of the wait instruction request in the merge request of the data processing method according to an embodiment of the present disclosure.

[0120] like Figure 10As shown, the slave node responds based on the wait instruction request in the merge request. For example, if all data from the master node is synchronized to the slave node, the response is successful, and the client is notified of the successful operation request. If the master node's data has not been synchronized to the slave node, the response is unsuccessful, and the success of the operation request in the merge request is further determined. If the master node succeeds in the operation request (e.g., a write request), it responds to the client with a custom error result. In this case, the master node has completed the operation request, but the slave node has not yet completed synchronizing the master node's data to the slave node. If the master node fails in the operation request (e.g., a write request), it responds to the client with an error result. In this case, the master node has not completed the operation request, and the slave node has not yet completed data synchronization.

[0121] Figure 11 The diagram illustrates the communication relationship between the agent, the registration center, and the monitoring component in the data processing method according to an embodiment of this disclosure.

[0122] like Figure 11 As shown, the monitoring component monitors the master and slave nodes in the storage nodes in real time. Specifically, it obtains the availability status of the slave nodes in real time and updates the availability status of the slave nodes to the registry center. After receiving an operation request from the client, the agent obtains the availability status of the slave nodes in the storage node corresponding to the operation request in real time through the registry center. Based on the availability status of the slave nodes, the number of slave nodes to be synchronized can be determined. Therefore, based on the number of available slave nodes, the agent receives the execution result generated by the storage node based on the waiting instruction request in the merge request, thus accurately responding to the synchronization status of the slave nodes.

[0123] Figure 12 A block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0124] like Figure 12 As shown, the data processing device 800 includes a determining module 810, a generating module 820, a processing module 830, and a receiving and forwarding module 840.

[0125] The determination module 810 is configured to determine the storage node corresponding to the operation request obtained from the client. The storage node includes a master node and M slave nodes, where M is a positive integer. In one embodiment, the determination module 810 can be used to execute the operation S210 described above, which will not be repeated here.

[0126] The generation module 820 is configured to process operation requests and waiting instruction requests based on the real-time availability status of M slave nodes, and generate a merge request. The real-time availability status of the M slave nodes is obtained in real time through a monitoring component that communicates with the registry center. In one embodiment, the generation module 820 can be used to execute the operation S220 described above, which will not be repeated here.

[0127] The processing module 830 is configured to establish a dynamic connection between the agent and the storage node based on the merge request, and to release the dynamic connection when preset conditions are met. In one embodiment, the processing module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0128] The receive-forward module 840 is configured to send a merge request to the storage node based on the dynamic connection relationship. The agent receives and forwards the response result generated based on the execution result of the storage node. The execution result is generated by the master and slave nodes of the storage node according to the merge request. In one embodiment, the receive-forward module 840 can be used to perform the operation S240 described above, which will not be repeated here.

[0129] In some exemplary embodiments of this disclosure, the data processing apparatus further includes an acquisition module, which is configured to acquire, from the registry center, the availability status of each of the M slave nodes as acquired in real time by the monitoring component, before processing the operation request and the waiting instruction request based on the real-time availability status of the M slave nodes to generate a merge request.

[0130] In some exemplary embodiments of this disclosure, the data processing apparatus further includes a storage module configured to: obtain an operation request from a client before determining a storage node corresponding to the operation request based on the operation request obtained from the client; and, after receiving the operation request obtained from the client, store client information and request data associated with the operation request in a target queue.

[0131] In some exemplary embodiments of this disclosure, the determining module includes a determining submodule, which is configured to: parse the operation request obtained from the client according to a preset algorithm to determine target data; and determine the storage node corresponding to the operation request according to the target data.

[0132] In some exemplary embodiments of this disclosure, the generation module includes a generation submodule, which is configured to: count the number of available slave nodes based on the available status of each of the M slave nodes; and when the number of available slave nodes is greater than a set threshold, process the operation request and the waiting instruction request to generate a merge request.

[0133] In some exemplary embodiments of this disclosure, the data processing apparatus further includes a deletion and release module, which is configured to: delete the request queue associated with the dynamic connection relationship and release the stored memory information associated with the dynamic connection relationship when the dynamic connection relationship is terminated.

[0134] In some exemplary embodiments of this disclosure, the receiving and forwarding module further includes a receiving and forwarding submodule, which is configured as follows: the proxy receives a first execution result, which is generated by the master node according to the operation request in the merge request; the proxy receives a second execution result, which is generated by the slave node synchronizing the data of the master node to M slave nodes according to the wait instruction request in the merge request; the proxy generates a response result based on the received first and second execution results and forwards it to the client.

[0135] According to embodiments of this disclosure, any plurality of modules among the determining module 810, generating module 820, processing module 830, and receiving / forwarding module 840 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the determining module 810, generating module 820, processing module 830, and receiving / forwarding module 840 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these methods. Alternatively, at least one of the determining module 810, generating module 820, processing module 830, and receiving / forwarding module 840 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0136] Figure 13 A block diagram of an electronic device for implementing a data processing method according to an embodiment of the present disclosure is shown schematically. Figure 13 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0137] like Figure 13As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0138] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0139] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0140] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the data processing method according to the embodiments of this disclosure.

[0141] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0142] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the data processing methods provided in the embodiments of this disclosure.

[0143] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0144] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0145] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0146] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0148] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0149] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data processing method, comprising: Based on the operation request obtained from the client, determine the storage node corresponding to the operation request. The storage node includes a master node and M slave nodes, where M is a positive integer. Based on the real-time availability status of the M slave nodes, the operation request and the waiting instruction request are processed to generate a merge request. The real-time availability status of the M slave nodes is obtained in real time through a monitoring component that communicates with the registry center. The waiting instruction request is used to determine the synchronization result of the master node synchronizing data with the slave nodes in the storage node, including counting the number of available slave nodes based on the availability status of each of the M slave nodes. When the number of available slave nodes is greater than a set threshold, the operation request and the waiting instruction request are processed to generate a merge request. According to the merge request, a dynamic connection is established between the agent and the storage node. The dynamic connection is terminated when preset conditions are met. The preset conditions include: the time taken to establish the dynamic connection exceeds a set time; and the slave node generates an execution result according to the merge request. When the dynamic connection is terminated, the request queue associated with the dynamic connection is deleted, and the memory information associated with the dynamic connection is released. Based on the dynamic connection relationship, the merge request is sent to the storage node. The agent receives and forwards the response result generated based on the execution result of the storage node. The execution result is generated by the master node and slave node of the storage node according to the merge request. When at least one slave node is available, a wait instruction request is sent to the slave node to obtain the data synchronization result of the slave node. When the number of available slave nodes is 0, the execution result of the master node based on the operation request is returned.

2. The method according to claim 1, wherein, Also includes: Before processing the operation request and wait instruction request based on the real-time availability status of the M slave nodes and generating the merge request, The availability status of each of the M slave nodes is obtained from the registration center in real time by the monitoring component.

3. The method according to claim 1, wherein, Also includes: Before determining the storage node corresponding to the operation request based on the operation request obtained from the client, Obtain operation requests from the client; Upon receiving an operation request from the client, the client information and request data associated with the operation request are stored in the target queue.

4. The method according to claim 1, wherein, The step of determining the storage node corresponding to the operation request based on the operation request obtained from the client includes: The operation request obtained from the client is parsed according to a preset algorithm to determine the target data; Based on the target data, determine the storage node corresponding to the operation request.

5. The method according to claim 1, wherein, The proxy receives and forwards a response result generated based on the execution result of the storage node, including: The agent receives a first execution result, which is generated by the master node based on the operation request in the merge request. The agent receives a second execution result, which is generated by the slave node synchronizing the data of the master node to the M slave nodes according to the waiting instruction request in the merge request. The agent generates a response result based on the received first and second execution results and forwards it to the client.

6. A data processing apparatus, comprising: The determination module is configured to determine the storage node corresponding to the operation request based on the operation request obtained from the client. The storage node includes a master node and M slave nodes, where M is a positive integer. The generation module is configured to process the operation request and the waiting instruction request based on the real-time availability status of the M slave nodes, and generate a merge request. The real-time availability status of the M slave nodes is obtained in real time through a monitoring component that communicates with the registry center. The waiting instruction request is used to determine the synchronization result of the master node synchronizing data with the slave nodes in the storage nodes, including counting the number of available slave nodes based on the availability status of each of the M slave nodes. When the number of available slave nodes is greater than a set threshold, the operation request and the waiting instruction request are processed to generate a merge request. The processing module is configured to establish a dynamic connection between the agent and the storage node according to the merge request, and to terminate the dynamic connection when the preset conditions are met. The preset conditions include: the time taken to establish the dynamic connection exceeds a set time; and the slave node generates an execution result according to the merge request. When the dynamic connection is terminated, the request queue associated with the dynamic connection is deleted, and the memory information associated with the dynamic connection is released. The receiving and forwarding module is configured to send the merge request to the storage node according to the dynamic connection relationship. The proxy end receives and forwards the response result generated based on the execution result of the storage node. The execution result is generated by the master node and slave node of the storage node according to the merge request. The device is also configured to send a wait instruction request to a slave node to obtain the data synchronization result of the slave node when at least one slave node is available, and return the execution result of the master node based on the operation request when the number of available slave nodes is 0.

7. An electronic device, comprising: One or more processors; A storage device for storing executable instructions, which, when executed by the processor, implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, implement the method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images