Data processing method and device

Abstract

The invention relates to a data processing method and device, relates to the technical field of data processing, is applied to any controller group in a controller cluster, and comprises the following steps: when a data query request is received, determining an identification parameter of at least one target controller group based on the data query request; the controller cluster comprises a plurality of controller groups in communication connection; forwarding the data query request to each target controller group corresponding to the identification parameter; the data query request is used for indicating each target controller group to query a matched database to obtain query result data; and when the query result data returned by the at least one target controller group is received, returning the summarized query result data to the data requester. In the embodiment of the invention, the controller group receiving the data query request is used as a transfer point of data processing and is used for sending the data query request to the target controller group and receiving and summarizing the query result data returned by the target controller group, so that the high availability of each node in the controller cluster is improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data processing method and apparatus. Background Art

[0002] With the development of large-scale distributed computing environments, the load requirements of computer systems have been continuously increasing. In related technologies, by adding multiple new controller nodes, the data processing capacity of the computer system and the concurrent capacity of services are expanded. However, with the increase in the number of controller nodes, how to ensure the high availability of each node has become a complex challenge. Summary of the Invention

[0003] Based on this, in view of the above technical problems, it is necessary to provide a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product that can ensure the high availability of each node.

[0004] In a first aspect, this application provides a data processing method, which is applied to any one of the controller groups in a controller cluster, and includes:

[0005] When receiving a data query request, determining identification parameters of at least one target controller group based on the data query request; the controller cluster includes multiple communicatively connected controller groups;

[0006] Forwarding the data query request to each of the target controller groups corresponding to the identification parameters; the data query request is used to instruct each of the target controller groups to query its matching database to obtain query result data associated with the data query request;

[0007] When receiving the query result data returned by at least one of the target controller groups, returning the aggregated query result data to the data requester.

[0008] In this embodiment, when any controller group in the controller cluster receives a data query request sent by a data requester, it can determine, in the data query request, identification parameters associated with at least one target controller group in the controller cluster. Then, this controller group forwards the received data query request to the target controller groups corresponding to the respective identification parameters, so as to instruct the target controller groups to be able to query the databases matching them to obtain query result data associated with the data query request. After that, it receives the query result data returned by each target controller group, and after summarizing the received query result data, returns it to the data requester, thereby completing the data processing work for the data query request. It can be seen that in the data processing method provided in this application, when any controller group in the controller cluster receives a data query request, it can forward the data query request to the target controller group associated with the data query request, so that the target controller group can directly query the database that matches it to obtain the query result data associated with the data query request, ensuring that the data query request can be accurately transmitted to the target controller group. Thus, it avoids the situation where the query result data associated with the data query request cannot be found in the database matched by the controller group that receives the data query request, and can also avoid the situation where the controller group that receives the data query request cannot forward the data query request to other controller groups that can query the query result data associated with the data query request, which is beneficial to improving the data processing accuracy and efficiency of the data query request. At the same time, the data processing method provided in this application can use the controller group that receives the data query request as a transfer point for data processing, for sending the data query request to the target controller group and for receiving and summarizing the query result data returned by the target controller group, which is beneficial to improving the high availability of each controller group node in the controller cluster.

[0009] In one embodiment, the method further includes:

[0010] When the determination of the identification parameters based on the data query request fails, determine the associated annotation data based on the data query request;

[0011] Determine the target query data corresponding to the data query request based on the annotation data;

[0012] Determine at least one target controller group based on the target query data, and forward the data query request to each of the target controller groups; the data query request is used to instruct each of the target controller groups to query its matching database to obtain query result data associated with each of the target query data.

[0013] In this embodiment, when the current controller group that receives a data query request cannot determine, from the data query request, an identification parameter associated with any target controller group in the controller cluster, it indicates that no valid identification parameter is carried in the data query request. At this time, it can be determined whether the data query request includes annotation data. If the data query request includes annotation data, the target query data associated with the data query request can be determined based on the annotation data. Furthermore, based on the matching relationship between the target query data and each controller group in the controller cluster, it can be determined which target controller groups can be used to query the relevant query result data for each target query data. Then, the data query request is forwarded by the current controller group to each target controller group to instruct each target controller group to query the database that matches it to obtain the query result data of each target query data associated with the data query request. After that, the query result data returned by each target controller group is received, and after summarizing the received query result data, it is returned to the data requestor, thus completing the data processing work for the data query request. It can be seen that by providing this embodiment, when the current controller group that receives a data query request finds that the data query request does not have an identification parameter associated with any target controller group, another method for determining the target controller group associated with the data query request is provided, which is beneficial to improving the data processing ability and data processing efficiency of the controller cluster for the data query request. Specifically, in this embodiment, when the data query request includes annotation data, the target query data corresponding to the data query request is determined based on the annotation data, and then the target controller group corresponding to the data query request is determined based on the target query data. In this way, the current controller group can also forward the data query request to the target controller group associated with the data query request, so that the target controller group can directly query the database that matches it to obtain the query result data associated with the data query request, ensuring that the data query request can be accurately transmitted to the target controller group. Thus, it avoids the situation where the query result data associated with the data query request cannot be queried in the database matched by the controller group that receives the data query request, and also avoids the situation where the controller group that receives the data query request cannot forward the data query request to other controller groups that can query the query result data associated with the data query request, which is beneficial to improving the data processing accuracy and efficiency of the data query request. At the same time, the data processing method provided by this application can use the controller group that receives the data query request as a data processing transfer point to send the data query request to the target controller group and to receive and summarize the query result data returned by the target controller group, which is beneficial to improving the high availability of each controller group node in the controller cluster.

[0014] In one embodiment, determining at least one target controller group based on the target query data and forwarding the data query request to each of the target controller groups includes:

[0015] Determining at least one target controller group associated based on each of the target query data;

[0016] Generating a sub-query request corresponding to the data query request based on each of the target query data associated with the same target controller group;

[0017] Forwarding each of the sub-query requests to the associated target controller group.

[0018] In this embodiment, the current controller group first determines, based on the matching relationship between each target query data and each controller group in the controller cluster, which target controller groups can be used to query and obtain relevant query result data for each target query data. Then, based on each target query data associated with the same target controller group, a sub-query request corresponding to the data query request for each target controller group is generated. Then, each sub-query request is forwarded to the corresponding target controller group to indicate that each target controller group can query the database that matches it to obtain the query result data of each target query data associated with the data query request. In this way, the data query request can be split into multiple sub-query requests by the current controller group, and then the corresponding associated sub-query requests are forwarded to multiple target controller groups, which is beneficial to reducing the data volume of the query requests processed by each target controller group, thereby further improving the data processing accuracy and efficiency of the data query request, and also further improving the high availability of each controller group node in the controller cluster.

[0019] In one embodiment, the method further includes:

[0020] In the case where the identification parameter cannot be determined based on the data query request and the annotation data cannot be determined, obtaining the query result data corresponding to the data query request based on the current controller group.

[0021] In this embodiment, when the current controller group that receives a data query request cannot determine, from the data query request, an identification parameter associated with any target controller group in the controller cluster, and the data query request does not include annotation data, the query result data corresponding to the data query request can be directly obtained based on the current controller group. That is, the current controller group directly queries the database corresponding to it for the query result data of the data query request. It can be seen that this application provides this embodiment, which provides another method for querying relevant query result data based on the data query request when the current controller group that receives the data query request finds that the data query request has no identification parameter associated with any target controller group and does not contain annotation data, thereby facilitating the guarantee of the data processing ability and data processing efficiency of the controller cluster for the data query request.

[0022] In one embodiment, the determining, based on the data query request, an identification parameter of at least one target controller group includes:

[0023] Obtain multiple target parameters in the data query request, and determine the identification parameter of the target controller group from the multiple target parameters.

[0024] In this embodiment, the current controller group can first obtain the multiple target parameters included in the data query request, and then screen out the identification parameter associated with the target controller group from the multiple target parameters, thereby improving the determination efficiency and accuracy of the identification parameter.

[0025] In one embodiment, the determining, based on the data query request, an identification parameter of at least one target controller group when receiving the data query request includes:

[0026] When receiving the data query request, identify the type of the current controller that receives the data query request; the current controller belongs to the current controller group;

[0027] When the type of the current controller is the master controller, determine an identification parameter of at least one target controller group based on the data query request.

[0028] In this embodiment, when the current controller group receives the data query request, it can first identify the type of the current controller that receives the data query request in the current controller group. When it is identified that the type of the current controller is the master controller, the identification parameter associated with at least one target controller group in the controller cluster can be further determined in the data query request; in this way, it is beneficial to avoid that the slave controller in the current controller group receives the data query request, so as to ensure that the master controller that receives the data query request can further execute other steps in the data processing method provided by this application.

[0029] In one embodiment, the method further includes:

[0030] When the type of the current controller is a slave controller, forward the data query request to the master controller in the current controller group.

[0031] In this embodiment, when the current controller group identifies that the type of the current controller receiving the data query request is a slave controller rather than a master controller, it will forward the data query request to the master controller in the current controller group; since the slave controller cannot execute other steps in the data processing method provided in this application, therefore, when the slave controller in the current controller receives the data query request, forwarding the data query request to the master controller is beneficial to ensuring that the master controller receiving the data query request can further execute other steps in the data processing method provided in this application.

[0032] In one embodiment, the method further includes:

[0033] When it is identified that the master controller in the current controller group fails, update any one of the slave controllers in the current controller group to be the new master controller; the current controller group includes one master controller and multiple slave controllers;

[0034] And update the log information of the failed master controller to the new master controller.

[0035] In this embodiment, each controller group includes one master controller and multiple slave controllers; when any controller group identifies that the master controller in the current controller group fails, any one of the slave controllers in the current controller group can be updated to be the new master controller in the current controller group, and the old master controller is deprecated; at the same time, the log information of the failed old master controller is updated to the new master controller, so that the log information of the new master controller includes the log information of the old master controller, so that the new master controller can continue the old master controller to process the data processing requests received by the current controller group, ensuring the high availability of the current controller group.

[0036] In one embodiment, the method further includes:

[0037] When the master controller in the current controller group receives any data update request, send the log information of the data update request to at least one slave controller in the current controller group; the current controller group includes one master controller and multiple slave controllers.

[0038] In this embodiment, each controller group includes a main controller and multiple slave controllers; when any controller group identifies the main controller in the current controller group and receives any data update request, it can send the log information corresponding to the data update request to at least one slave controller in the current controller group, so that the log information of at least one slave controller in the current controller group is the same as that of the main controller; this is beneficial to reducing the amount of log information updated from the old main controller to the new main controller when the slave controller is updated to the new main controller in the current controller group, thereby facilitating improving the efficiency of the new main controller in processing the data processing requests received by the current controller group.

[0039] In a second aspect, the present application further provides a data processing device, which is applied to any controller group in a controller cluster and includes:

[0040] An identification determination module, configured to determine identification parameters of at least one target controller group based on the data query request when receiving the data query request; the controller cluster includes multiple communicatively connected controller groups;

[0041] A request forwarding module, configured to forward the data query request to each of the target controller groups corresponding to the identification parameters; the data query request is used to instruct each of the target controller groups to query its matching database to obtain query result data associated with the data query request;

[0042] A result summarization module, configured to return the summarized query result data to the data requester when receiving the query result data returned by at least one of the target controller groups.

[0043] In a third aspect, the present application further provides a computer device, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the method described in the first aspect above.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the method described in the first aspect above.

[0045] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps of the method described in the first aspect above.

[0046] In the data processing method, apparatus, computer device, computer-readable storage medium, and computer program product provided by the present application, when any controller group in the controller cluster receives a data query request, the data query request can be forwarded to the target controller group associated with the data query request, so that the target controller group can directly query the query result data associated with the data query request from the matching database, ensuring that the data query request can be accurately transmitted to the target controller group; thus, it is possible to avoid the situation where the query result data associated with the data query request cannot be found in the database matched by the controller group that receives the data query request, and it is also possible to avoid the situation where the controller group that receives the data query request cannot forward the data query request to other controller groups that can query the query result data associated with the data query request, which is beneficial to improving the accuracy and efficiency of data processing for the data query request; at the same time, the data processing method provided by the present application can use the controller group that receives the data query request as a transfer point for data processing to send the data query request to the target controller group and receive and summarize the query result data returned by the target controller group, which is beneficial to improving the high availability of each controller group node in the controller cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can be obtained based on these drawings.

[0048] Figure 1 It is an application environment diagram of the data processing method in an embodiment;

[0049] Figure 2 It is a flowchart of the data processing method in an embodiment;

[0050] Figure 3 It is a flowchart of sub-steps of the data processing method in an embodiment;

[0051] Figure 4 It is a flowchart of the data processing method in another embodiment;

[0052] Figure 5 It is a structural block diagram of the data processing apparatus in an embodiment;

[0053] Figure 6 It is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] In order to make the objectives, technical solutions, and advantages of this application more clearly understood, the following further details this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely used to explain this application and are not used to limit this application.

[0055] In today's large-scale distributed computing environment, horizontal scaling (scale-out) is a key technology to ensure that the system can handle continuously growing load demands. By adding new nodes, the system can expand its processing power, storage capacity, and concurrent service capabilities. However, as the number of nodes increases, ensuring data consistency and high availability among the nodes in the system becomes a complex challenge.

[0056] In existing distributed systems, data synchronization and consistency management are usually achieved through consistency protocols (such as Raft or Paxos). However, traditional synchronization mechanisms often encounter problems such as data inconsistency and system performance degradation when facing high-concurrency write operations or node failures. In addition, how to quickly and reliably synchronize data in a large-scale cluster is also a major problem in current technologies.

[0057] In the case of network partitioning or node failures, many existing technologies are difficult to ensure data consistency while maintaining the high availability of the system. Especially when the "split-brain" problem occurs in a distributed system, the consistency and availability of the system will be severely affected.

[0058] As the number of nodes increases, traditional replication mechanisms such as Paxos (a distributed consistency algorithm based on message passing) and Quorum (a data consistency protocol in a distributed system) replication face performance bottlenecks and cannot efficiently handle the synchronization and consistency issues of a large number of nodes. Gossip (a communication protocol that allows sharing of state in a distributed system), although having good scalability, cannot meet strict consistency requirements.

[0059] To solve the above problems existing in the related technologies, this application provides a data processing method that can solve the above-mentioned technical problems. This data processing method can be applied to an application environment as Figure 1 shown. Among them, the controller cluster 100 includes multiple controller groups 10. Each controller group 10 may include a primary controller 11 and multiple secondary controllers 12. An optional embodiment provided by this application is that each controller group 10 includes three secondary controllers 12; the multiple controller groups 10 in the controller cluster 100 can be communicatively connected.

[0060] In the data processing method provided by the embodiments of the present application, any controller group in the controller cluster, when receiving a data query request, can determine the identification parameters of at least one target controller group based on the data query request; and then forward the data query request to each target controller group corresponding to the identification parameters; the data query request is used to instruct each target controller group to query its matching database to obtain query result data associated with the data query request; when the current controller group receives the query result data returned by at least one target controller group, data summarization is performed, and then the summarized query result data is returned to the data requester to complete the data processing work for the data query request. By using the controller group that receives the data query request as a transfer point for data processing, which is used to send the data query request to the target controller group and to receive and summarize the query result data returned by the target controller group, this data processing method is beneficial to improving the high availability of each controller group node in the controller cluster.

[0061] In an exemplary embodiment, as Figure 2 shown, a data processing method is provided. Taking any one of the controller groups 10 to which this method is applied Figure 1 as an example for illustration, it includes the following steps 201 to step 203.

[0062] Step 201, when receiving a data query request, determine the identification parameters of at least one target controller group based on the data query request; the controller cluster includes multiple communicatively connected controller groups.

[0063] Among them, the data query request can be initiated by a user on the front-end page of a computer for a query request for the result corresponding to specific data.

[0064] The identification parameter can be the ID (Identity document) of each controller group in the controller cluster, or rather, the number of each controller group. Equivalently, the identification parameter is the unique identifier of each controller group.

[0065] Exemplarily, when any controller group in the controller cluster receives a data query request sent by a data requester, the identification parameters associated with at least one target controller group in the controller cluster can be determined in the data query request. Among them, each controller group corresponds to a fixed identification parameter, and the identification parameters corresponding to each controller group are all different.

[0066] It should be noted that the identification parameters of the target controller group that can be determined by each data query request can be one or multiple, and the present application does not make specific limitations on this.

[0067] Step 202: Forward the data query request to each target controller group corresponding to the identification parameter; the data query request is used to instruct each target controller group to query its matching database to obtain the query result data associated with the data query request.

[0068] Among them, it can be optionally set that each controller group corresponds to and matches one database, or it can be set that multiple controller groups correspond to and match one database; among them, the data stored in each database can be all different. Each controller group can only query data from the database associated with it and obtain the query result data associated with the data query request stored in the matching database.

[0069] Exemplarily, the current controller group forwards the received data query request to each target controller group corresponding to the identification parameter to instruct the target controller group to query the database associated with it and obtain the query result data associated with the data query request.

[0070] After the query result data associated with the data query request is queried in each database, the relevant query result data will be sent to the corresponding matching target controller group first, and then each target controller group will return the query result data to the current controller group.

[0071] Step 203: When the query result data returned by at least one target controller group is received, return the aggregated query result data to the data requester.

[0072] Exemplarily, after the current controller group receives the query result data returned by each target controller group, it aggregates the received query result data and then returns the aggregated data to the data requester, so that the data requester (such as the front-end page) can receive the required query result data and display it to the user, thereby completing the data processing work for the data query request.

[0073] In the data processing method provided by this application, any controller group in the controller cluster, when receiving a data query request, can forward the data query request to the target controller group associated with the data query request, so that the target controller group can directly query the query result data associated with the data query request from the matching database, ensuring that the data query request can be accurately transmitted to the target controller group; thus avoiding the situation where the query result data associated with the data query request cannot be queried from the database matched by the controller group receiving the data query request, and also being able to avoid the situation where the controller group receiving the data query request cannot forward the data query request to other controller groups that can query the query result data associated with the data query request, which is beneficial to improving the data processing accuracy and efficiency of the data query request; at the same time, the data processing method provided by this application can use the controller group receiving the data query request as a data processing transfer point to send the data query request to the target controller group and to receive and summarize the query result data returned by the target controller group, which is beneficial to improving the high availability of each controller group node in the controller cluster.

[0074] That is to say, the data processing method provided by this application can achieve that when any controller group receives a business request, it will forward the relevant request to the corresponding target controller group; and when performing data query, it can summarize the information of multiple controller groups.

[0075] In an exemplary embodiment, as Figure 3 shown, the data processing method provided by this application may further include the following steps 301 to 303.

[0076] Step 301, when it fails to determine the identification parameter based on the data query request, determine the associated annotation data based on the data query request.

[0077] Among them, the annotation data is also the annotation on the interface corresponding to the data query request, and this annotation can be a custom annotation added to the interface; in this application, the annotation data includes the relevant data of multiple target parameters included in some data query requests. The existence of the annotation data means that it is necessary to first query the data to be queried from the metadata, and specifically which databases matched by which controller groups the data to be queried is distributed in, and then based on each controller group, query the relevant results of the data to be queried from the matched database.

[0078] Exemplarily, in the case where the current controller group that receives the data query request cannot determine the identification parameter associated with any target controller group in the controller cluster from the data query request, it indicates that no valid identification parameter is carried in the data query request; at this time, it can be determined whether the data query request includes annotation data.

[0079] Step 302, determine the target query data corresponding to the data query request based on the annotation data.

[0080] Among them, the annotation data includes the relevant data of multiple target parameters included in some data query requests. Specifically, the annotation data may include, among the multiple target parameters included in the data query request, the corresponding target query data, such as the data to be queried input by the user on the front-end page.

[0081] Exemplarily, in the case where the current controller group determines that the data query request includes annotation data, the target query data associated with the data query request can be determined based on the annotation data.

[0082] Step 303, determine at least one target controller group based on the target query data, and forward the data query request to each target controller group; the data query request is used to instruct each target controller group to query its matching database to obtain the query result data associated with each target query data.

[0083] Among them, there is an association relationship between the target query data and the target controller group, or rather, there is an association relationship between the target query data and the database matched by the target controller group. A relevant association relationship can be, for example, that any target query data corresponds to a relevant target controller group, or any target query data corresponds to a relevant database.

[0084] Exemplarily, the current controller group can determine, based on the matching relationship between the target query data and each controller group in the controller cluster, which target controller groups can be used to query and obtain the relevant query result data for each target query data. Then, the current controller group forwards the data query request to each target controller group to instruct each target controller group to query its matching database to obtain the query result data associated with each target query data in the data query request; afterwards, receive the query result data returned by each target controller group, and after summarizing the received query result data, return it to the data requestor, thereby completing the data processing work for the data query request.

[0085] In this embodiment, when it is found in the current controller group receiving a data query request that none of the identification parameters of the target controller groups are associated with the data query request, another method for determining the target controller group associated with the data query request is provided, which is beneficial to improving the data processing ability and data processing efficiency of the controller cluster for data query requests.

[0086] Among them, in this embodiment, when the data query request includes annotation data, the target query data corresponding to the data query request is determined based on the annotation data, and then the target controller group corresponding to the data query request is determined based on the target query data; in this way, the current controller group can also forward the data query request to the target controller group associated with the data query request, so that the target controller group can directly query the query result data associated with the data query request from the matching database, ensuring that the data query request can be accurately transmitted to the target controller group; thus, it is avoided that in the database matched by the controller group receiving the data query request, the query result data associated with the data query request cannot be queried, and it is also avoided that the controller group receiving the data query request cannot forward the data query request to other controller groups that can query the query result data associated with the data query request, which is beneficial to improving the data processing accuracy and efficiency of the data query request; at the same time, the data processing method provided by this application can use the controller group receiving the data query request as a data processing transfer point to send the data query request to the target controller group and receive and summarize the query result data returned by the target controller group, which is beneficial to improving the high availability of each controller group node in the controller cluster.

[0087] In addition, an alternative embodiment is provided. When it is possible to determine the identification parameters of at least one target controller group based on the data query request, the priority of the data query request that can determine the identification parameters can be set to the highest, so that when the controller cluster receives multiple data query requests or multiple data processing requests, it can first process the data query requests including the identification parameters.

[0088] In an exemplary embodiment, determining at least one target controller group based on the target query data and forwarding the data query request to each target controller group in the above steps may specifically include: determining at least one target controller group associated with each target query data; generating sub-query requests corresponding to the data query request based on the target query data associated with the same target controller group; and forwarding each sub-query request to the associated target controller group.

[0089] Among them, a sub-query request refers to at least two sub-query requests split from a data query request; each sub-query request includes relevant data to be queried (target query data) that needs to be queried through an associated target controller group.

[0090] Exemplarily, the current controller group first determines, based on the matching relationship between each target query data and each controller group in the controller cluster, which target controller groups can be used to query and obtain relevant query result data for each target query data. Then, based on each target query data associated with the same target controller group, sub-query requests corresponding to each target controller group for the data query request are generated, and then each sub-query request is forwarded to the corresponding target controller group to instruct each target controller group to query the database that matches it to obtain the query result data for each target query data associated with the data query request.

[0091] In this embodiment, by splitting the data query request into multiple sub-query requests through the current controller group and then forwarding the corresponding associated sub-query requests to multiple target controller groups, it is beneficial to reduce the amount of data in the query requests processed by each target controller group, thereby further improving the data processing accuracy and efficiency of the data query request, and also further enhancing the high availability of each controller group node in the controller cluster.

[0092] In an exemplary embodiment, the data processing method provided by this application further includes: when the identification parameter of the data query request cannot be determined and the annotation data cannot be determined, obtaining the query result data corresponding to the data query request based on the current controller group.

[0093] Exemplarily, when the current controller group receiving the data query request cannot determine the identification parameter associated with any target controller group in the controller cluster from the data query request and the data query request does not include annotation data, it means that the currently received data query request can be processed by the current controller group and does not involve the situation where the data is distributed in the databases matched by other different controller groups; at this time, the query result data corresponding to the data query request can be directly obtained based on the current controller group, that is, the current controller group directly queries the database that matches it for the query result data of the data query request.

[0094] This application provides this embodiment, which provides another method for querying relevant query result data based on the data query request when the current controller group receiving the data query request finds that the data query request has no identification parameter associated with any target controller group and does not contain annotation data, thereby facilitating ensuring the data processing ability and data processing efficiency of the controller cluster for the data query request.

[0095] In an exemplary embodiment, determining the identification parameter of at least one target controller group based on the data query request in the above steps may specifically include: obtaining a plurality of target parameters in the data query request, and determining the identification parameter of the target controller group among the plurality of target parameters.

[0096] Among them, the data query request may include a plurality of target parameters, and at least the target query data and the identification parameter may be included in the plurality of target parameters, and the identification parameter is the identification parameter corresponding to the controller group.

[0097] In this embodiment, by setting the current controller group to first obtain the plurality of target parameters included in the data query request, and then screening out the identification parameter associated with the target controller group from the plurality of target parameters, the determination efficiency and accuracy of the identification parameter can be improved; furthermore, it is also beneficial to improve the accuracy of the query result data finally obtained.

[0098] In an exemplary embodiment, in the case of receiving a data query request, determining the identification parameter of at least one target controller group based on the data query request may specifically include: in the case of receiving a data query request, identifying the type of the current controller that receives the data query request; the current controller belongs to the current controller group; in the case where the type of the current controller is the master controller, determining the identification parameter of at least one target controller group based on the data query request.

[0099] Among them, as described above, each controller group may include a master controller and a plurality of slave controllers. Therefore, the types of controllers may include two types: master controllers and slave controllers.

[0100] Exemplarily, when the current controller group receives a data query request, it may first identify the type of the current controller that receives the data query request in the current controller group. In the case where it is identified that the type of the current controller is the master controller, it may further determine the identification parameter associated with at least one target controller group in the controller cluster in the data query request, and execute the subsequent query and acquisition of the query result data related to the data query request for the target controller.

[0101] Since the slave controller is unable to execute the subsequent query acquisition step of the query result data related to the data query request based on the target controller, when the current controller that receives the data query request in the current controller group is set as the master controller in this application, the associated identification parameter is determined from the data query request, and the subsequent query acquisition of the query result data related to the data query request based on the target controller is executed, which helps to avoid the situation that the slave controller in the current controller group receives the data query request and is unable to execute the query acquisition action for the query result data, thus helping to ensure that the master controller that receives the data query request can further execute other steps in the data processing method provided in this application.

[0102] In an exemplary embodiment, the data processing method provided in this application further includes: when the type of the current controller is a slave controller, forwarding the data query request to the master controller in the current controller group.

[0103] In this embodiment, when the current controller group identifies that the type of the current controller that receives the data query request is a slave controller rather than a master controller, the data query request will be forwarded to the master controller in the current controller group; since the slave controller is unable to execute other steps in the data processing method provided in this application, therefore, when the slave controller in the current controller receives the data query request, forwarding the data query request to the master controller helps to ensure that the master controller that receives the data query request can further execute other steps in the data processing method provided in this application.

[0104] In an exemplary embodiment, the data processing method provided in this application further includes: when it is identified that the master controller in the current controller group fails, updating any one of the slave controllers in the current controller group as the new master controller; the current controller group includes one master controller and multiple slave controllers; and updating the log information of the failed master controller to the new master controller.

[0105] In this embodiment, when it is identified that the master controller in any controller group fails, any one of the slave controllers in the current controller group can be updated as the new master controller in the current controller group, and the old master controller is deprecated; at the same time, the log information of the failed old master controller is updated to the new master controller, so that the log information of the new master controller includes the log information of the old master controller, so that the new master controller can continue the old master controller to process the data processing requests received by the current controller group, ensuring the high availability of the current controller group.

[0106] An alternative embodiment is that each controller group node in this application is composed of a master-slave node (a master controller and a slave controller). When the master controller in a certain controller group drops offline due to a network failure or other reasons, the computer system can immediately detect this failure situation and notify any one of the slave nodes (slave controllers) in this controller group to be promoted to a new master node (master controller). At the same time, update the log entries backed up from the old master node (master controller) to the new master controller, so as to ensure the availability of each controller group in the controller cluster.

[0107] In this application, by introducing the one-master-multi-slave (one master controller and multiple slave controllers) mechanism and the automatic failover mechanism within the controller group, it is ensured that the system can still continue to provide services in the case of any node failure or network instability, thus achieving high availability of the system. This enables the system to maintain business continuity in various abnormal situations.

[0108] In an exemplary embodiment, the data processing method provided by this application further includes: when the master controller in the current controller group receives any data update request, send the log information of the data update request to at least one slave controller in the current controller group; the current controller group includes one master controller and multiple slave controllers.

[0109] In this embodiment, by recognizing that the master controller in any controller group receives any data update request, the log information corresponding to the data update request can be sent to at least one slave controller in the current controller group, so that the log information of at least one slave controller in the current controller group is the same as that of the master controller; it is beneficial to reduce the amount of log information updated from the old master controller to the new master controller when the slave controller is updated to be the new master controller in the current controller group, thus facilitating the improvement of the processing efficiency of the new master controller for the data processing requests received by the current controller group.

[0110] An alternative embodiment is that any master controller, when receiving a data change operation, can take each received data change operation as a log entry and copy it to all slave controllers in the controller group corresponding to the master controller in sequence. When more than half of the slave nodes (slave controllers) confirm receiving the log entry sent by the master controller, this entry is considered to have been committed.

[0111] This application adopts the method of sharing log replication among multiple slave nodes in the same controller group, which can greatly reduce the actions of data resynchronization during the master-slave failure process. That is, it avoids the need for slave nodes to fully synchronize the log data related to the master node after the master node fails. Through the incremental method, the complete data processing ability of the controller group can be quickly restored.

[0112] For the data processing method provided by this application, an alternative implementation provided by this application includes the following steps. In these steps, the controller group 1 is used as the current controller group; the master controller in the current controller group is, for example, master controller A. Please refer to Figure 4 , where:

[0113] When the controller group 1 receives a data query request, based on the Filter function, encapsulate the received data query request to obtain an encapsulated request; among them, the Filter function can filter a series of data based on defined conditions;

[0114] In the controller group 1, determine the master controller A; this step can be implemented based on a master selection interceptor;

[0115] Judge whether the encapsulated request is sent to the master controller A;

[0116] When the encapsulated request is sent to the master controller A, judge whether the encapsulated request includes a target controller group ID identifier (identification parameter);

[0117] When the encapsulated request includes a target controller group ID identifier, judge whether the ID identifier points to the controller group 1;

[0118] When the ID identifier points to the controller group 1, extract the target data (query result data) corresponding to the encapsulated request from the associated database through the controller group 1;

[0119] When the encapsulated request is not sent to the master controller A, forward the encapsulated request to the master controller A;

[0120] When the encapsulated request does not include a target controller group ID identifier, judge whether the annotation data of the encapsulated request can be obtained;

[0121] When the annotation data of the encapsulated request is not obtained, extract the target data corresponding to the encapsulated request from the associated database through the controller group 1;

[0122] When the annotation data of the encapsulated request is obtained, determine the metadata based on the annotation data, determine the target controller group where the data to be requested is located based on the metadata, and obtain the target data corresponding to each data to be requested in the matching database through each target controller group;

[0123] When the ID identifier does not point to the controller group 1, forward the encapsulated request to the target controller group pointed to by the ID identifier.

[0124] It should be noted that Figure 4"Y" in it represents "yes", that is, the case where the judgment result is "yes"; "N" represents "no", that is, the case where the judgment result is "no".

[0125] Among them, for all steps after the judgment result is that the encapsulated request is sent to the main controller A, it can be implemented based on the cluster forwarding interceptor.

[0126] It can be seen that in the data processing method provided by this application, when performing data query, one implementation method that can be selected is to obtain basic data through cluster metadata during the query, and then use the unique identifier of each controller group to perform a full-volume data query summary and return it to the user. Among them, metadata management refers to providing the annotations required for metadata, annotation parsing, metadata configuration caching, and metadata data structures.

[0127] This application is equivalent to proposing a highly available Scale-out data synchronization management method based on the Raft consensus algorithm, aiming to overcome the limitations existing in the related technologies; to provide a data synchronization solution that can not only ensure data consistency but also have good scalability and fault tolerance.

[0128] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or steps or stages in other steps.

[0129] Based on the same inventive concept, the embodiment of this application also provides a data processing device 500 for implementing the above-mentioned data processing method. The solution provided by this device 500 to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the following data processing device 500 can refer to the limitations on the data processing method in the above text and will not be repeated here.

[0130] In an exemplary embodiment, as Figure 5 shown, a data processing device 500 is provided, including: an identifier determination module 501, a request forwarding module 502, and a result summary module 503, where:

[0131] An identification determination module 501, configured to determine identification parameters of at least one target controller group based on a data query request when receiving the data query request; a plurality of communicatively connected controller groups are included in the controller cluster;

[0132] A request forwarding module 502, configured to forward the data query request to each target controller group corresponding to the identification parameters; the data query request is used to instruct each target controller group to query its matching database to obtain query result data associated with the data query request;

[0133] A result summarization module 503, configured to return the summarized query result data to the data requestor when receiving the query result data returned by at least one target controller group.

[0134] In one embodiment, the data processing device 500 further includes an annotation determination module and a target query data determination module. Among them, the annotation determination module is configured to determine associated annotation data based on the data query request when the determination of the identification parameters based on the data query request fails; the target query data determination module is configured to determine target query data corresponding to the data query request based on the annotation data; the request forwarding module 502 is configured to determine at least one target controller group based on the target query data and forward the data query request to each target controller group; the data query request is used to instruct each target controller group to query its matching database to obtain query result data associated with each target query data.

[0135] In one embodiment, the request forwarding module 502 is further configured to determine at least one target controller group associated with each target query data; generate a sub-query request corresponding to the data query request based on each target query data associated with the same target controller group; and forward each sub-query request to the associated target controller group.

[0136] In one embodiment, the result summarization module 503 is further configured to obtain query result data corresponding to the data query request based on the current controller group when the determination of the identification parameters based on the data query request fails and the determination of the annotation data fails.

[0137] In one embodiment, the identification determination module 501 is further configured to obtain a plurality of target parameters in the data query request and determine the identification parameters of the target controller group among the plurality of target parameters.

[0138] In one embodiment, the identification determination module 501 is further configured to identify the type of the current controller that receives the data query request when receiving the data query request; the current controller belongs to the current controller group; and when the type of the current controller is a master controller, determine the identification parameters of at least one target controller group based on the data query request.

[0139] In one embodiment, the request forwarding module 502 is further configured to forward the data query request to the master controller in the current controller group when the type of the current controller is a slave controller.

[0140] In one embodiment, the data processing device 500 further includes a controller group update module, configured to update any one of the slave controllers in the current controller group to be the new master controller when it is recognized that the master controller in the current controller group fails; the current controller group includes one master controller and multiple slave controllers; and update the log information of the failed master controller to the new master controller.

[0141] In one embodiment, the controller group update module is further configured to send the log information of the data update request to at least one slave controller in the current controller group when the master controller in the current controller group receives any data update request; the current controller group includes one master controller and multiple slave controllers.

[0142] Each module in the above data processing device 500 can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0143] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as Figure 6As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a data processing method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0144] Those skilled in the art can understand that Figure 6 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0145] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, it implements the steps in the data processing method provided by the present application.

[0146] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, it implements the steps in the data processing method provided by the present application.

[0147] In an embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, it implements the steps in the data processing method provided by the present application.

[0148] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0149] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., without limitation.

[0150] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0151] The above-described embodiments merely represent several implementation manners of this application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application shall be subject to the appended claims.

Claims

1. A data processing method, characterized in that, Applied to any controller group in a controller cluster, the method includes: When receiving a data query request, determining identification parameters of at least one target controller group based on the data query request; a plurality of communicatively connected controller groups are included in the controller cluster; Forwarding the data query request to each of the target controller groups corresponding to the identification parameters; the data query request is used to instruct each of the target controller groups to query its matching database to obtain query result data associated with the data query request; When receiving the query result data returned by at least one of the target controller groups, returning the aggregated query result data to the data requester.

2. The method according to claim 1, wherein The method further includes: When the determination of the identification parameters based on the data query request fails, determining annotation data associated with the data query request; Determining target query data corresponding to the data query request based on the annotation data; Determining at least one target controller group based on the target query data, and forwarding the data query request to each of the target controller groups; the data query request is used to instruct each of the target controller groups to query its matching database to obtain query result data associated with each of the target query data.

3. The method according to claim 2, characterized in that The determining at least one target controller group based on the target query data and forwarding the data query request to each of the target controller groups includes: Determining at least one target controller group associated with each of the target query data; Generating a sub-query request corresponding to the data query request based on each of the target query data associated with the same target controller group; Forwarding each of the sub-query requests to the associated target controller group.

4. The method according to claim 2, wherein The method further includes: When the determination of the identification parameters based on the data query request fails and the determination of the annotation data fails, obtaining query result data corresponding to the data query request based on the current controller group.

5. The method according to claim 1, wherein The determining identification parameters of at least one target controller group based on the data query request includes: Obtaining a plurality of target parameters in the data query request, and determining the identification parameters of the target controller group among the plurality of target parameters.

6. The method according to claim 1, wherein The when receiving a data query request, determining identification parameters of at least one target controller group based on the data query request includes: When receiving a data query request, identifying the type of the current controller that receives the data query request; the current controller belongs to the current controller group; When the type of the current controller is a master controller, determining identification parameters of at least one target controller group based on the data query request.

7. The method according to claim 6, wherein The method further includes: When the type of the current controller is a slave controller, forwarding the data query request to the master controller in the current controller group.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: In the case of identifying a failure of the master controller in the current controller group, update any one of the slave controllers in the current controller group to be the new master controller; the current controller group includes one master controller and multiple slave controllers; And update the log information of the failed master controller to the new master controller.

9. The method according to any one of claims 1-7, characterized in that, The method further includes: In the case where the master controller in the current controller group receives any data update request, send the log information of the data update request to at least one slave controller in the current controller group; the current controller group includes one master controller and multiple slave controllers.

10. A data processing device, characterized in that, Applied to any controller group in the controller cluster, the device includes: An identification determination module, configured to determine identification parameters of at least one target controller group based on the data query request when receiving the data query request; the controller cluster includes multiple communicatively connected controller groups; A request forwarding module, configured to forward the data query request to each of the target controller groups corresponding to the identification parameters; the data query request is used to instruct each of the target controller groups to query its matching database to obtain query result data associated with the data query request; A result summarization module, configured to return the summarized query result data to the data requestor when receiving the query result data returned by at least one of the target controller groups.

Images