Data processing method and device

Abstract

The present application relates to a data processing method and device, and relates to the field of data processing technology, and is applied to any controller group in a controller cluster. The method includes determining an identification parameter of at least one target controller group based on the data query request when a data query request is received; the controller cluster includes multiple controller groups with communication connections; forwarding the data query request to each target controller group corresponding to the identification parameter; the data query request is used to instruct each target controller group to query its matching database to obtain query result data; and upon receiving the query result data returned by at least one target controller group, returning the aggregated query result data to the data requester. The controller group that receives the data query request is used as a transit point for data processing, and is used to send data query requests to the target controller group, and receive and aggregate the query result data returned by the target controller group, which is conducive to improving the high availability of each node in the controller cluster.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and device. Background Art

[0002] With the development of large-scale distributed computing environments, the load demands on computer systems continue to grow. Related technologies expand the data processing capabilities and service concurrency of computer systems by adding multiple new controller nodes. However, as the number of controller nodes increases, ensuring the high availability of each node becomes a complex challenge. Summary of the Invention

[0003] Based on this, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can ensure high availability of each node in response to the above technical problems.

[0004] In a first aspect, the present application provides a data processing method, which is applied to any controller group in a controller cluster, comprising:

[0005] In the case of receiving a data query request, determining an identification parameter of at least one target controller group based on the data query request; the controller cluster includes a plurality of the controller groups in communication connection;

[0006] Forwarding the data query request to each of the target controller groups corresponding to the identification parameter; the data query request is used to instruct each of the target controller groups to query its matching database to obtain query result data associated with the data query request;

[0007] When the query result data returned by at least one target controller group is received, the aggregated query result data is returned to the data requester.

[0008] In this embodiment, when any controller group in the controller cluster receives a data query request sent by a data requester, it can determine the identification parameters associated with at least one target controller group in the controller cluster in the data query request. Then, the controller group forwards the received data query request to the target controller groups corresponding to each identification parameter to indicate that the target controller group can query the database that matches it and obtain the query result data associated with the data query request; thereafter, it receives the query result data returned by each target controller group, summarizes the received query result data, and returns it to the data requester, thereby completing the data processing work for the data query request. It can be seen that in the data processing method provided by the present application, any controller group in the controller cluster can forward the data query request to the target controller group associated with the data query request when receiving a data query request, so that the target controller group can directly query the query result data associated with the data query request from the matching database, thereby ensuring that the data query request can be accurately transmitted to the target controller group; thereby avoiding the situation where the query result data associated with the data query request cannot be queried in the database matched by the controller group receiving the data query request, and can also avoid the situation where the controller group receiving the data query request cannot forward the data query request to other controller groups that can query the query result data associated with the data query request, which is beneficial to improving the data processing accuracy and efficiency of the data query request; at the same time, the data processing method provided by the present application can use the controller group receiving the data query request as a transit point for data processing, for sending data query requests to the target controller group, and for receiving and summarizing the query result data returned by the target controller group, which is beneficial to improving the high availability of each controller group node in the controller cluster.

[0009] In one embodiment, the method further comprises:

[0010] In the event that determining the identification parameter based on the data query request fails, determining associated annotation data based on the data query request;

[0011] Determining target query data corresponding to the data query request based on the annotation data;

[0012] At least one target controller group is determined based on the target query data, and the data query request is forwarded to each target controller group; the data query request is used to instruct each target controller group to query its matching database to obtain query result data associated with each target query data.

[0013] In this embodiment, when the current controller group receiving the data query request cannot determine the identification parameters associated with any target controller group in the controller cluster from the data query request, it means that the data query request does not carry any valid identification parameters; at this time, it can be determined whether the data query request includes annotation data. If the data query request includes annotation data, the target query data associated with the data query request can be determined based on the annotation data, and then, based on the matching relationship between the target query data and the controller groups in the controller cluster, it can be determined based on which target controller groups each target query data can be queried and relevant query result data can be obtained; then, the data query request is forwarded to each target controller group through the current controller group to indicate that each target controller group can query the database matching it to obtain the query result data of each target query data associated with the data query request; thereafter, the query result data returned by each target controller group is received, and after summarizing the received query result data, it is returned to the data requester, thereby completing the data processing work for the data query request. It can be seen that the present application provides this embodiment, which provides another method for determining the target controller group associated with the data query request when the current controller group receiving the data query request finds that the data query request is not associated with any identification parameters of the target controller group, thereby facilitating improving the data processing capability and data processing efficiency of the controller cluster for data query requests. Specifically, in this embodiment, when a data query request includes annotation data, the target query data corresponding to the data query request is determined based on the annotation data, and then the target controller group corresponding to the data query request is determined based on the target query data. In this way, the current controller group can also forward the data query request to the target controller group associated with the data query request, so that the target controller group can directly query the matching database for the query result data associated with the data query request, thereby ensuring that the data query request can be accurately transmitted to the target controller group. This avoids the situation where the query result data associated with the data query request cannot be queried in the database matched by the controller group receiving the data query request, and also avoids the situation where the controller group receiving the data query request cannot forward the data query request to other controller groups that can query the query result data associated with the data query request, which is conducive to improving the data processing accuracy and efficiency of the data query request. At the same time, the data processing method provided by the present application can use the controller group receiving the data query request as a transit point for data processing to send the data query request to the target controller group and to receive and aggregate the query result data returned by the target controller group, which is conducive to improving the high availability of each controller group node in the controller cluster.

[0014] In one embodiment, determining at least one target controller group based on the target query data and forwarding the data query request to each target controller group includes:

[0015] Determining at least one associated target controller group based on each target query data;

[0016] generating a sub-query request corresponding to the data query request based on each target query data associated with the same target controller group;

[0017] Each sub-query request is forwarded to the associated target controller group.

[0018] In this embodiment, the current controller group first determines which target controller groups can be used to query and obtain relevant query result data for each target query data based on the matching relationship between each target query data and each controller group in the controller cluster, and then generates a data query request corresponding to a sub-query request of each target controller group based on each target query data associated with the same target controller group, and then forwards each sub-query request to the corresponding target controller group to indicate that each target controller group can query the database matching it to obtain the query result data of each target query data associated with the data query request; in this way, the data query request can be split into multiple sub-query requests through the current controller group, and then the corresponding associated sub-query requests can be forwarded to multiple target controller groups, which is beneficial to reducing the data volume of the query requests processed by each target controller group, thereby further improving the data processing accuracy and efficiency of the data query requests, and further improving the high availability of each controller group node in the controller cluster.

[0019] In one embodiment, the method further comprises:

[0020] In the case that the identification parameter fails to be determined based on the data query request and the annotation data fails to be determined, query result data corresponding to the data query request is acquired based on the current controller group.

[0021] In this embodiment, when the current controller group receiving the data query request is unable to determine the identification parameters associated with any target controller group in the controller cluster from the data query request, and the data query request does not include annotation data, the query result data corresponding to the data query request can be directly obtained based on the current controller group, that is, the current controller group directly queries the query result data of the data query request to its corresponding matching database. It can be seen that the present application provides this embodiment, which provides another method for querying related query result data based on the data query request when the current controller group receiving the data query request finds that the data query request does not have the identification parameters of any target controller group associated with it and does not include annotation data, thereby facilitating the data processing capability and efficiency of the controller cluster for the data query request.

[0022] In one embodiment, determining identification parameters of at least one target controller group based on the data query request includes:

[0023] A plurality of target parameters in the data query request is acquired, and an identification parameter of a target controller group is determined from the plurality of target parameters.

[0024] In this embodiment, the current controller group may first obtain multiple target parameters included in the data query request, and then filter out identification parameters associated with the target controller group from these multiple target parameters, thereby improving the efficiency and accuracy of determining the identification parameters.

[0025] In one embodiment, upon receiving a data query request, determining identification parameters of at least one target controller group based on the data query request includes:

[0026] In the case of receiving a data query request, identifying the type of the current controller receiving the data query request; the current controller belongs to a current controller group;

[0027] In a case where the type of the current controller is a master controller, identification parameters of at least one target controller group are determined based on the data query request.

[0028] In this embodiment, when the current controller group receives a data query request, it can first identify the type of the current controller in the current controller group that receives the data query request. When it is identified that the type of the current controller is the main controller, it can further determine the identification parameters associated with at least one target controller group in the controller cluster in the data query request; this is helpful to avoid the data query request being received from the controller in the current controller group, so as to ensure that the main controller that receives the data query request can further execute other steps in the data processing method provided in this application.

[0029] In one embodiment, the method further comprises:

[0030] In a case where the type of the current controller is a slave controller, the data query request is forwarded to the master controller in the current controller group.

[0031] In this embodiment, when the current controller group identifies that the type of the current controller receiving the data query request is a slave controller rather than a master controller, the data query request will be forwarded to the master controller in the current controller group; since the slave controller cannot execute other steps in the data processing method provided in this application, therefore, when the slave controller in the current controller receives the data query request, the data query request will be forwarded to the master controller, which is beneficial to ensure that the master controller that receives the data query request can further execute other steps in the data processing method provided in this application.

[0032] In one embodiment, the method further comprises:

[0033] When a master controller in a current controller group is identified to be faulty, updating any slave controller in the current controller group to be a new master controller; the current controller group includes one master controller and multiple slave controllers;

[0034] And update the log information of the failed main controller to the new main controller.

[0035] In this embodiment, any controller group includes a master controller and multiple slave controllers; when any controller group identifies that the master controller in the current controller group has failed, any slave controller in the current controller group can be updated to the new master controller in the current controller group, and the old master controller can be abandoned; at the same time, the log information of the failed old master controller is updated to the new master controller, so that the log information of the new master controller includes the log information of the old master controller, so that the new master controller can take over from the old master controller to process the data processing requests received by the current controller group, thereby ensuring the high availability of the current controller group.

[0036] In one embodiment, the method further comprises:

[0037] When the master controller in the current controller group receives any data update request, the log information of the data update request is sent to at least one slave controller in the current controller group; the current controller group includes one master controller and multiple slave controllers.

[0038] In this embodiment, any controller group includes a master controller and multiple slave controllers; when any controller group identifies the master controller in the current controller group and receives any data update request, it can send the log information corresponding to the data update request to at least one slave controller in the current controller group, so that the log information of at least one slave controller in the current controller group is the same as the log information of the master controller; this is beneficial in that when the slave controller is updated to the new master controller in the current controller group, the amount of update information of the log information of the old master controller to the new master controller can be reduced, thereby helping to improve the efficiency of the new master controller in processing the data processing requests received by the current controller group.

[0039] In a second aspect, the present application further provides a data processing device, applied to any controller group in a controller cluster, comprising:

[0040] an identification determination module configured to, upon receiving a data query request, determine an identification parameter of at least one target controller group based on the data query request; the controller cluster comprising a plurality of communicatively connected controller groups;

[0041] a request forwarding module, configured to forward the data query request to each of the target controller groups corresponding to the identification parameter; the data query request is used to instruct each of the target controller groups to query its matching database to obtain query result data associated with the data query request;

[0042] The result aggregation module is configured to return the aggregated query result data to the data requester upon receiving the query result data returned by at least one target controller group.

[0043] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect above when executing the computer program.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0045] In a fifth aspect, the present application further provides a computer program product, comprising a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0046] In the data processing method, apparatus, computer device, computer-readable storage medium, and computer program product provided by the present application, the data processing method enables any controller group in the controller cluster to forward the data query request to the target controller group associated with the data query request upon receiving the data query request, so that the target controller group can directly query the matching database for the query result data associated with the data query request, thereby ensuring that the data query request can be accurately transmitted to the target controller group; thereby avoiding the situation where the query result data associated with the data query request cannot be queried in the database matched by the controller group receiving the data query request, and also avoiding the situation where the controller group receiving the data query request cannot forward the data query request to other controller groups that can query the query result data associated with the data query request, which is conducive to improving the data processing accuracy and efficiency of the data query request; at the same time, the data processing method provided by the present application can use the controller group receiving the data query request as a transit point for data processing, for sending the data query request to the target controller group, and for receiving and aggregating the query result data returned by the target controller group, which is conducive to improving the high availability of each controller group node in the controller cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A diagram of an application environment of a data processing method in one embodiment;

[0049] Figure 2 1 is a flow chart of a data processing method in one embodiment;

[0050] Figure 3 1 is a schematic flow chart of sub-steps of a data processing method in one embodiment;

[0051] Figure 4 is a flow chart of a data processing method in another embodiment;

[0052] Figure 5 is a structural block diagram of a data processing device in one embodiment;

[0053] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0055] In today's large-scale distributed computing environments, scale-out is a critical technology to ensure systems can cope with growing loads. By adding new nodes, a system can expand its processing power, storage capacity, and service concurrency. However, as the number of nodes increases, ensuring data consistency and high availability across the system becomes a complex challenge.

[0056] Existing distributed systems typically implement data synchronization and consistency management through consensus protocols such as Raft or Paxos. However, traditional synchronization mechanisms often suffer from data inconsistencies and system performance degradation when faced with high-concurrency write operations or node failures. Furthermore, how to quickly and reliably synchronize data across large clusters is a major technical challenge.

[0057] In the event of network partitions or node failures, many existing technologies find it difficult to ensure data consistency while maintaining high system availability. In particular, when a "split brain" problem occurs in a distributed system, the consistency and availability of the system will be seriously affected.

[0058] As the number of nodes increases, traditional replication mechanisms such as Paxos (a distributed consensus algorithm based on message passing) and Quorum (a data consistency protocol in distributed systems) face performance bottlenecks and are unable to efficiently handle the synchronization and consistency issues of large numbers of nodes. Gossip (a communication protocol that allows state sharing in distributed systems), while offering good scalability, cannot meet strict consistency requirements.

[0059] In order to solve the above problems in the related art, the present application provides a data processing method that can solve the above technical problems. The data processing method can be applied to Figure 1 In the application environment shown, the controller cluster 100 includes multiple controller groups 10, each of which may include a master controller 11 and multiple slave controllers 12. In an optional embodiment provided by the present application, each controller group 10 includes three slave controllers 12; the multiple controller groups 10 in the controller cluster 100 are communicatively connected.

[0060] In the data processing method provided by the embodiment of the present application, any controller group in the controller cluster, when receiving a data query request, can determine the identification parameters of at least one target controller group based on the data query request; and then forward the data query request to each target controller group corresponding to the identification parameter; the data query request is used to instruct each target controller group to query its matching database to obtain query result data associated with the data query request; when the current controller group receives the query result data returned by at least one target controller group, it aggregates the data and then returns the aggregated query result data to the data requester to complete the data processing work for the data query request. This data processing method is beneficial to improving the high availability of each controller group node in the controller cluster by using the controller group that receives the data query request as a transit point for data processing, for sending data query requests to the target controller group, and for receiving and aggregating the query result data returned by the target controller group.

[0061] In an exemplary embodiment, Figure 2 As shown, a data processing method is provided, which is applied to Figure 1 Taking any controller group 10 in the example as an example, the following steps 201 to 203 are included.

[0062] Step 201 : upon receiving a data query request, determining identification parameters of at least one target controller group based on the data query request; a controller cluster includes a plurality of controller groups in communication connection.

[0063] The data query request may be a query request initiated by a user on a front-end page of a computer for results corresponding to specific data.

[0064] The identification parameter can be the ID (identity document) of each controller group in the controller cluster, or the number of each controller group. In other words, the identification parameter is the unique identifier of each controller group.

[0065] For example, when any controller group in a controller cluster receives a data query request from a data requester, an identification parameter associated with at least one target controller group in the controller cluster can be determined from the data query request. Each controller group has a fixed identification parameter, and the identification parameter corresponding to each controller group is different.

[0066] It should be noted that the identification parameter of the target controller group that can be determined by each data query request can be one or more, and this application does not make any specific limitation on this.

[0067] Step 202 : forwarding the data query request to each target controller group corresponding to the identification parameter; the data query request is used to instruct each target controller group to query its matching database to obtain query result data associated with the data query request.

[0068] Each controller group can be assigned a database, or multiple controller groups can be assigned a database. The data stored in each database can be different. Each controller group can only query data from its corresponding database and retrieve the query result data associated with the query request stored in the matching database.

[0069] Exemplarily, the current controller group forwards the received data query request to the target controller group corresponding to each identification parameter to instruct the target controller group to query the database matching it and obtain query result data associated with the data query request.

[0070] After querying the query result data associated with the data query request in each database, the relevant query result data will be sent to the corresponding matching target controller group first, and then each target controller group will return the query result data to the current controller group.

[0071] Step 203: upon receiving query result data returned by at least one target controller group, returning the aggregated query result data to the data requester.

[0072] Exemplarily, after receiving the query result data returned by each target controller group, the current controller group summarizes the received query result data, and then returns the summarized data to the data requester, so that the data requester (such as the front-end page) can receive the required query result data and display it to the user, thereby completing the data processing work for the data query request.

[0073] In the data processing method provided by the present application, any controller group in the controller cluster, upon receiving a data query request, can forward the data query request to the target controller group associated with the data query request, so that the target controller group can directly query the query result data associated with the data query request from the matching database, thereby ensuring that the data query request can be accurately transmitted to the target controller group; thereby avoiding the situation where the query result data associated with the data query request cannot be queried in the database matched by the controller group receiving the data query request, and can also avoid the situation where the controller group receiving the data query request cannot forward the data query request to other controller groups that can query the query result data associated with the data query request, which is beneficial to improving the data processing accuracy and efficiency of the data query request; at the same time, the data processing method provided by the present application can use the controller group receiving the data query request as a transit point for data processing, for sending data query requests to the target controller group, and for receiving and summarizing the query result data returned by the target controller group, which is beneficial to improving the high availability of each controller group node in the controller cluster.

[0074] That is, the data processing method provided by the present application can realize that when any controller group receives a business request, the relevant request will be forwarded to the corresponding target controller group; and when performing data query, the information of multiple controller groups can be summarized.

[0075] In an exemplary embodiment, Figure 3 As shown, the data processing method provided by the present application may further include the following steps 301 to 303.

[0076] Step 301 : When determining the identification parameter based on the data query request fails, determining the associated annotation data based on the data query request.

[0077] The annotation data is the annotation on the interface corresponding to the data query request, which can be a custom annotation added to the interface. In this application, the annotation data includes some data related to multiple target parameters included in the data query request. The existence of annotation data means that it is necessary to first query the metadata to obtain the data to be queried as requested by the user. The specific distribution of these data to be queried in the databases matching the controller groups is then used to query the results related to the data to be queried from the matching databases based on each controller group.

[0078] For example, when the current controller group receiving the data query request cannot determine the identification parameters associated with any target controller group in the controller cluster from the data query request, it means that the data query request does not carry any valid identification parameters; at this time, it can be determined whether the data query request includes annotation data.

[0079] Step 302: Determine target query data corresponding to the data query request based on the annotation data.

[0080] Among them, the annotation data includes relevant data of multiple target parameters included in some data query requests. Specifically, the annotation data may include the corresponding target query data among the multiple target parameters included in the data query request, such as the data to be queried entered by the user in the front-end page.

[0081] For example, when the current controller group determines that the data query request includes annotation data, the target query data associated with the data query request may be determined based on the annotation data.

[0082] Step 303: determine at least one target controller group based on the target query data, and forward a data query request to each target controller group; the data query request is used to instruct each target controller group to query its matching database to obtain query result data associated with each target query data.

[0083] There is an association relationship between the target query data and the target controller group, or there is an association relationship between the target query data and the database that matches the target controller group. The relevant association relationship can be, for example, that any target query data corresponds to a corresponding target controller group, or any target query data corresponds to a corresponding database.

[0084] Exemplarily, the current controller group can determine which target controller groups can be used to query and obtain relevant query result data for each target query data based on the matching relationship between the target query data and the controller groups in the controller cluster, and then forward the data query request to each target controller group through the current controller group to indicate that each target controller group can query the database that matches it to obtain the query result data of each target query data associated with the data query request; thereafter, receive the query result data returned by each target controller group, summarize the received query result data, and return it to the data requester, thereby completing the data processing work for the data query request.

[0085] In this embodiment, when the current controller group receiving the data query request finds that the data query request is not associated with any identification parameters of the target controller group, another method for determining the target controller group associated with the data query request is provided, which is beneficial to improving the data processing capability and data processing efficiency of the controller cluster for data query requests.

[0086] In this embodiment, when a data query request includes annotation data, the target query data corresponding to the data query request is determined based on the annotation data, and then the target controller group corresponding to the data query request is determined based on the target query data. In this way, the current controller group can also forward the data query request to the target controller group associated with the data query request, so that the target controller group can directly query the matching database for the query result data associated with the data query request, thereby ensuring that the data query request can be accurately transmitted to the target controller group. This avoids the situation where the query result data associated with the data query request cannot be queried in the database matched by the controller group receiving the data query request, and also avoids the situation where the controller group receiving the data query request cannot forward the data query request to other controller groups that can query the query result data associated with the data query request, which is conducive to improving the data processing accuracy and efficiency of the data query request. At the same time, the data processing method provided by the present application can use the controller group receiving the data query request as a transit point for data processing to send the data query request to the target controller group and to receive and aggregate the query result data returned by the target controller group, which is conducive to improving the high availability of each controller group node in the controller cluster.

[0087] In addition, an optional embodiment is provided, in which, when the identification parameters of at least one target controller group can be determined based on a data query request, the priority of the data query request that can determine the identification parameters can be set to the highest, so that when the controller cluster receives multiple data query requests or multiple data processing requests, it can first process the data query request that includes the identification parameters.

[0088] In an exemplary embodiment, the above steps of determining at least one target controller group based on the target query data and forwarding the data query request to each target controller group may specifically include: determining at least one associated target controller group based on each target query data; generating a sub-query request corresponding to the data query request based on each target query data associated with the same target controller group; and forwarding each sub-query request to the associated target controller group.

[0089] The sub-query request refers to at least two sub-query requests split based on the data query request; each sub-query request includes relevant data to be queried (target query data) that needs to be queried through the associated target controller group.

[0090] Exemplarily, the current controller group first determines which target controller groups can be used to query and obtain relevant query result data for each target query data based on the matching relationship between each target query data and each controller group in the controller cluster, and then, based on each target query data associated with the same target controller group, generates a data query request corresponding to a sub-query request for each target controller group, and then forwards each sub-query request to the corresponding target controller group to indicate that each target controller group can query the database that matches it, and obtain the query result data of each target query data associated with the data query request.

[0091] In this embodiment, the data query request is split into multiple sub-query requests by the current controller group, and then the corresponding associated sub-query requests are forwarded to multiple target controller groups, which is beneficial to reducing the amount of data of the query requests processed by each target controller group, thereby further improving the data processing accuracy and efficiency of the data query requests, and further improving the high availability of each controller group node in the controller cluster.

[0092] In an exemplary embodiment, the data processing method provided by the present application further includes: when determining the identification parameter based on the data query request fails and determining the annotation data fails, obtaining query result data corresponding to the data query request based on the current controller group.

[0093] For example, when the current controller group receiving the data query request is unable to determine the identification parameters associated with any target controller group in the controller cluster from the data query request, and the data query request does not include annotation data, it means that the currently received data query request only needs to be processed in the current controller group, and does not involve the situation where the data is distributed on databases matching other different controller groups; at this time, the query result data corresponding to the data query request can be directly obtained based on the current controller group, that is, the current controller group directly queries the query result data of the data query request to its corresponding matching database.

[0094] The present application provides this embodiment, which provides another method for querying related query result data based on the data query request when the current controller group receiving the data query request finds that the data query request does not have any identification parameters associated with any target controller group and does not contain annotation data, thereby facilitating the data processing capability and data processing efficiency of the controller cluster for the data query request.

[0095] In an exemplary embodiment, determining the identification parameter of at least one target controller group based on the data query request in the above step may specifically include: acquiring multiple target parameters in the data query request, and determining the identification parameter of the target controller group from the multiple target parameters.

[0096] The data query request may include multiple target parameters, and the multiple target parameters may include at least target query data and an identification parameter, where the identification parameter is an identification parameter corresponding to the controller group.

[0097] In this embodiment, by setting the current controller group to first obtain multiple target parameters included in the data query request, and then filtering out identification parameters associated with the target controller group from these multiple target parameters, the efficiency and accuracy of determining the identification parameters can be improved; and this is also beneficial to improving the accuracy of the query result data finally obtained.

[0098] In an exemplary embodiment, in the above steps, when a data query request is received, determining the identification parameters of at least one target controller group based on the data query request may specifically include: when a data query request is received, identifying the type of the current controller receiving the data query request; the current controller belongs to the current controller group; when the type of the current controller is a master controller, determining the identification parameters of at least one target controller group based on the data query request.

[0099] As mentioned above, each controller group may include a master controller and multiple slave controllers. Therefore, the types of controllers may include master controllers and slave controllers.

[0100] For example, when the current controller group receives a data query request, it can first identify the type of the current controller in the current controller group that receives the data query request. When it is identified that the type of the current controller is a master controller, it can further determine the identification parameters associated with at least one target controller group in the controller cluster in the data query request, and perform subsequent query acquisition based on the query result data related to the data query request based on the target controller.

[0101] Since the slave controller is not capable of executing subsequent query acquisition steps based on the target controller for the query result data related to the data query request, the present application is arranged such that when the current controller receiving the data query request in the current controller group is the main controller, the associated identification parameters are determined from the data query request, and subsequent query acquisition of the query result data related to the data query request based on the target controller is executed. This is beneficial to avoid the situation where the slave controller in the current controller group receives the data query request and is unable to execute the query acquisition action for the query result data, thereby ensuring that the main controller receiving the data query request can further execute other steps in the data processing method provided by the present application.

[0102] In an exemplary embodiment, the data processing method provided by the present application further includes: when the type of the current controller is a slave controller, forwarding the data query request to the master controller in the current controller group.

[0103] In this embodiment, when the current controller group identifies that the type of the current controller receiving the data query request is a slave controller rather than a master controller, the data query request will be forwarded to the master controller in the current controller group; since the slave controller cannot execute other steps in the data processing method provided in this application, therefore, when the slave controller in the current controller receives the data query request, the data query request will be forwarded to the master controller, which is beneficial to ensure that the master controller that receives the data query request can further execute other steps in the data processing method provided in this application.

[0104] In an exemplary embodiment, the data processing method provided in the present application also includes: when a master controller failure in the current controller group is identified, updating any slave controller in the current controller group to be a new master controller; the current controller group includes a master controller and multiple slave controllers; and updating the log information of the failed master controller to the new master controller.

[0105] In this embodiment, when any controller group identifies that the main controller in the current controller group has failed, any slave controller in the current controller group can be updated to the new main controller in the current controller group, and the old main controller can be abandoned; at the same time, the log information of the failed old main controller is updated to the new main controller, so that the log information of the new main controller includes the log information of the old main controller, so that the new main controller can take over from the old main controller to process the data processing requests received by the current controller group, thereby ensuring the high availability of the current controller group.

[0106] An optional embodiment is provided in which the present application sets each controller group node to be composed of master-slave nodes (master controller and slave controller). When the master controller in a controller group goes offline due to network failure or other reasons, the computer system can immediately detect the failure and notify any slave node (slave controller) in the controller group to be promoted to the new master node (master controller). At the same time, the log entries backed up from the old master node (master controller) are updated to the new master controller, thereby ensuring the availability of each controller group in the controller cluster.

[0107] This application achieves high system availability by introducing a one-master-multiple-slave mechanism (one master controller and multiple slave controllers) within a controller group and an automatic failover mechanism. This ensures that the system can continue to provide services even if any node fails or the network is unstable. This allows the system to maintain business continuity in various abnormal situations.

[0108] In an exemplary embodiment, the data processing method provided in the present application also includes: when the master controller in the current controller group receives any data update request, sending the log information of the data update request to at least one slave controller in the current controller group; the current controller group includes a master controller and multiple slave controllers.

[0109] In this embodiment, by identifying the master controller in the current controller group in any controller group and receiving any data update request, the log information corresponding to the data update request can be sent to at least one slave controller in the current controller group, so that the log information of at least one slave controller in the current controller group is the same as the log information of the master controller; this is beneficial in that when the slave controller is updated to the new master controller in the current controller group, the amount of update information of the log information of the old master controller to the new master controller can be reduced, thereby helping to improve the efficiency of the new master controller in processing the data processing requests received by the current controller group.

[0110] In an alternative embodiment, upon receiving a data change operation, any master controller may replicate each received data change operation as a log entry to all slave controllers in the master controller's corresponding controller group. When more than half of the slave nodes (slave controllers) confirm receipt of a log entry sent by the master controller, the entry is considered committed.

[0111] This application utilizes shared log replication across multiple slave nodes within the same controller group, significantly reducing the need for data resynchronization during master-slave failures. This eliminates the need for slave nodes to fully synchronize the master's log data after a master failure. By incrementally resynchronizing, the controller group's full data processing capabilities can be quickly restored.

[0112] For the data processing method provided in this application, this application also provides an optional implementation method, including the following steps, wherein the controller group 1 is used as the current controller group; the main controller in the current controller group is, for example, the main controller A. Figure 4 ,in:

[0113] When controller group 1 receives a data query request, it encapsulates the received data query request based on the Filter function to obtain an encapsulated request; wherein the Filter function can filter a series of data based on defined conditions;

[0114] In controller group 1, determine the main controller A; this step can be implemented based on the main interceptor selection;

[0115] Determine whether the encapsulated request is sent to the main controller A;

[0116] When the encapsulated request is sent to the main controller A, it is determined whether the encapsulated request includes the target controller group ID (identification parameter);

[0117] In the case where the encapsulated request includes the target controller group ID, determining whether the ID points to controller group 1;

[0118] When the ID identifier points to controller group 1, the corresponding target data (query result data) is extracted and encapsulated from the associated database through controller group 1;

[0119] If the encapsulated request is not sent to the main controller A, forward the encapsulated request to the main controller A;

[0120] If the target controller group ID is not included in the encapsulated request, determine whether the annotation data of the encapsulated request can be obtained;

[0121] In the case where the annotation data of the encapsulated request is not obtained, the target data corresponding to the encapsulated request is retrieved from the associated database through controller group 1;

[0122] When the annotation data of the encapsulated request is obtained, metadata is determined based on the annotation data, the target controller group where the data to be requested is located is determined based on the metadata, and target data corresponding to each data to be requested is obtained from the matching database through each target controller group;

[0123] When the ID identifier does not point to the controller group 1, the encapsulated request is forwarded to the target controller group pointed to by the ID identifier.

[0124] It should be noted that Figure 4The “Y” in the table represents “yes”, that is, the judgment result is “yes”; the “N” represents “no”, that is, the judgment result is “no”.

[0125] Among them, all steps after the determination result is that the encapsulated request is sent to the main controller A can be implemented based on the cluster forwarding interceptor.

[0126] As can be seen, in the data processing method provided by this application, when performing data queries, one optional implementation method is to obtain basic data through cluster metadata during the query, and then use the unique identifier of each controller group to query and summarize the full data and return it to the user. Among them, metadata management refers to providing the annotations required for metadata, annotation parsing, metadata configuration cache, and metadata data structure.

[0127] This application is equivalent to proposing a highly available scale-out data synchronization management method based on the Raft consensus algorithm, aiming to overcome the limitations of related technologies; to provide a data synchronization solution that can not only ensure data consistency but also have good scalability and fault tolerance.

[0128] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0129] Based on the same inventive concept, the present application also provides a data processing device 500 for implementing the aforementioned data processing method. The implementation solution provided by the device 500 is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the data processing device 500 provided below can be found in the above-mentioned limitations of the data processing method and will not be repeated here.

[0130] In an exemplary embodiment, Figure 5 As shown, a data processing device 500 is provided, comprising: an identification determination module 501, a request forwarding module 502 and a result aggregation module 503, wherein:

[0131] The identification determination module 501 is configured to determine identification parameters of at least one target controller group based on the data query request when a data query request is received; the controller cluster includes multiple controller groups in communication connection;

[0132] The request forwarding module 502 is used to forward the data query request to each target controller group corresponding to the identification parameter; the data query request is used to instruct each target controller group to query its matching database to obtain query result data associated with the data query request;

[0133] The result aggregation module 503 is configured to return the aggregated query result data to the data requester upon receiving the query result data returned by at least one target controller group.

[0134] In one embodiment, the data processing device 500 also includes an annotation determination module and a target query data determination module, wherein the annotation determination module is used to determine the associated annotation data based on the data query request when the identification parameters fail to be determined based on the data query request; the target query data determination module is used to determine the target query data corresponding to the data query request based on the annotation data; the request forwarding module 502 is used to determine at least one target controller group based on the target query data and forward the data query request to each target controller group; the data query request is used to instruct each target controller group to query its matching database to obtain query result data associated with each target query data.

[0135] In one embodiment, the request forwarding module 502 is also used to determine at least one associated target controller group based on each target query data; generate a sub-query request corresponding to the data query request based on each target query data associated with the same target controller group; and forward each sub-query request to the associated target controller group.

[0136] In one embodiment, the result aggregation module 503 is further configured to obtain query result data corresponding to the data query request based on the current controller group when determining the identification parameter based on the data query request fails and determining the annotation data fails.

[0137] In one embodiment, the identification determination module 501 is further configured to obtain multiple target parameters in the data query request, and determine the identification parameter of the target controller group from the multiple target parameters.

[0138] In one embodiment, the identification determination module 501 is also used to identify the type of the current controller receiving the data query request when a data query request is received; the current controller belongs to the current controller group; when the type of the current controller is the main controller, determine the identification parameters of at least one target controller group based on the data query request.

[0139] In one embodiment, the request forwarding module 502 is further configured to forward the data query request to a master controller in the current controller group when the current controller is a slave controller.

[0140] In one embodiment, the data processing device 500 also includes a controller group update module, which is used to update any slave controller in the current controller group to a new master controller when a master controller failure in the current controller group is identified; the current controller group includes a master controller and multiple slave controllers; and update the log information of the failed master controller to the new master controller.

[0141] In one embodiment, the controller group update module is also used to send log information of the data update request to at least one slave controller in the current controller group when the master controller in the current controller group receives any data update request; the current controller group includes a master controller and multiple slave controllers.

[0142] Each module in the data processing device 500 may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0143] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, a mobile cellular network, near-field communication (NFC), or other technologies. When executed by the processor, the computer program implements a data processing method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0144] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0145] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the data processing method provided in the present application are implemented.

[0146] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the data processing method provided in the present application are implemented.

[0147] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the data processing method provided in the present application when executed by a processor.

[0148] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0149] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0150] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0151] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data processing method, characterized in that: Applicable to any controller group in a controller cluster, each controller group includes a master controller and multiple slave controllers; the method includes: Upon receiving a data query request, identifying the type of a current controller receiving the data query request, where the current controller belongs to a current controller group; if the type of the current controller is the master controller, determining an identification parameter of at least one target controller group based on the data query request, and setting the priority of the data query request corresponding to the identification parameter to the highest; the controller cluster includes a plurality of controller groups in communication connection; Forwarding the data query request to each of the target controller groups corresponding to the identification parameter; the data query request is used to instruct each of the target controller groups to query its matching database to obtain query result data associated with the data query request; In the event that determining the identification parameter based on the data query request fails, determining associated annotation data based on the data query request; determining target query data corresponding to the data query request based on the annotation data; determining at least one target controller group based on the target query data, and forwarding the data query request to each of the target controller groups; In a case where determining the identification parameter based on the data query request fails and determining the annotation data fails, acquiring query result data corresponding to the data query request based on the current controller group; Upon receiving the query result data returned by at least one target controller group, returning the aggregated query result data to the data requester; Among them, determining at least one target controller group based on the target query data and forwarding the data query request to each target controller group includes: determining at least one target controller group associated with each target query data based on the matching relationship between each target query data and each controller group in the controller cluster; generating a sub-query request of the data query request corresponding to each target controller group based on each target query data associated with the same target controller group; and forwarding each sub-query request to the associated target controller group.

2. The method according to claim 1, characterized in that The determining of identification parameters of at least one target controller group based on the data query request includes: A plurality of target parameters in the data query request is acquired, and an identification parameter of a target controller group is determined from the plurality of target parameters.

3. The method according to claim 1, characterized in that The method further comprises: In a case where the type of the current controller is a slave controller, the data query request is forwarded to the master controller in the current controller group.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: When a master controller in a current controller group is identified to be faulty, updating any slave controller in the current controller group to be a new master controller; the current controller group includes one master controller and multiple slave controllers; And update the log information of the failed main controller to the new main controller.

5. The method according to any one of claims 1 to 3, characterized in that The method further comprises: When the master controller in the current controller group receives any data update request, the log information of the data update request is sent to at least one slave controller in the current controller group; the current controller group includes one master controller and multiple slave controllers.

6. A data processing device, characterized in that: Applicable to any controller group in a controller cluster, each controller group includes a master controller and multiple slave controllers; the device includes: an identification determination module for, upon receiving a data query request, identifying the type of a current controller receiving the data query request, wherein the current controller belongs to a current controller group; and, if the type of the current controller is the master controller, determining an identification parameter of at least one target controller group based on the data query request, and setting the priority of the data query request corresponding to the identification parameter to the highest; wherein the controller cluster includes a plurality of communicatively connected controller groups; a request forwarding module, configured to forward the data query request to each of the target controller groups corresponding to the identification parameter; the data query request is used to instruct each of the target controller groups to query its matching database to obtain query result data associated with the data query request; An annotation determination module is configured to determine associated annotation data based on the data query request if the identification parameter fails to be determined based on the data query request; a target query data determining module, configured to determine target query data corresponding to the data query request based on the annotation data; The request forwarding module is further configured to determine at least one target controller group based on the target query data, and forward the data query request to each of the target controller groups; A result aggregation module is configured to obtain query result data corresponding to the data query request based on the current controller group when determining the identification parameter and the annotation data based on the data query request fails; The result aggregation module is further configured to return the aggregated query result data to the data requester upon receiving the query result data returned by at least one target controller group; Among them, the request forwarding module is used to determine at least one target controller group based on the target query data, and forward the data query request to each target controller group, including: based on the matching relationship between each target query data and each controller group in the controller cluster, determining at least one target controller group associated with each target query data; based on each target query data associated with the same target controller group, generating a sub-query request of the data query request corresponding to each target controller group; and forwarding each sub-query request to the associated target controller group.

Images