Data processing system and method for distributed cluster in weak cloud environment

Abstract

The invention discloses a data processing system and method for a distributed cluster in a weak cloud environment, and relates to the technical field of data processing, the system comprises at least two service data links, and each service data link comprises a service data source, a service data processing module, a service data writing module and a service data cluster; when the service data processing module obtains a service processing request, quasi-real-time processing is carried out based on service system source data sent by a service data source, and a data processing result is obtained; the data processing result is sent to a service data writing module and another service data link; and the service data write-in module writes a data processing result into the service data cluster when receiving the data processing result sent by the service data processing module and / or another service data link. Due to the fact that the service data link is active-active, user data missing caused by node faults during interface query is avoided, and the efficiency and reliability of the data processing system are guaranteed.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing system and method for a distributed cluster in a weak cloud environment. Background Art

[0002] The HBase distributed database provides support for massive data and can read and write large-scale data sets in real time. It is highly scalable and economical, and reduces the operating costs of excessive relational database instances caused by table and database sharding when large amounts of data are present.

[0003] However, when using HBase, when a node in the cluster fails or a Region is unavailable, it is necessary to wait until the Region migration is completed before providing services to the outside world. This process is related to the amount of data in the Region and generally takes several minutes or even hours. Summary of the invention

[0004] The main purpose of this application is to provide a data processing system and method for a distributed cluster in a weak cloud environment, aiming to solve the technical problem in the prior art that the interface access performance deteriorates when some nodes in the cluster fail during the use of HBase.

[0005] To achieve the above object, the present application proposes a data processing system for a distributed cluster in a weak cloud environment, the data processing system comprising: at least two business data links, each of the business data links comprising: a business data source, a business data processing module, a business data writing module and a business data cluster;

[0006] The business data processing module is used to perform quasi-real-time processing based on the business system source data sent by the business data source when a business processing request is obtained to obtain a data processing result;

[0007] The business data processing module is further used to send the data processing result to the business data writing module and another business data link;

[0008] The business data writing module is further used to write the data processing result into the business data cluster when receiving the data processing result sent by the business data processing module and / or the other business data link.

[0009] In one embodiment, the business data processing module is further used to obtain business rules;

[0010] The business data processing module is further used to generate data timestamp information according to business rules when performing quasi-real-time processing on the business system source data;

[0011] The business data processing module is also used to add the data timestamp information to the data processing results obtained by quasi-real-time processing based on the business system source data.

[0012] In one embodiment, the data processing system is further configured to obtain a first data processing result from the business data cluster and a second data processing result from the other business data link when obtaining a data acquisition request from a user;

[0013] The data processing system is further used to obtain first data timestamp information in the first data processing result and second data timestamp information in the second data processing result;

[0014] The data processing system is further used to compare the first data timestamp information with the second data timestamp information, and determine the target data timestamp information according to the comparison result;

[0015] The data processing system is further used to use the data processing result corresponding to the target data timestamp information as the request response result of the data acquisition request.

[0016] In one embodiment, the service data chain further includes: a fuse control module;

[0017] The fuse control module is used to obtain the fuse information of the service data cluster in another service data chain when the data processing system obtains the data acquisition request of the user if there is a node abnormality in the service data cluster;

[0018] The fusing control module is further used to determine the fusing state of a service data cluster in another service data chain according to the fusing information;

[0019] The fuse control module is further used to obtain the fuse logic of the service data cluster when the fuse state is not blown;

[0020] The fuse control module is further used to perform fuse processing on the service data cluster based on the fuse logic.

[0021] In one embodiment, the fuse control module is further used to detect the activity of a service data cluster in another service data chain when the fuse state is blown;

[0022] The fuse control module is also used to obtain the fuse logic of the business data cluster when the detection is successful.

[0023] In one embodiment, the fuse control module is further used to record the fuse start time and the fuse sleep time when the fuse state of the service data cluster enters the fuse state;

[0024] The fuse control module is further used to determine the fuse judgment time based on the fuse sleep time and the fuse sleep time;

[0025] The fuse control module is further configured to keep the fuse state of the service data cluster in the state of being blown when the fuse judgment time is less than the current time.

[0026] In addition, to achieve the above purpose, the present application also proposes a data processing method for a weak cloud environment distributed cluster based on the data processing system for a weak cloud environment distributed cluster as described above, the method comprising:

[0027] When the business data processing module obtains the business processing request, it performs quasi-real-time processing based on the business system source data sent by the business data source to obtain the data processing result;

[0028] The business data processing module sends the data processing result to the business data writing module and another business data link;

[0029] When receiving the data processing result sent by the business data processing module and / or the other business data link, the business data writing module writes the data processing result into the business data cluster.

[0030] In one embodiment, when executing the step of performing quasi-real-time processing on the business system source data sent by the business data source to obtain the data processing result, the method further includes:

[0031] The business data processing module generates data timestamp information according to business rules;

[0032] The business data processing module adds the data timestamp information to the data processing result obtained by performing quasi-real-time processing on the business system source data.

[0033] In one embodiment, the method further comprises:

[0034] When the data processing system obtains the data acquisition request of the user, the data processing system obtains the first data processing result and the second data processing result from the business data cluster and the other business data link respectively;

[0035] The data processing system obtains first data timestamp information in the first data processing result and second data timestamp information in the second data processing result;

[0036] The data processing system is further used to compare the first data timestamp information with the second data timestamp information, and determine the target data timestamp information according to the comparison result;

[0037] The data processing system uses the data processing result corresponding to the target data timestamp information as the request response result of the data acquisition request.

[0038] In one embodiment, the method further comprises:

[0039] When the data processing system obtains a data acquisition request from a user, the fuse control module obtains fuse information of a service data cluster in another service data chain if a node abnormality exists in the service data cluster;

[0040] The fusing control module determines the fusing state of a service data cluster in another service data chain according to the fusing information;

[0041] The fuse control module obtains the fuse logic of the service data cluster when the fuse state is not blown;

[0042] The fusing control module performs fusing processing on the first service data cluster based on the fusing logic.

[0043] One or more technical solutions proposed in this application have at least the following technical effects:

[0044] The data processing system of the weak cloud environment distributed cluster of the embodiment of the present application includes at least two business data links, each of which includes: a business data source, a business data processing module, a business data writing module and a business data cluster; wherein, when the business data processing module obtains a business processing request, it performs quasi-real-time processing based on the business system source data sent by the business data source to obtain a data processing result; the data processing result is sent to the business data writing module and another business data link; when the business data writing module receives the data processing result sent by the business data processing module and / or another business data link, the data processing result is written to the business data cluster. Since it is active-active through the business data link, the loss of user data during interface query due to node failure is avoided, and the efficiency and reliability of the data processing system of the present application are guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1A functional flow chart of the first embodiment of the data processing system for a distributed cluster in a weak cloud environment of the present application;

[0048] Figure 2 A system working block diagram provided for the third embodiment of the data processing system for a distributed cluster in a weak cloud environment of the present application;

[0049] Figure 3 This is a flow chart of an implementation method of a data processing system for a distributed cluster in a weak cloud environment of the present application;

[0050] Figure 4 A schematic diagram of a workflow provided for Embodiment 1 of the method for processing data in a distributed cluster in a weak cloud environment according to an embodiment of the present application.

[0051] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The data processing system of the weak cloud environment distributed cluster in the embodiment of the present application includes at least two business data links, each of which includes: a business data source, a business data processing module, a business data writing module and a business data cluster; wherein the business data processing module is used to perform quasi-real-time processing on the business system source data sent by the business data source when a business processing request is obtained to obtain a data processing result; the business data processing module is also used to send the data processing result to the business data writing module and another business data link; the business data writing module is also used to write the data processing result to the business data cluster when receiving the data processing result sent by the business data processing module and / or the other business data link.

[0055] With current existing technologies, when a node in a cluster fails or a Region is unavailable, it is necessary to wait until the Region migration is completed before providing services to the outside world. This process is related to the amount of data in the Region and generally takes several minutes or even hours.

[0056] This application provides a solution, which is to avoid the loss of user data during interface query due to node failure by building a data processing system of at least two business data chains. In addition, this application uses bilateral parallel reading and customized fuse strategy to quickly remove the faulty cluster and ensure that user access is not affected as much as possible.

[0057] Based on this, the embodiment of the present application provides a data processing system for a distributed cluster in a weak cloud environment, referring to Figure 1 , Figure 1 This is a system structure block diagram of the first embodiment of the data processing system of the weak cloud environment distributed cluster of the present application.

[0058] The data processing system in the embodiment of the present application may include at least two business data links. In one implementation of the embodiment of the present application, mirroring may be performed based on the business data links in the system to obtain other business data links in the data processing system.

[0059] For ease of description, the data processing system of the distributed cluster in a weak cloud environment of the present application is described in detail in the embodiment of the present application and the following embodiments, taking two business data links as an example.

[0060] In the embodiment of the present application, each of the business data links includes: a business data source, a business data processing module, a business data writing module and a business data cluster;

[0061] Specifically, based on these two service data links, a dual-active data processing system can be formed. For ease of description, in the embodiment of the present application, these two service data links can be set as a first service data link and a second service data link (the second service data link can also be referred to as another service data link).

[0062] It should be noted that in the embodiment of the present application, the entire business data link from the business data source to the HBase cluster (that is, the business data cluster) applies the active-active technology, so that the system of the present application has the advantages of high applicability and high performance under the distributed cluster, and has great reference significance for the construction of high-availability solutions in weak cloud environments.

[0063] It should be understood that the second business data link in the embodiment of the present application can be regarded as a mirror link of the first business data link, has the same structure as the first business data link, and realizes the same function as the first business data link. It can be understood that the first business data link can include: a first business data source, a first business data processing module, a first business data writing module, and a first business data cluster; the second business data link can include a second business data source, a second business data processing module, a second business data writing module, and a second business data cluster.

[0064] It should be noted that, for the sake of convenience, in the embodiments of the present application, when describing the relevant modules or generated data in the first service data link, the form of not adding the label "first" may be adopted. In order to distinguish, the relevant modules or generated data in the second service data link are all added with the label "second" or "another" to form a distinction from the modules or data in the first service data link.

[0065] It can be understood that active-active is a high-availability and data redundancy architecture model, which refers to the real-time mirroring of key data and application status through efficient and reliable data replication and synchronization technology between two or more geographically independent data centers or clusters.

[0066] The business data processing module is used to perform quasi-real-time processing based on the business system source data sent by the business data source when a business processing request is obtained to obtain a data processing result;

[0067] The business data processing module is further used to send the data processing result to the business data writing module and another business data link;

[0068] The business data writing module is further used to write the data processing result into the business data cluster when receiving the data processing result sent by the business data processing module and / or the other business data link.

[0069] It is understandable that the above-mentioned business data cluster refers to a high-performance computing system composed of multiple computers or servers. In the business data cluster, these computers / services can be connected together through a network or other means to achieve specific business data processing tasks. The above-mentioned business data processing module is a module or system that can perform a series of functions such as processing, conversion, analysis and mining of business data. The above-mentioned business data writing module is also a module that can write the data processing results or original data processed by the business data processing module into the business data cluster.

[0070] It should be understood that the above-mentioned business data source is also a computing module or server cluster that provides business data for the business data cluster, such as other databases, file systems, or logs generated by applications, or public data sets, data received by API interfaces, etc. The embodiments of the present application are not limited to this.

[0071] It should be noted that in the embodiments of the present application, quasi-real-time processing means that when the business data sent by the business data source is obtained, it is processed and analyzed at a speed close to real-time. Compared with real-time processing, business data processing through quasi-real-time processing can quickly respond to data changes while allowing a certain delay, thereby improving the scope of application of the system of the present application.

[0072] The embodiment of the present application designs a dual-active and data mirror synchronization cluster to reduce the impact on user access when some nodes of the distributed database fail, and introduces a custom fuse strategy.

[0073] It can be understood that when the business data processing module of the embodiment of the present application processes the business data in quasi-real time, it will synchronize the obtained data processing results to the data writing module in each business data link in the data processing system, so that the data writing module of each business data link can write the received data processing results to the business data cluster corresponding to the data link, thereby realizing data backup and disaster recovery processing of the data cluster, and avoiding the problems of data loss and slow data feedback due to node failure.

[0074] The data processing system of the weak cloud environment distributed cluster of the embodiment of the present application includes at least two business data links, each of which includes: a business data source, a business data processing module, a business data writing module and a business data cluster; wherein, when the business data processing module obtains a business processing request, it performs quasi-real-time processing based on the business system source data sent by the business data source to obtain a data processing result; the data processing result is sent to the business data writing module and another business data link; when the business data writing module receives the data processing result sent by the business data processing module and / or another business data link, the data processing result is written to the business data cluster. Since it is active-active through the business data link, the loss of user data during interface query due to node failure is avoided, and the efficiency and reliability of the data processing system of the present application are guaranteed.

[0075] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated later. On this basis, in the embodiment of the present application, the business data processing module is also used to obtain business rules;

[0076] The business data processing module is further used to generate data timestamp information according to business rules when performing quasi-real-time processing on the business system source data;

[0077] The business data processing module is also used to add the data timestamp information to the data processing results obtained by quasi-real-time processing based on the business system source data.

[0078] It should be noted that the above business rules are also preset rules based on data processing requirements, and the logic of data processing and the rules for generating timestamps can be determined based on business rules. Through business rules, data synchronization and consistency requirements between different business data links can be met to ensure that data between two or more data clusters is consistent in time, which facilitates timely and seamless switching when a node in a business data cluster fails.

[0079] It is understandable that when the quasi-real-time processing is completed, the generated data timestamp information can be added to the generated data processing result, and a mirror can be established based on the data processing result, and the mirror data can be synchronized to another data processing link.

[0080] It should be noted that when the first business data writing module in the first business data link receives the first data processing result sent by the first business data processing module and / or the second data processing result sent by the second business data processing module, the first business data processing result and / or the second business data processing result can be consumed and written into the first business data cluster. When writing data into the business data cluster, the data timestamp information can be written together as the version number so that when the data is subsequently read, the latest data can be determined based on the version number.

[0081] Specifically, the data processing system is further configured to obtain a first data processing result from the business data cluster and a second data processing result from the other business data link when obtaining a data acquisition request from a user;

[0082] The data processing system is further used to obtain first data timestamp information in the first data processing result and second data timestamp information in the second data processing result;

[0083] The data processing system is further used to compare the first data timestamp information with the second data timestamp information, and determine the target data timestamp information according to the comparison result;

[0084] The data processing system is further used to use the data processing result corresponding to the target data timestamp information as the request response result of the data acquisition request.

[0085] It should be noted that, when data is read in the embodiment of the present application, bilateral reading can be adopted, that is, the data processing system can simultaneously obtain the first data processing result from the first business data cluster and the second data processing result from the second business data cluster when obtaining the user's data acquisition request. Based on the version number of the first data processing result (including the first data timestamp information) and the version number of the second data processing result (including the second data timestamp information), the first data timestamp information and the second data timestamp information are compared, and the latest data timestamp information is used as the target data timestamp information.

[0086] It can be understood that by outputting the data processing result corresponding to the target timestamp information as the request response result of the data acquisition request, it can be ensured that the latest data processing result is fed back to the user.

[0087] In the embodiment of the present application, the business data processing module obtains business rules; the business data processing module is also used to generate data timestamp information according to the business rules when processing the business system source data in quasi-real time; and the data timestamp information is added to the data processing result obtained by processing the business system source data in quasi-real time. Since the timestamp information is added to the data processing result, the latest data processing result can be obtained when reading the data.

[0088] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those of the above-mentioned first embodiment and / or the second embodiment can refer to the above introduction, and will not be repeated later. Figure 2 , In the embodiment of the present application, the service data chain also includes: a fuse control module;

[0089] The fuse control module is used to obtain the fuse information of the service data cluster in another service data chain when the data processing system obtains the data acquisition request of the user if there is a node abnormality in the service data cluster;

[0090] The fusing control module is further used to determine the fusing state of a service data cluster in another service data chain according to the fusing information;

[0091] The fuse control module is further used to obtain the fuse logic of the service data cluster when the fuse state is not blown;

[0092] The fuse control module is further used to perform fuse processing on the service data cluster based on the fuse logic.

[0093] As shown in the figure, in an embodiment of the present application, at least one thread pool may be included in the data acquisition application, and thread pool A and thread pool B are taken as examples in the diagram implemented in the present application. In each thread pool, several working threads may be included, and working thread 1, working thread 2, and working thread 3 are taken as examples in the diagram implemented in the present application. In each business data cluster, at least one data area (i.e., node) may also be included, and area 1, area 2, and area 3 are taken as examples in the diagram of the embodiment of the present application. The thread pool can read the data content such as the data processing results in the data area, and when a node fails, the faulty node or the business data cluster as a whole can be fused.

[0094] It should be noted that in the case of high concurrency or burst traffic, the system may be overloaded due to high load, causing failures in certain nodes in the business data cluster. In this case, the failed nodes need to be isolated in time to avoid paralyzing the entire system, that is, to fuse the failed nodes.

[0095] It can be understood that the fuse control module in the embodiment of the present application will trigger the fuse of node or cluster when a problem occurs in node A in the business data cluster (which can be any node in the business data cluster). At this time, the corresponding node in another business data link should be used to provide services to the outside world.

[0096] It should be noted that if a problem occurs in the same node in the same table of the active-active cluster in the embodiment of the present application, both service data links may enter the fuse, which will cause the entire request corresponding to the entire table to be unavailable. In the embodiment of the present application, a preset fuse strategy with bilateral fuse mutual exclusion is used to control the fuse of the active-active cluster (that is, the first service data cluster and the second service data cluster) in the active-active service data link. For details, please refer to Figure 3 , Figure 3 A schematic diagram of the workflow of a preset fuse strategy in one implementation of the present application.

[0097] It should be explained that when receiving the user's data acquisition request, if there is a node abnormality in the business data cluster, the fuse information of the second business data cluster in the second business data link can be obtained, and based on the fuse information, it can be determined whether the second business data cluster (or the target node corresponding to the abnormal node in the second business data cluster) is blown; if the second business data cluster (or the target node corresponding to the abnormal node in the second business data cluster) is not blown, the first business data cluster or the abnormal node in the first business data cluster can be blown based on the fuse logic. If the second business data cluster (or the target node corresponding to the abnormal node in the second business data cluster) has been blown, the second business data cluster (or the target node corresponding to the abnormal node in the second business data cluster) can be detected for recovery status to detect the second business data cluster (or the target node corresponding to the abnormal node in the second business data cluster). If the detection is successful, it means that the second business data cluster (or the target node corresponding to the abnormal node in the second business data cluster) has been restored, and the first business data cluster or the abnormal node in the first business data cluster can be blown. Specifically, the fuse control module is further used to detect the activity of a service data cluster in another service data chain when the fuse state is blown;

[0098] The fuse control module is also used to obtain the fuse logic of the business data cluster when the detection is successful.

[0099] It is understandable that when the detection fails, it means that the second business data cluster (or the target node corresponding to the abnormal node in the second business data cluster) has been blown, and non-fuse logic is used for processing, that is, the first business data cluster or the abnormal node in the first business data cluster is not blown to ensure that the user's data acquisition request can be responded to.

[0100] In one implementation of the present application, in order to maximize the protection of data provision services, it is necessary to minimize the scope of the fuse. In the present application, the problem of business data cluster failure is a single node failure in most cases, and the scope of the failure is the data stored in the node. Therefore, the statistical information collection of the fuse in this application adopts the granularity of cluster + data table, and only the request of the table with node failure is fused, and the minimum scope affects the normal request.

[0101] In one implementation of the embodiment of the present application, the adjustment of the fuse parameter can adjust the error rate that needs to be reached when the interface is blown. Since there are many nodes in the distributed database, the error rate can be set to 5% in the present application. Compared with the default 50%, the solution of the embodiment of the present application can ensure timely fuse when a node fails, while ensuring the stability of data supply.

[0102] In one implementation of the embodiment of the present application, the present application also adjusts the fuse exit mechanism, analyzes the reasons for the unstable fuse duration, checks the Hystrix source code, and locates the markSuccess method in the class HystrixCircuitBreakerImpl. The conditional judgment of the fuse logic is inconsistent with expectations. If the previous request of two requests falls on the faulty data node at the same time, the request fails and meets the condition for opening the fuse switch, the switch is set to true. If the next request enters the normal data node successfully, it will enter the conditional judgment, causing the fuse switch to be set to false and closed. Based on the above factors, the application can record the fuse start time and the fuse sleep time when the fuse is turned on for the first time, and add a judgment condition to the markSuccess method. When the sum of the fuse sleep time and the fuse start time is less than the current time, the fuse will continue to be blown, thereby ensuring the stability of the fuse. Specifically, the fuse control module is also used to record the fuse start time and the fuse sleep time when the fuse state of the business data cluster enters the blown state;

[0103] The fuse control module is further used to determine the fuse judgment time based on the fuse sleep time and the fuse sleep time;

[0104] The fuse control module is further configured to keep the fuse state of the service data cluster in the state of being blown when the fuse judgment time is less than the current time.

[0105] The business data chain of the embodiment of the present application also includes: a fuse control module; when the data processing system obtains the user's data acquisition request, if there is a node abnormality in the business data cluster, the fuse control module obtains the fuse information of the business data cluster in another business data chain; determines the fuse state of the business data cluster in another business data chain according to the fuse information; when the fuse state is not blown, obtains the fuse logic of the business data cluster; and performs fuse processing on the business data cluster based on the fuse logic. Since the fuse control is performed on the abnormal nodes in the business data cluster, the stability of data processing is guaranteed.

[0106] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data processing method of the distributed cluster in a weak cloud environment of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0107] Based on the data processing system of the weak cloud environment distributed cluster as described above, the present application also provides a data processing method of the weak cloud environment distributed cluster, please refer to Figure 4 , the data processing method of the weak cloud environment distributed cluster includes:

[0108] Step S10, when the business data processing module obtains the business processing request, it performs quasi-real-time processing based on the business system source data sent by the business data source to obtain the data processing result;

[0109] Step S20, the business data processing module sends the data processing result to the business data writing module and another business data link;

[0110] Step S30: When receiving the data processing result sent by the business data processing module and / or the other business data link, the business data writing module writes the data processing result into the business data cluster.

[0111] The data processing method for a distributed cluster in a weak cloud environment provided by the present application adopts the data processing system for a distributed cluster in a weak cloud environment in the above-mentioned embodiment, and can solve the technical problem of the prior art that the interface access performance is reduced when some nodes in the cluster fail during the use of HBase. Compared with the prior art, the beneficial effects of the data processing method for a distributed cluster in a weak cloud environment provided by the present application are the same as the beneficial effects of the data processing system for a distributed cluster in a weak cloud environment provided by the above-mentioned embodiment, and the other technical features of the data processing method for a distributed cluster in a weak cloud environment are the same as the features disclosed by the above-mentioned embodiment system, which will not be repeated here.

[0112] In an implementation of the embodiment of the present application, when executing the step of performing quasi-real-time processing on the business system source data sent by the business data source to obtain the data processing result, the method further includes:

[0113] The business data processing module generates data timestamp information according to business rules;

[0114] The business data processing module adds the data timestamp information to the data processing result obtained by performing quasi-real-time processing on the business system source data.

[0115] In one implementation of the embodiment of the present application, the method further includes:

[0116] When the data processing system obtains the data acquisition request of the user, the data processing system obtains the first data processing result and the second data processing result from the business data cluster and the other business data link respectively;

[0117] The data processing system obtains first data timestamp information in the first data processing result and second data timestamp information in the second data processing result;

[0118] The data processing system is further used to compare the first data timestamp information with the second data timestamp information, and determine the target data timestamp information according to the comparison result;

[0119] The data processing system uses the data processing result corresponding to the target data timestamp information as the request response result of the data acquisition request.

[0120] In one implementation of the embodiment of the present application, the method further includes:

[0121] When the data processing system obtains a data acquisition request from a user, the fuse control module obtains fuse information of a service data cluster in another service data chain if a node abnormality exists in the service data cluster;

[0122] The fusing control module determines the fusing state of a service data cluster in another service data chain according to the fusing information;

[0123] The fuse control module obtains the fuse logic of the service data cluster when the fuse state is not blown;

[0124] The fusing control module performs fusing processing on the first service data cluster based on the fusing logic.

[0125] In an implementation of the embodiment of the present application, after the step of determining the fuse status of the service data cluster in another service data chain according to the fuse information by the fuse control module, the step further includes:

[0126] The fuse control module is further configured to detect activation of a service data cluster in another service data chain when the fuse state is blown;

[0127] The fuse control module is also used to obtain the fuse logic of the business data cluster when the detection is successful.

[0128] In one implementation of the embodiment of the present application, after the step of the fuse control module performing fuse processing on the first business data cluster based on the fuse logic, the step further includes: the fuse control module is also used to record the fuse start time and the fuse sleep time when the fuse state of the business data cluster enters the fuse state;

[0129] The fuse control module is further used to determine the fuse judgment time based on the fuse sleep time and the fuse sleep time;

[0130] The fuse control module is further configured to keep the fuse state of the service data cluster in the state of being blown when the fuse judgment time is less than the current time.

[0131] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data processing system for a distributed cluster in a weak cloud environment, characterized in that: The data processing system comprises: at least two business data links, each of which comprises: a business data source, a business data processing module, a business data writing module and a business data cluster; The business data processing module is used to perform quasi-real-time processing based on the business system source data sent by the business data source when a business processing request is obtained to obtain a data processing result; The business data processing module is further used to send the data processing result to the business data writing module and another business data link; The business data writing module is further used to write the data processing result into the business data cluster when receiving the data processing result sent by the business data processing module and / or the other business data link.

2. The data processing system of a distributed cluster in a weak cloud environment as claimed in claim 1, characterized in that: The business data processing module is also used to obtain business rules; The business data processing module is further used to generate data timestamp information according to business rules when performing quasi-real-time processing on the business system source data; The business data processing module is also used to add the data timestamp information to the data processing results obtained by quasi-real-time processing based on the business system source data.

3. The data processing system of a distributed cluster in a weak cloud environment as claimed in claim 2, characterized in that: The data processing system is further configured to obtain a first data processing result from the business data cluster and a second data processing result from the other business data link when a data acquisition request from a user is obtained; The data processing system is further used to obtain first data timestamp information in the first data processing result and second data timestamp information in the second data processing result; The data processing system is further used to compare the first data timestamp information with the second data timestamp information, and determine the target data timestamp information according to the comparison result; The data processing system is further used to use the data processing result corresponding to the target data timestamp information as the request response result of the data acquisition request.

4. The data processing system of a distributed cluster in a weak cloud environment as claimed in claim 1, characterized in that: The service data chain also includes: a fuse control module; The fuse control module is used to obtain the fuse information of the service data cluster in another service data chain when the data processing system obtains the data acquisition request of the user if there is a node abnormality in the service data cluster; The fusing control module is further used to determine the fusing state of a service data cluster in another service data chain according to the fusing information; The fuse control module is further used to obtain the fuse logic of the service data cluster when the fuse state is not blown; The fuse control module is further used to perform fuse processing on the service data cluster based on the fuse logic.

5. The data processing system of a distributed cluster in a weak cloud environment as claimed in claim 4, characterized in that: The fuse control module is further configured to detect activation of a service data cluster in another service data chain when the fuse state is blown; The fuse control module is also used to obtain the fuse logic of the business data cluster when the detection is successful.

6. The data processing system of a distributed cluster in a weak cloud environment as claimed in claim 4, characterized in that: The fusing control module is further configured to record the fusing start time and the fusing sleep time when the fusing state of the service data cluster enters the fusing state; The fuse control module is further used to determine the fuse judgment time based on the fuse sleep time and the fuse sleep time; The fuse control module is further configured to keep the fuse state of the service data cluster in the state of being blown when the fuse judgment time is less than the current time.

7. A data processing method for a weak cloud environment distributed cluster based on the data processing system for a weak cloud environment distributed cluster according to any one of claims 1 to 6, characterized in that: The method comprises: When the business data processing module obtains the business processing request, it performs quasi-real-time processing based on the business system source data sent by the business data source to obtain the data processing result; The business data processing module sends the data processing result to the business data writing module and another business data link; When receiving the data processing result sent by the business data processing module and / or the other business data link, the business data writing module writes the data processing result into the business data cluster.

8. The data processing method of a distributed cluster in a weak cloud environment as claimed in claim 7, characterized in that: When executing the step of performing quasi-real-time processing on the business system source data sent by the business data source to obtain the data processing result, the method further includes: The business data processing module generates data timestamp information according to business rules; The business data processing module adds the data timestamp information to the data processing result obtained by performing quasi-real-time processing on the business system source data.

9. The data processing method of a distributed cluster in a weak cloud environment as claimed in claim 8, characterized in that: The method further comprises: When the data processing system obtains the data acquisition request of the user, the data processing system obtains the first data processing result and the second data processing result from the business data cluster and the other business data link respectively; The data processing system obtains first data timestamp information in the first data processing result and second data timestamp information in the second data processing result; The data processing system is further used to compare the first data timestamp information with the second data timestamp information, and determine the target data timestamp information according to the comparison result; The data processing system uses the data processing result corresponding to the target data timestamp information as the request response result of the data acquisition request.

10. The data processing method of a distributed cluster in a weak cloud environment according to claim 7, characterized in that: The method further comprises: When the data processing system obtains a data acquisition request from a user, the fuse control module obtains fuse information of a service data cluster in another service data chain if a node abnormality exists in the service data cluster; The fusing control module determines the fusing state of a service data cluster in another service data chain according to the fusing information; The fuse control module obtains the fuse logic of the service data cluster when the fuse state is not blown; The fusing control module performs fusing processing on the first service data cluster based on the fusing logic.

Images