Intelligent processing method and system for data fault tolerance
通过获取、分析和容错处理数据,创建并验证数据副本,解决了数据处理中异常判断单一的问题,提高了数据处理的严谨性。
Patent Information
- Application Number
- CN202510426714.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-07
- Publication Date
- 2025-07-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the data processing process, the data abnormality judgment is single, resulting in deviations in the processing conclusions and insufficient rigor.
Ensure the rigor of data processing by acquiring data, analyzing data situations, performing fault-tolerant processing and storing processing, including confirming data exception types, creating data copies and verifying their correctness.
After data abnormalities, further judgment and intelligent processing are carried out to ensure the rigor of data processing and avoid misjudgment.
Smart Images
Figure CN120276902A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing, and specifically, to an intelligent processing method and system for data fault tolerance. Background Art
[0002] With the advent of the big data era, it not only provides great convenience for people's production and life, but also provides strong information guarantee for the long-term development of society. Under the background of big data at the present stage, the means of data processing and data monitoring have achieved a qualitative leap. With the continuous progress of technology, the era of manual data processing by people has passed. However, in the current methods of data processing and data monitoring, although an automated method has been formed, in the current automated processing process, the judgment process for data anomalies is often relatively single, and the rigor of the data processing process is not high. There may be some omissions in the data judgment, resulting in deviations in the conclusions of data processing.
[0003] Therefore, how to provide a method that can improve the rigor of data processing has become an urgent problem to be solved in this field. Summary of the Invention
[0004] This application proposes an intelligent processing method for data fault tolerance, including the following steps: obtaining the input data; analyzing the input data; performing fault tolerance processing on the data according to the analysis result; and performing storage processing on the data according to the result of the fault tolerance processing.
[0005] As described above, among them, analyzing the input data includes the following sub-steps: confirming the situation of the input data; analyzing the data in different situations to determine whether the data is abnormal; if the data is abnormal, performing fault tolerance processing on the data; if the data is normal, performing storage processing on the data.
[0006] As described above, among them, confirming the situation of the input data includes: the data itself has an anomaly, but there is no anomaly in this data; the data itself has an anomaly, and there is an anomaly in the data; the data itself is normal, but there is an anomaly.
[0007] As described above, among them, performing storage processing on the data includes the following sub-steps: creating a data copy according to the original data; accessing the data copy to confirm whether it is created correctly; if it is created correctly, storing the data copy and the original data; if the data copy is created incorrectly, re-creating the data copy.
[0008] As described above, among them, traversing all data points in the data copy, if all data points of the data copy are the same as those of the original data, it is considered that the data copy is created correctly.
[0009] An intelligent processing system for data fault tolerance, specifically including: an acquisition unit, an analysis unit, a fault tolerance processing unit, and a storage unit; the acquisition unit is used to acquire the input data; the analysis unit is used to analyze the input data; the fault tolerance processing unit is used to perform fault tolerance processing on the data according to the analysis result; the storage unit is used to perform storage processing on the data according to the fault tolerance processing result.
[0010] As described above, among them, the analysis unit analyzing the input data includes the following sub-steps: confirming the situation of the input data; analyzing the data in different situations to determine whether the data is abnormal; if the data is abnormal, performing fault tolerance processing on the data; if the data is normal, performing storage processing on the data.
[0011] As described above, among them, the analysis unit confirming the situation of the input data includes: the data itself has an abnormality, but there is no abnormal situation for this data; the data itself has an abnormality and there is an abnormal situation for the data; the data itself is normal, but there is an abnormal situation.
[0012] As described above, among them, the storage unit performing storage processing on the data includes the following sub-steps: creating a data copy according to the original data; accessing the data copy to confirm whether the creation is correct; if the creation is correct, storing the data copy and the original data; if the creation of the data copy is incorrect, re-creating the data copy.
[0013] As described above, among them, traversing all data points in the data copy, if all data points of the data copy are the same as those of the original data, it is considered that the data copy is created correctly.
[0014] This application has the following beneficial effects:
[0015] This application can, after the data has an abnormality, further judge the abnormal data, confirm again whether it is abnormal data, and perform intelligent processing after the data is abnormal, ensuring the rigor of data processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings described below are only some embodiments recorded in the present application. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings.
[0017] Figure 1 is a flowchart of an intelligent processing method for data fault tolerance provided by an embodiment of the present application;
[0018] Figure 2 is an internal structure schematic diagram of an intelligent processing system for data fault tolerance provided by an embodiment of the present application. Detailed implementation manners
[0019] Next, in combination with the accompanying drawings in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.
[0020] The present application provides an intelligent processing method and system for data fault tolerance, which can further judge the abnormal data after the data anomaly occurs, reconfirm whether it is abnormal data, and perform intelligent processing after the data anomaly to ensure the rigor of data processing.
[0021] Embodiment 1
[0022] As Figure 1 shown, this embodiment provides an intelligent processing method for data fault tolerance, which specifically includes the following steps:
[0023] Step S110: Obtain the input data.
[0024] The input data is the data determined to be abnormal.
[0025] Step S120: Analyze the input data.
[0026] In this embodiment, the data considered abnormal may have the following several situations:
[0027] The first type of data: The data itself is abnormal, but there is no abnormal situation for this data.
[0028] The second type of data: The data itself is abnormal, and there is an abnormal situation for the data.
[0029] The third type of data: The data itself is normal, but there is an abnormal situation.
[0030] Among them, the above data is usually considered abnormal data, but this embodiment will further analyze the above considered abnormal data to analyze which are the truly abnormal data, so as to perform different processing on the truly abnormal and non-truly abnormal data.
[0031] Step S120 specifically includes the following sub-steps:
[0032] Step S1201: Confirm the situation of the input data.
[0033] The input data is confirmed according to the above situation to confirm whether the above three situations exist. If they exist, step S1202 is executed; otherwise, it is considered that the data is not abnormal and the process exits.
[0034] Step S1202: Analyze data under different conditions to determine whether the data is abnormal.
[0035] Among them, confirm the operation time of different data under the same path. Each type of data is divided into multiple data points. The operation time T is specifically expressed as:
[0036]
[0037] v i represents the execution time of executing data α i (i = 1, 2,..., n), h(i) represents the number of other data being executed before executing data α i v j represents the execution time of other task sets h(i) being executed before the system executes data α i t j represents the period of other data being executed before executing data α i The period of executing one data point in the current data α represents the execution of the current data α i One of the data points, where j represents a natural number.
[0038] After separately confirming the operation times of the data of the first to third categories, if the operation time T is greater than the specified threshold, it means that the operation time is too long, then it is considered that the data is abnormal, and step S130 is executed; otherwise, step S140 is executed.
[0039] Step S130: Perform fault tolerance processing on the data according to the analysis results.
[0040] Since the operation time of the above data is greater than the specified threshold, it indicates that the data is abnormal. However, in this case, there may be a possibility of the data itself being abnormal or the data input being abnormal. If the data itself is abnormal, then during the process of determining the operation time, the data points may also be abnormal, so the operation time will be relatively long. If it is due to data input abnormality, then the overall operation time may be relatively short. Therefore, if the operation time is greater than the specified threshold, the data is further divided into true abnormality and false abnormality according to the operation time value.
[0041] The fault tolerance processing specifically includes the following sub-steps:
[0042] Step S1301: Divide the abnormal data.
[0043] In this step, the "specified threshold" with an operation time T greater than the specified threshold is defined as the first threshold. If the operation time of this data is greater than the first specified threshold and less than the second specified threshold, then this data is defined as a false anomaly, that is, the conclusion of this fault tolerance processing is that the data is a false anomaly, and step S140 is executed.
[0044] If the operation time of this data is greater than the second specified threshold and less than the third specified threshold, then this data is defined as a true anomaly, that is, the conclusion of this fault tolerance processing is that the data is a true anomaly, and step S1302 is executed.
[0045] Among them, the values of the first to third thresholds increase in sequence. For example, the first threshold is 10, the second threshold is 15, and the third threshold is 30.
[0046] Step S1302: Replace the data according to the partitioning result.
[0047] Among them, the input data is re-obtained to complete the re-replacement of the data.
[0048] Step S140: Perform storage processing on the data.
[0049] Among them, the data considered to be normal data or false anomalies is stored, specifically including the following sub-steps:
[0050] Step S1401: Create a data copy based on the original data.
[0051] Among them, the original data is the data considered to be normal and false anomalies.
[0052] The method of creating a data copy can be created according to the existing technology and will not be elaborated here.
[0053] Step S1402: Access the data copy to confirm whether it is created correctly.
[0054] Among them, specifically select any data point in the data copy, and check whether there is the same data point in the original data. If the similarity is 100%, then it is considered that the data point in the data copy and the data point in the original data are the same data point.
[0055] Traverse all data points in the data copy. If all data points in the data copy are the same as those in the original data, then it is considered that the data copy is created correctly, and step S1403 is executed. Otherwise, it is considered that the data copy is created incorrectly, and the data copy is re-created.
[0056] Among them, any data point f in the data copy j =[f j1 ,f j2 ,...f jk ,...f jp and any data point f in the original datai = [f i1 , f i2 ,... f ik ,... f ip is specifically expressed as:
[0057]
[0058] where f ik represents the value of data point a in the data replica on variable l, and f i represents the value of data point a in the original data on variable l. jk j
[0059] Step S1403: Store the data replica and the original data.
[0060] After storing the data replica and the original data, if an error occurs when accessing the data, the data replica can be selected for access, thus ensuring that the access process can still proceed smoothly.
[0061] Embodiment 2
[0062] As Figure 2 shown, this embodiment provides an intelligent processing system for data fault tolerance, specifically including: an acquisition unit 201, an analysis unit 202, a fault tolerance processing unit 203, and a storage unit 204.
[0063] The acquisition unit 201 is used to acquire the input data.
[0064] The input data is the data determined to be abnormal.
[0065] The analysis unit 202 is used to analyze the input data.
[0066] In this embodiment, the data considered abnormal may have the following situations:
[0067] The first type of data: The data itself is abnormal, but no abnormal situation occurs to this data.
[0068] The second type of data: The data itself is abnormal, and an abnormal situation occurs to the data.
[0069] The third type of data: The data itself is normal, but an abnormal situation occurs.
[0070] Among them, the above data is usually considered abnormal data, but this embodiment will further analyze the above data considered abnormal to analyze which are the truly abnormal data.
[0071] The analysis unit 202 specifically includes the following sub-steps:
[0072] Step S1: Confirm the specific situation of the input data.
[0073] Among them, the input data is confirmed according to the above situation to check whether the above three situations exist. If they exist, step S2 is executed; otherwise, it is considered that the data has no abnormality and the process exits.
[0074] Step S2: Analyze the data in different situations to determine whether the data is abnormal.
[0075] Among them, confirm the operation time of different data under the same path. Each type of data is divided into multiple data points, and the operation time T is specifically expressed as:
[0076]
[0077] v i represents the execution time of executing data α i (i = 1, 2,..., n), and h(i) represents the number of other data being executed before executing data α i v j represents the execution time of other task sets h(i) being executed before the system executes data α i t j represents the period of other data being executed before executing data α i represents the period of executing one data point in the current data α i where j represents a natural number.
[0078] After respectively confirming the operation times of the first to third types of data, if the operation time T is greater than the specified threshold, it means that the operation time is too long, so it is considered that the data is abnormal, and the fault tolerance unit 203 is executed; otherwise, the storage unit 204 is executed.
[0079] The fault tolerance unit 203 is used to perform fault tolerance processing on the data according to the analysis result.
[0080] Among them, since the operation time of the above data is greater than the specified threshold, it indicates that the data is abnormal. However, in this case, there may be a possibility of data itself being abnormal or data input being abnormal. If the data itself is abnormal, then during the process of determining the operation time, the data points may also be abnormal, so the operation time will be relatively long. If it is due to data input abnormality, then the overall operation time may be relatively short. Therefore, if the operation time is greater than the specified threshold, the data is further divided into true abnormality and false abnormality according to the operation time value.
[0081] Among them, the fault tolerance processing specifically includes the following sub-steps:
[0082] Step W1: Divide the abnormal data.
[0083] In this step, the "specified threshold" with an operation time T greater than the specified threshold is defined as the first threshold. If the operation time of this data is greater than the first specified threshold and less than the second specified threshold, then this data is defined as a false anomaly, and the storage unit 204 is executed.
[0084] If the operation time of this data is greater than the second specified threshold and less than the third specified threshold, then this data is defined as a true anomaly, and step W2 is executed.
[0085] Among them, the values of the first to third thresholds increase in sequence.
[0086] As an example, the first threshold is 10, the second threshold is 15, and the third threshold is 30.
[0087] Step W2: Replace the data according to the partitioning result.
[0088] Among them, the input data is re-obtained to complete the re-replacement of the data.
[0089] The storage unit 204 is used to perform storage processing on the data.
[0090] Among them, the data considered as normal data or false anomalies is stored, specifically including the following sub-steps:
[0091] Step Z1: Create a data copy.
[0092] Step Z2: Access the data copy to confirm whether it is created correctly.
[0093] Among them, specifically select any data point in the data copy, and search for whether there is the same data point in the original data. If the similarity is 100%, then it is considered that the data point in the data copy and the data point in the original data are the same data point.
[0094] Traverse all data points in the data copy. If all data points in the data copy are the same as those in the original data, then it is considered that the data copy is created correctly, and step Z3 is executed. Otherwise, it is considered that the data copy is created incorrectly, and the data copy is re-created.
[0095] Among them, any data point f in the data copy j =[f j1 ,f j2 ,...f jk ,...f jp and any data point f in the original data i =[f i1 ,f i2 ,...f ik ,...f ip The similarity is specifically expressed as:
[0096]
[0097] where f ik represents the value of data point a in the data replica i on variable l, and f jk represents the value of data point a in the original data j on variable l.
[0098] Step Z3: Store the data replica and the data.
[0099] After storing the data replica and the original data, if an error occurs when accessing the data, the data replica can be selected for access, thus ensuring that the access process can still proceed smoothly.
[0100] This application has the following beneficial effects:
[0101] This application can further judge the abnormal data after the data anomaly occurs, confirm again whether it is abnormal data, and perform intelligent processing after the data anomaly, ensuring the rigor of data processing.
[0102] Although the examples referred to in the current application are described, they are for illustrative purposes only and not a limitation of the present application. Changes, additions, and / or deletions to the embodiments can be made without departing from the scope of the present application.
[0103] The above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed by this application can easily think of changes or substitutions, which should all be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claimed rights.
Claims
1. An intelligent processing method for data fault tolerance, characterized in that It includes the following steps: Obtain the input data; Analyze the input data; Perform fault tolerance processing on the data according to the analysis results; Perform storage processing on the data according to the results of the fault tolerance processing.
2. The intelligent processing method for data fault tolerance according to claim 1, characterized in that, Analyzing the input data includes the following sub-steps: Confirm the situation of the input data; Analyze the data in different situations to determine whether the data is abnormal; If the data is abnormal, perform fault tolerance processing on the data; If the data is normal, perform storage processing on the data.
3. The intelligent processing method for data fault tolerance according to claim 2, characterized in that, Confirming the situation of the input data includes: The data itself is abnormal, but there is no abnormal situation for this data; The data itself is abnormal, and there is an abnormal situation for the data; The data itself is normal, but there is an abnormal situation.
4. The intelligent processing method for data fault tolerance according to claim 1, characterized in that, Performing storage processing on the data includes the following sub-steps: Create a data copy according to the original data; Access the data copy to confirm whether it is created correctly; If created correctly, store the data copy and the original data; The data copy is created incorrectly, and create the data copy again.
5. The intelligent processing method for data fault tolerance according to claim 4, characterized in that, Traverse all data points in the data copy. If all data points in the data copy are the same as those in the original data, it is considered that the data copy is created correctly.
6. An intelligent processing system for data fault tolerance, characterized in that, Specifically include: An acquisition unit, an analysis unit, a fault tolerance processing unit, and a storage unit; The acquisition unit is used to obtain the input data; The analysis unit is used to analyze the input data; The fault tolerance processing unit is used to perform fault tolerance processing on the data according to the analysis results; The storage unit is used to perform storage processing on the data according to the results of the fault tolerance processing.
7. The intelligent processing system for data fault tolerance according to claim 6, characterized in that The analysis unit analyzes the input data including the following sub-steps: Confirm the situation of the input data; Analyze the data in different situations to determine whether the data is abnormal; If the data is abnormal, perform fault tolerance processing on the data; If the data is normal, perform storage processing on the data.
8. The intelligent processing system for data fault tolerance according to claim 7, wherein The situation of confirming the input data in the analysis unit includes: The data itself is abnormal, but there is no abnormal situation for this data; The data itself is abnormal, and there is an abnormal situation for the data; The data itself is normal, but there is an abnormal situation.
9. The intelligent processing system for data fault tolerance according to claim 6, wherein The storage unit performs storage processing on the data including the following sub-steps: Create a data copy according to the original data; Access the data copy to confirm whether it is created correctly; If created correctly, store the data copy and the original data; The data copy is created incorrectly, and create the data copy again.
10. The intelligent processing system for data fault tolerance according to claim 9, characterized in that, Traverse all data points in the data copy. If all data points in the data copy are the same as those in the original data, it is considered that the data copy is created correctly.