Abnormal data compensation method

Abstract

The invention discloses an abnormal data compensation method. The abnormal data compensation method comprises the following steps of: directly accessing each business item into an abnormal compensation component to realize a data acquisition function; the data acquisition service and the specific business service are split, and corresponding data compensation is realized in independent items according to different business types; data before and after compensation are subjected to incoming line evaluation and are recorded through documents after evaluation, dependence on business services is eliminated through independent items, and it is guaranteed that the data are complete and not lost; according to the method, the business exception compensation access complexity is reduced, business unbinding and abnormal data compensation operations are carried out, too deep business intrusion is avoided, the business complexity is reduced, the access exception compensation is convenient and rapid, the research and development efficiency is improved, data collection and compensation behaviors are independently operated in a service mode, the influence of business service faults is avoided, data are independently managed, and the reliability is high. And data tracking and behavior analysis are facilitated.

Description

Technical Field

[0001] The invention relates to an abnormal data compensation method and belongs to the technical field of abnormal data compensation application. Background Art

[0002] During the actual use of the business system, the processing flow is often interrupted due to logic bugs, third-party interface exceptions, system crashes, network interruptions, etc., resulting in inconsistent upstream and downstream data status of the entire process. If the corresponding process node does not have a retry mechanism or automatic rollback protection, manual intervention is required to repair the data.

[0003] In the prior art, when different businesses access the abnormal compensation service, they all need to handle all abnormal access processes in their own businesses. The prior art has the following deficiencies: each business line needs to repeat the abnormal data access, reading and compensation processes, and the compensation behavior is bound to the business service. The abnormal data collection and compensation operations are often only required when a business line fails, and the abnormal data collection and compensation operations cannot be performed normally when the service fails. Summary of the invention

[0004] In this embodiment, an abnormal data compensation method is provided to solve the problem in the prior art that each business line needs to repeat the abnormal data access, reading and compensation process, the compensation behavior is bound to the business service, and the abnormal data collection and compensation operations are often only required when a business line fails, and the abnormal data collection and compensation operations cannot be performed normally when the service fails.

[0005] The present invention achieves the above-mentioned purpose through the following technical scheme, an abnormal data compensation method, and the abnormal data compensation method comprises the following steps:

[0006] S1. Each business project is directly connected to the abnormal compensation component to realize data collection function;

[0007] S2. Data collection services are separated from specific business services, and corresponding data compensation is implemented in independent projects according to different business types;

[0008] S3. Evaluate the data before and after compensation, record it in documents after evaluation, and use independent projects to eliminate dependence on business services to ensure data integrity and no loss.

[0009] Preferably, in the step (S1), the data is preprocessed according to the selected compensation method, such as data cleaning and data transformation.

[0010] Preferably, in the step (S1), the preprocessed data is applied to actual data analysis or model to perform abnormal data compensation.

[0011] Preferably, in identifying abnormal data in step (S1), first, it is necessary to collect and organize relevant data sets to ensure the integrity and accuracy of the data, use charts to visualize the data so as to more intuitively identify abnormal data points, determine the normal range of the data through statistical analysis methods, and identify abnormal data that exceeds the normal range.

[0012] Preferably, in the step (S2), the effect of the compensation method is evaluated by comparing the data before and after compensation. If the compensated data is used in a machine learning model, the model needs to be verified to ensure the accuracy and stability of the model.

[0013] Preferably, outlier processing includes deletion and replacement. Deletion means that if the number of abnormal data is small and has little impact on the overall analysis, they can be directly deleted; replacement means using statistical methods or machine learning algorithms to estimate and replace outliers.

[0014] Preferably, the missing value processing in the step (S1) includes interpolation, matrix completion and random forest method. The interpolation method is to use the information of existing data points to estimate and fill in the missing values ​​through interpolation methods. The matrix completion is to use matrix decomposition and other techniques to complete the missing data matrix. The random forest method is to use machine learning algorithms such as random forest to predict and fill in the missing values.

[0015] Preferably, in step (S2), evaluation is performed based on the quantity and distribution of abnormal data and the degree of their impact on the overall data analysis.

[0016] Preferably, in step (S3), the steps, methods and results of abnormal data compensation need to be recorded in detail for subsequent analysis and auditing.

[0017] Preferably, in the step (S3), the compensated data set is updated to a data warehouse or database for subsequent use, and the data is regularly maintained, such as data cleaning, data updating, etc., to ensure the continuous availability of the data.

[0018] The beneficial effects of the present invention are as follows: the abnormal data compensation method in the present invention reduces the complexity of accessing business abnormality compensation, unbinds business and abnormal data compensation operations, avoids too deep business intrusion and reduces business complexity, makes access to abnormality compensation convenient and fast, improves research and development efficiency, and runs data collection and compensation behaviors as independent services to avoid being affected by business service failures. Data is independently managed, which facilitates data tracking and behavior analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Embodiment 1:

[0022] The abnormal data compensation method comprises the following steps:

[0023] S1. Each business project is directly connected to the abnormal compensation component to realize data collection function;

[0024] S2. Data collection services are separated from specific business services, and corresponding data compensation is implemented in independent projects according to different business types;

[0025] S3. Evaluate the data before and after compensation, record it in documents after evaluation, and use independent projects to eliminate dependence on business services to ensure data integrity and no loss.

[0026] Preferably, in the step (S1), the data is preprocessed according to the selected compensation method, such as data cleaning and data transformation.

[0027] Preferably, in the step (S1), the preprocessed data is applied to actual data analysis or model to perform abnormal data compensation.

[0028] Preferably, in identifying abnormal data in step (S1), first, it is necessary to collect and organize relevant data sets to ensure the integrity and accuracy of the data, use charts to visualize the data so as to more intuitively identify abnormal data points, determine the normal range of the data through statistical analysis methods, and identify abnormal data that exceeds the normal range.

[0029] Preferably, in the step (S2), the effect of the compensation method is evaluated by comparing the data before and after compensation. If the compensated data is used in a machine learning model, the model needs to be verified to ensure the accuracy and stability of the model.

[0030] Preferably, outlier processing includes deletion and replacement. Deletion means that if the number of abnormal data is small and has little impact on the overall analysis, they can be directly deleted; replacement means using statistical methods or machine learning algorithms to estimate and replace outliers.

[0031] Preferably, the missing value processing in the step (S1) includes interpolation, matrix completion and random forest method. The interpolation method is to use the information of existing data points to estimate and fill in the missing values ​​through interpolation methods. The matrix completion is to use matrix decomposition and other techniques to complete the missing data matrix. The random forest method is to use machine learning algorithms such as random forest to predict and fill in the missing values.

[0032] Preferably, in step (S2), evaluation is performed based on the quantity and distribution of abnormal data and the degree of their impact on the overall data analysis.

[0033] Preferably, in step (S3), the steps, methods and results of abnormal data compensation need to be recorded in detail for subsequent analysis and auditing.

[0034] Preferably, in the step (S3), the compensated data set is updated to a data warehouse or database for subsequent use, and the data is regularly maintained, such as data cleaning, data updating, etc., to ensure the continuous availability of the data.

[0035] The above abnormal data compensation method reduces the complexity of accessing business abnormality compensation, unbinds business and abnormal data compensation operations, avoids too deep business intrusion, reduces business complexity, makes access to abnormality compensation convenient and fast, improves R&D efficiency, and runs data collection and compensation behaviors as independent services to avoid being affected by business service failures. Data is managed independently to facilitate data tracking and behavior analysis.

[0036] Embodiment 2:

[0037] The abnormal data compensation method comprises the following steps:

[0038] S1. Each business project is directly connected to the abnormal compensation component to realize data collection function;

[0039] S2. Data collection services are separated from specific business services, and corresponding data compensation is implemented in independent projects according to different business types;

[0040] S3. Evaluate the data before and after compensation, record it in documents after evaluation, and use independent projects to eliminate dependence on business services to ensure data integrity and no loss.

[0041] Preferably, in the step (S1), the data is preprocessed according to the selected compensation method, such as data cleaning and data transformation.

[0042] Preferably, in the step (S1), the preprocessed data is applied to actual data analysis or model to perform abnormal data compensation.

[0043] Preferably, in identifying abnormal data in step (S1), first, it is necessary to collect and organize relevant data sets to ensure the integrity and accuracy of the data, use charts to visualize the data so as to more intuitively identify abnormal data points, determine the normal range of the data through statistical analysis methods, and identify abnormal data that exceeds the normal range.

[0044] Preferably, in the step (S2), the effect of the compensation method is evaluated by comparing the data before and after compensation. If the compensated data is used in a machine learning model, the model needs to be verified to ensure the accuracy and stability of the model.

[0045] Preferably, outlier processing includes deletion and replacement. Deletion means that if the number of abnormal data is small and has little impact on the overall analysis, they can be directly deleted; replacement means using statistical methods or machine learning algorithms to estimate and replace outliers.

[0046] Preferably, the missing value processing in the step (S1) includes interpolation, matrix completion and random forest method. The interpolation method is to use the information of existing data points to estimate and fill in the missing values ​​through interpolation methods. The matrix completion is to use matrix decomposition and other techniques to complete the missing data matrix. The random forest method is to use machine learning algorithms such as random forest to predict and fill in the missing values.

[0047] Preferably, in step (S2), evaluation is performed based on the quantity and distribution of abnormal data and the degree of their impact on the overall data analysis.

[0048] Preferably, in step (S3), the steps, methods and results of abnormal data compensation need to be recorded in detail for subsequent analysis and auditing.

[0049] Preferably, in the step (S3), the compensated data set is updated to a data warehouse or database for subsequent use, and the data is regularly maintained, such as data cleaning, data updating, etc., to ensure the continuous availability of the data.

[0050] The above abnormal data compensation method reduces the complexity of accessing business abnormality compensation, unbinds business and abnormal data compensation operations, avoids too deep business intrusion, reduces business complexity, makes access to abnormality compensation convenient and fast, improves R&D efficiency, and runs data collection and compensation behaviors as independent services to avoid being affected by business service failures. Data is managed independently to facilitate data tracking and behavior analysis.

[0051] Embodiment three:

[0052] The abnormal data compensation method comprises the following steps:

[0053] S1. Each business project is directly connected to the abnormal compensation component to realize data collection function;

[0054] S2. Data collection services are separated from specific business services, and corresponding data compensation is implemented in independent projects according to different business types;

[0055] S3. Evaluate the data before and after compensation, record it in documents after evaluation, and use independent projects to eliminate dependence on business services to ensure data integrity and no loss.

[0056] Preferably, in the step (S1), the data is preprocessed according to the selected compensation method, such as data cleaning and data transformation.

[0057] Preferably, in the step (S1), the preprocessed data is applied to actual data analysis or model to perform abnormal data compensation.

[0058] Preferably, in identifying abnormal data in step (S1), first, it is necessary to collect and organize relevant data sets to ensure the integrity and accuracy of the data, use charts to visualize the data so as to more intuitively identify abnormal data points, determine the normal range of the data through statistical analysis methods, and identify abnormal data that exceeds the normal range.

[0059] Preferably, in the step (S2), the effect of the compensation method is evaluated by comparing the data before and after compensation. If the compensated data is used in a machine learning model, the model needs to be verified to ensure the accuracy and stability of the model.

[0060] Preferably, outlier processing includes deletion and replacement. Deletion means that if the number of abnormal data is small and has little impact on the overall analysis, they can be directly deleted; replacement means using statistical methods or machine learning algorithms to estimate and replace outliers.

[0061] Preferably, the missing value processing in the step (S1) includes interpolation, matrix completion and random forest method. The interpolation method is to use the information of existing data points to estimate and fill in the missing values ​​through interpolation methods. The matrix completion is to use matrix decomposition and other techniques to complete the missing data matrix. The random forest method is to use machine learning algorithms such as random forest to predict and fill in the missing values.

[0062] Preferably, in step (S2), evaluation is performed based on the quantity and distribution of abnormal data and the degree of their impact on the overall data analysis.

[0063] Preferably, in step (S3), the steps, methods and results of abnormal data compensation need to be recorded in detail for subsequent analysis and auditing.

[0064] Preferably, in the step (S3), the compensated data set is updated to a data warehouse or database for subsequent use, and the data is regularly maintained, such as data cleaning, data updating, etc., to ensure the continuous availability of the data.

[0065] The above abnormal data compensation method avoids too deep business intrusion and reduces business complexity, making access abnormality compensation convenient and fast, improving R&D efficiency, and running data collection and compensation behaviors as independent services to avoid being affected by business service failures. Data is managed independently, which facilitates data tracking and behavior analysis.

[0066] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0067] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. An abnormal data compensation method, characterized in that: The abnormal data compensation method comprises the following steps: S1. Each business project is directly connected to the abnormal compensation component to realize data collection function; S2. Data collection services are separated from specific business services, and corresponding data compensation is implemented in independent projects according to different business types; S3. Evaluate the data before and after compensation, record it in documents after evaluation, and use independent projects to eliminate dependence on business services to ensure data integrity and no loss.

2. The abnormal data compensation method according to claim 1, characterized in that: In the step (S1), the data is preprocessed according to the selected compensation method, such as data cleaning and data transformation.

3. The abnormal data compensation method according to claim 1, characterized in that: In the step (S1), the preprocessed data is applied to actual data analysis or model to perform abnormal data compensation.

4. The abnormal data compensation method according to claim 1, characterized in that: In the step (S1) of identifying abnormal data, first, it is necessary to collect and organize relevant data sets to ensure the integrity and accuracy of the data, use charts to visualize the data so as to more intuitively identify abnormal data points, determine the normal range of the data through statistical analysis methods, and identify abnormal data that exceeds the normal range.

5. The abnormal data compensation method according to claim 1, characterized in that: In the step (S2), the effect of the compensation method is evaluated by comparing the data before and after compensation. If the compensated data is used in a machine learning model, the model needs to be verified to ensure the accuracy and stability of the model.

6. The abnormal data compensation method according to claim 1, characterized in that: The outlier processing in step (S1) includes deletion and replacement. Deletion means that if the number of abnormal data is small and has little impact on the overall analysis, they can be directly deleted; replacement means using statistical methods or machine learning algorithms to estimate and replace the outliers.

7. The abnormal data compensation method according to claim 1, characterized in that: The missing value processing in the step (S1) includes interpolation, matrix completion and random forest method. The interpolation method is to use the information of existing data points to estimate and fill in the missing values ​​through interpolation methods. The matrix completion is to use matrix decomposition and other technologies to complete the missing data matrix. The random forest method is to use machine learning algorithms such as random forest to predict and fill in the missing values.

8. The abnormal data compensation method according to claim 1, characterized in that: In the step (S2), an evaluation is performed based on the quantity and distribution of abnormal data and the degree of its impact on the overall data analysis.

9. The abnormal data compensation method according to claim 1, characterized in that: In the step (S3), the steps, methods and results of abnormal data compensation need to be recorded in detail for subsequent analysis and audit.

10. The abnormal data compensation method according to claim 1, characterized in that: In the step (S3), the compensated data set is updated to the data warehouse or database for subsequent use, and the data is regularly maintained, such as data cleaning, data updating, etc., to ensure the continuous availability of the data.

Images