Data processing method and device, equipment and storage medium

Abstract

The application provides a data processing method, device and equipment and a storage medium. The target data and abnormal data of the target data are determined, the abnormal data is used to determine target historical abnormal data from historical abnormal data, and the number of times of repair of the target historical abnormal data is determined. If the number of times of repair of the target historical abnormal data is greater than a preset number of times of repair, the abnormal data is deleted from the target data, and the data state of the target historical abnormal data is adjusted to an unrepairable state. Finally, the target historical abnormal data is displayed, and the target data from which the abnormal data is deleted is subjected to the next round of auditing. Thus, the auditing workload of auditors is reduced, and the auditing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device and storage medium. Background Technology

[0002] In the telecommunications industry, data auditing is an important task that can identify and correct anomalies in data, ensuring its authenticity and accuracy.

[0003] Currently, the common auditing method involves auditing all data daily. When anomalies are detected, dedicated personnel are required to investigate and correct the issues to ensure the accuracy of business data. However, due to various objective reasons, anomalies may not be corrected promptly or may not be successfully corrected, causing them to repeatedly appear in the audit results. This necessitates repeated processing of these data by auditors, significantly disrupting the audit process.

[0004] Therefore, existing technologies suffer from low audit efficiency. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and storage medium to solve the technical problem of low audit efficiency in the prior art.

[0006] Firstly, this application provides a data processing method, including:

[0007] The target data and the abnormal data of the target data are identified, wherein the abnormal data is the data identified during the current round of auditing of the target data.

[0008] Based on the abnormal data, target historical abnormal data is determined from the historical abnormal data. The historical abnormal data is the historical data of the target data in the previous round of audit. The target historical abnormal data is the data corresponding to the abnormal data. The previous round of audit is the audit before the current round of audit.

[0009] Determine the number of times the target historical anomaly data has been repaired;

[0010] If the number of times the target historical abnormal data has been repaired is greater than the preset number of repairs, then the abnormal data will be deleted from the target data and the data status of the target historical abnormal data will be adjusted to an unrepairable state.

[0011] Display the target's historical abnormal data, and then perform the next round of auditing on the target data after deleting the abnormal data, wherein the next round of auditing is the audit following the current round of auditing.

[0012] Optionally, the method described above, determining the target data and the abnormal data of the target data, includes:

[0013] The target data, the previous audit data, the previous abnormal data, and other data of the target data in the previous audit are determined. The previous audit data is the data whose data status is normal after the previous audit. The previous abnormal data is the data whose data status is abnormal after the previous audit. The other data is the newly added data of the target data after the previous audit.

[0014] The previous audit data is processed in the current round of auditing to obtain the first audit result;

[0015] The previous abnormal data is stored in the target database. After the previous abnormal data is repaired, the previous abnormal data is subjected to the current round of audit processing to obtain the second audit result.

[0016] The other data are subjected to the current round of audit processing to obtain the third audit result; the target audit result is obtained based on the first audit result, the second audit result, and the third audit result.

[0017] Based on the audit results of the target, abnormal data in the target data are identified.

[0018] Optionally, in the method described above, after determining the number of times the target historical anomaly data has been repaired, the method includes:

[0019] If the number of times the target historical abnormal data has been repaired is less than or equal to the preset number of repairs, then the abnormal data is repaired to obtain a repair result, and the data status of the abnormal data is adjusted to the repair processing status.

[0020] If the repair result indicates successful repair, the data status of the abnormal data is adjusted from the repair processing status to the repair completed status, so as to conduct the next round of auditing on the abnormal data in the target data whose data status is repair completed.

[0021] Optionally, in the method described above, after determining the number of times the target historical anomaly data has been repaired, the method includes:

[0022] If the number of times the target historical abnormal data has been repaired is less than or equal to the preset number of repairs, then the abnormal data is repaired to obtain a repair result, and the data status of the abnormal data is adjusted to the repair processing status.

[0023] If the repair result indicates successful repair, the data status of the abnormal data is adjusted from the repair processing status to the repair completed status, so as to conduct the next round of auditing on the abnormal data in the target data whose data status is repair completed.

[0024] Optionally, in the method described above, after the step of repairing the abnormal data if the number of repairs to the target historical abnormal data is less than or equal to a preset number of repairs, obtaining a repair result, and adjusting the data status of the abnormal data to a repair processing status, the method further includes:

[0025] If the repair result indicates that the repair has failed, the data status of the abnormal data is kept in the repair processing status, and the number of times the target historical abnormal data has been repaired is updated to obtain the updated number of repairs.

[0026] Based on the number of times the update has been repaired, the abnormal data is repaired until the number of times the update has been repaired is greater than the preset number of repairs, or the abnormal data is repaired, and then the data status of the abnormal data is determined.

[0027] Based on the data status of the abnormal data, the target data is updated for the next round of auditing.

[0028] Optionally, in the method described above, after repairing the abnormal data and obtaining the repair result, the method further includes:

[0029] Display the abnormal data and its data status;

[0030] Feedback information is obtained in response to the user's feedback operation based on the abnormal data and the data status of the abnormal data;

[0031] Based on the feedback information, the abnormal data is deleted from the target data, and the data status of the target historical abnormal data corresponding to the abnormal data is adjusted.

[0032] Optionally, in the method described above, determining the target historical anomalous data from the historical anomalous data based on the anomalous data includes:

[0033] Obtain the data identifier of the abnormal data and the data identifier of the historical abnormal data;

[0034] Compare the data identifier of the abnormal data with the historical data identifier of the historical abnormal data;

[0035] If a target historical data identifier corresponding to the data identifier exists in the historical data identifier, then the target historical abnormal data is determined based on the target historical data identifier.

[0036] Optionally, in the method described above, after comparing the data identifier of the abnormal data and the historical data identifier of the historical abnormal data, the method further includes:

[0037] If no target historical data identifier corresponding to the data identifier exists in the historical data identifier, the abnormal data is repaired to obtain the repair result;

[0038] Based on the repair process results, determine the data status of the abnormal data;

[0039] The abnormal data and its data status are stored in a target database, wherein the target database is a database that stores the historical abnormal data.

[0040] Optionally, in the method described above, when the data status of the target data changes, or when the data status of the target historical abnormal data changes, change information is generated; the change information is stored in the change information database so that the change information is displayed when the change information database receives a query instruction.

[0041] Secondly, this application provides a data processing apparatus, comprising:

[0042] The acquisition module is used to determine the target data and any abnormal data within the target data.

[0043] The target historical anomaly data determination module is used to determine the target historical anomaly data from the historical anomaly data based on the anomaly data.

[0044] The repair count determination module is used to determine the number of times the target historical abnormal data has been repaired;

[0045] The processing module is used to adjust the abnormal data according to the number of times the target historical abnormal data has been repaired;

[0046] The display module is used to display the target's historical abnormal data and to perform the next round of auditing on the target data after deleting the abnormal data.

[0047] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0048] The memory stores computer-executed instructions;

[0049] The processor executes computer execution instructions stored in the memory to implement the above-described method.

[0050] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the above-described method.

[0051] The data processing method, apparatus, equipment, and storage medium provided in this application determine the target data and its abnormal data. Based on the abnormal data, the method identifies historical abnormal data from historical abnormal data and determines the number of times the historical abnormal data has been repaired. If the number of repairs for the historical abnormal data exceeds a preset number of repairs, the abnormal data is deleted from the target data, and its data status is adjusted to an unrepairable state. Finally, the historical abnormal data is displayed, and the target data with the deleted abnormal data is used for the next round of auditing. This reduces the workload of auditors and improves audit efficiency. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0053] Figure 1 A flowchart illustrating the data processing method provided in an embodiment of this application;

[0054] Figure 2 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0055] Figure 3 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0056] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0057] Reference numerals: 300-Data processing device, 310-Acquisition module, 320-Target historical abnormal data determination module, 330-Repair count determination module, 340-Processing module, 350-Display module, 400-Electronic device, 401-Processor, 402-Memory, 403-Communication component.

[0058] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0059] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0060] During the auditing of business data, auditors need to collect and analyze data to ensure its authenticity and accuracy. When auditing data, auditors may discover inaccuracies, such as excessive data volume, incomplete data, or data errors. These issues may affect the accuracy and reliability of the audit results. Therefore, auditors also need to conduct in-depth analysis of the audited anomalies to promptly identify and resolve problems, ensuring the normal operation and management of the telecommunications industry.

[0061] In related technologies, daily audit results are stored independently. Daily audit results represent a comprehensive review and record of the day's business data; storing them independently ensures data integrity and security. Audit results contain a large amount of business data and anomalies; storing them independently facilitates data analysis and reporting.

[0062] Therefore, independently stored audit results can help ensure data security and integrity, support data traceability and auditing, and provide data analysis and reporting capabilities.

[0063] However, during the development process, the R&D personnel discovered that if abnormal data appearing during the audit process was not corrected, it would reappear in the daily audit results. This meant that auditors would have to audit the abnormal data repeatedly on subsequent workdays, increasing their workload and reducing the audit speed.

[0064] To address the aforementioned issues, this application provides a data processing method. This method involves identifying target data and its abnormal data, then determining historical abnormal data from historical abnormal data based on the abnormal data, and further determining the number of times the historical abnormal data has been repaired. If the number of repairs for the historical abnormal data exceeds a preset number of repairs, the abnormal data is deleted from the target data, and its data status is adjusted to an unrepairable state. Finally, the historical abnormal data is displayed, and the target data with the abnormal data removed is used for the next round of auditing. This reduces the workload of auditors and improves audit efficiency.

[0065] The execution subject of the data processing method provided in this application embodiment can be a server, which can be a computer server. This embodiment does not impose any particular restrictions on the implementation method of the execution subject, as long as the execution subject can determine the target data and the abnormal data of the target data; determine the target historical abnormal data from the historical abnormal data based on the abnormal data; determine the number of times the target historical abnormal data has been repaired; if the number of times the target historical abnormal data has been repaired is greater than the preset number of repairs, then delete the abnormal data from the target data and adjust the data status of the target historical abnormal data to an unrepairable state; display the target historical abnormal data, and perform the next round of auditing on the target data with the abnormal data deleted.

[0066] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0067] Figure 1 This is a flowchart illustrating the data processing method provided in an embodiment of this application. The execution entity of this method can be a computer or other server; however, this embodiment does not impose any particular limitations. Figure 1 As shown, the method may include:

[0068] S101. Determine the target data and any abnormal data within the target data.

[0069] Abnormal data refers to data identified during the current round of auditing of the target data; that is, data showing anomalies obtained by auditors during the audit of the target data. Abnormal data may include inconsistencies in user information, mismatches in marketing plan data, and abnormal billing data.

[0070] The target data can be data that needs to be audited daily, such as business data, marketing campaign data, and billing data within the telecommunications industry. Business data can be data generated from routine services such as refunds, premium number registrations, and user bundling (monthly rebates, prepaid retention). Telecommunications operators launch marketing campaigns to adapt to market changes. Each campaign has specific business rules, typically including minimum spending, minimum prepaid deposits, user bundling requirements, time limits, and mutually exclusive restrictions. Users who subscribe to a campaign may receive rewards such as gifts, points, call credit, or premium cards. Marketing data is a series of data generated from these marketing campaigns. In this embodiment, the target data is business data collected by the telecommunications system from various business departments.

[0071] In some embodiments, determining the target data and abnormal data of the target data in step S100 includes the following steps:

[0072] Identify the target data, as well as the previous audit data, the previous abnormal data, and other data in the previous audit round.

[0073] Based on the nature of the business data, it can be divided into previous audit data, previous abnormal data, and other data. Previous audit data refers to data whose status was normal after the previous audit. Previous abnormal data refers to data whose status was abnormal after the previous audit. Other data refers to newly added target data after the previous audit. As business data is added daily, the target data for the current day will contain additional business data compared to the target data of the previous day; the aforementioned other data constitutes this newly added business data.

[0074] The current round of auditing processes the data to be audited from the previous round, resulting in the first audit result. In other words, the data whose status was abnormal after the previous round of auditing is audited again to confirm whether its status is normal or has been restored to normal, thus obtaining the first audit result.

[0075] The previous abnormal data is stored in the target database. After the previous abnormal data is repaired, it is then subjected to the current round of auditing to obtain the second audit result. In other words, data whose status was abnormal after the previous audit is stored in the target database. The previous abnormal data can be repaired in the target database, and a repair result is obtained. After the repair result is obtained, the previous abnormal data is subjected to the current round of auditing to obtain the second audit result. The target database can be connected to a repair port to repair abnormal business data using preset repair logic.

[0076] Perform current-round audit processing on other data to obtain the third audit result. Based on the first, second, and third audit results, obtain the target audit result. Audit the newly added business data in the current round to obtain the third audit result. Summarize the first, second, and third audit results to obtain the target audit result, which is the audit result of the current round.

[0077] Based on the above steps, abnormal data in the target data are identified according to the target audit results, and the summary abnormal data of the current round of audit is obtained.

[0078] Through the above steps, before the current round of auditing, business data is categorized according to its nature into previous audit data, previous abnormal data, and other data. These three categories are aggregated to form target data. Then, the three categories are audited separately to obtain the first audit result, the second audit result, and the third audit result. These results are then aggregated to obtain the aggregated abnormal data for the current round of auditing. Notably, previous abnormal data is entered into the target database for repair before the current round of auditing. This means that previous abnormal data undergoes system repair or manual repair to ensure the accuracy of abnormal data status assessment and the timeliness of repair, thereby improving audit accuracy.

[0079] S102. Based on the abnormal data, determine the target historical abnormal data from the historical abnormal data.

[0080] Among them, historical abnormal data refers to the historical data of the target data in the previous round of audit. Specifically, when the auditors or the telecommunications system conduct a round of audit on the target data of the day, the historical abnormal data is the collection of abnormal data audited on the previous day.

[0081] On the same day, the audit of the target data by auditors or telecommunications systems is the current round of audit, and also the round described below; the audit of the target data for the next day by auditors or telecommunications systems is the next round of audit; the audit of the target data for the day before yesterday by auditors or telecommunications systems is the previous round of audit. Here, "the day before yesterday" refers to the day before the current day, and "the day after today" refers to the day after the current day.

[0082] The target historical abnormal data is the data corresponding to the abnormal data. The abnormal data audited on the current day was also audited as abnormal in the previous round of audits, and the abnormal data audited in the previous round is the target historical abnormal data for this round of abnormal data.

[0083] In some embodiments, the correspondence between abnormal data and its corresponding target historical abnormal data is determined by data identifiers, specifically including the following steps:

[0084] First, the system acquires the data identifiers of the abnormal data and historical abnormal data. The system first obtains the data identifier of the currently detected abnormal data. The data identifier can be any information that uniquely identifies the data, such as data number, timestamp, or ID. Simultaneously, the system also acquires the data identifiers of historical abnormal data for subsequent comparative analysis.

[0085] Next, the data identifiers of the abnormal data are compared with the historical data identifiers of the historical abnormal data. The data identifiers of the current abnormal data are compared with the data identifiers of the historical abnormal data to check whether there is any historical abnormal data associated with the current abnormal data.

[0086] Finally, if a target historical data identifier corresponding to the current data identifier exists in the historical data identifiers, the target historical anomalous data is determined based on the target historical data identifier. If the system finds a target historical data identifier that matches the data identifier of the current anomalous data in the historical data identifiers, then the system determines the historical anomalous data associated with this target historical data identifier as the target historical anomalous data.

[0087] The abnormal data processing system based on data identification in this embodiment can effectively correlate new abnormal data with historical abnormal data, and use data identification to compare and analyze abnormal data, thereby improving the accuracy and efficiency of abnormal data processing.

[0088] After comparing the data identifiers of the abnormal data with the historical data identifiers of the historical abnormal data, in order to accurately determine the current round data status of the abnormal data, some embodiments of this application can perform the following processing on the abnormal data:

[0089] If no matching historical data identifier exists in the historical data identifier list, the abnormal data will be repaired to obtain a repair result. If the system cannot find a matching historical data identifier for the current abnormal data, it will repair the current abnormal data. Repair processing may include data cleaning, data verification, and data correction to restore the correctness of the abnormal data as much as possible.

[0090] Based on the repair process results, the data status of the abnormal data is determined. If the repair process is successful, the data status of the abnormal data can be adjusted to a normal state; if the repair process fails, the data status of the abnormal data will remain abnormal.

[0091] The abnormal data and its data status are stored in the target database. The target database is a database that stores historical abnormal data. The abnormal data and its data status are stored in the target database. The target database can be a database that stores historical abnormal data, used to store and manage all abnormal data and its corresponding data status.

[0092] S103. Determine the number of times the target historical abnormal data has been repaired.

[0093] After anomalies are detected in the current round of audits, the data can be moved to the target database for repair. Each audit round corresponds to one repair. That is, if anomaly data remains abnormal after n rounds of audits, it has undergone n repairs, where n is the number of times the data has been repaired. Therefore, the number of times historical anomaly data has been repaired is the same as the number of audit rounds it has undergone. The target database has a counting function; if business data with the same identifier remains abnormal after one repair, its repair count increases by one.

[0094] In some embodiments, after determining the number of times the target historical anomaly data has been repaired, the following steps are included:

[0095] If the number of times the target historical anomaly data has been repaired is less than or equal to the preset number of repairs, the anomaly data will be repaired, a repair result will be obtained, and the data status of the anomaly data will be adjusted to the repair processing status. The system determines whether to repair the current anomaly data based on the preset number of repairs and the number of times the target historical anomaly data has been repaired. If the number of repairs is less than or equal to the preset number of repairs, repair processing will be performed.

[0096] If the repair result indicates successful repair, the data status of the abnormal data will be changed from repair processing status to repair completed status, so that the next round of auditing can be carried out on the abnormal data in the target data whose data status is repair completed.

[0097] If the repair result indicates that the repair has failed, the data status of the abnormal data will be maintained as the repair processing status, and the number of times the target historical abnormal data has been repaired will be updated to obtain the updated number of repairs.

[0098] Based on the number of times the abnormal data has been repaired, the abnormal data is repaired until the number of times the abnormal data has been repaired exceeds the preset number of repairs, or until the abnormal data has been repaired, at which point the data status of the abnormal data is determined.

[0099] Based on the status of the abnormal data, the target data is updated in preparation for the next round of auditing.

[0100] This embodiment processes the current round of anomaly data based on the number of times the anomaly data has been repaired and the historical data status, adjusting its data status to effectively improve the efficiency of anomaly data processing. By repairing the anomaly data, the repaired data will be closer to the true value, making the data analysis results more accurate and reliable, and reducing false positives and false negatives in the audit process.

[0101] To reduce the efficiency of auditing work, some embodiments of this application perform repair processing on abnormal data included in the target database, specifically including the following steps:

[0102] If no target historical data identifier corresponding to the data identifier exists in the historical data identifier, the abnormal data is repaired to obtain the repair result.

[0103] Based on the repair process results, the data status of the abnormal data is determined. This status can be categorized into repair processing status, successful repair status, failed repair status, and unrepairable status. Specifically, the repair processing status indicates that the abnormal data is currently undergoing repair; the successful repair status indicates that the abnormal data has been repaired and its status has returned to normal; the failed repair status indicates that the abnormal data underwent this round of repair, but the repair failed; and the unrepairable status indicates that the abnormal data still fails to recover after a preset number of repair attempts, in which case the data status is adjusted to unrepairable.

[0104] The abnormal data and its status are stored in the target database, which is a database that stores historical abnormal data.

[0105] In some embodiments of this application, after the abnormal data has been repaired, the data status of the abnormal data is displayed to allow auditors to understand the processing status of the abnormal data during subsequent auditing work. Specifically, this includes the following steps:

[0106] Displays abnormal data and its status.

[0107] The system displays abnormal data and its status to the user, so that the user can intuitively understand the status and processing of abnormal data.

[0108] The system responds to user feedback regarding abnormal data and its status, providing feedback information. For example, after manual confirmation by auditors, the status of the abnormal data is manually adjusted.

[0109] The system receives user feedback based on abnormal data and its status, and generates feedback information accordingly. This feedback information may include user suggestions for handling abnormal data and adjusting the status of the abnormal data.

[0110] Based on the feedback, the abnormal data is removed from the target data, and the data status of the target historical abnormal data corresponding to the abnormal data is adjusted.

[0111] The system processes abnormal data based on feedback. If the feedback requests the deletion of abnormal data, the system will remove the abnormal data from the target data. Simultaneously, the system will adjust the data status of the corresponding historical abnormal data based on the feedback. For example, if the feedback requests that the abnormal data be processed to a repair-completed state, the system will adjust the data status of the target historical abnormal data to that state.

[0112] Through the above steps, this embodiment can process abnormal data based on feedback from auditors or other users, and adjust the data status of abnormal data and the data status of target historical abnormal data based on the processing results, thereby achieving comprehensive management and processing of abnormal data.

[0113] S104. If the number of times the target historical abnormal data has been repaired is greater than the preset number of repairs, then the abnormal data will be deleted from the target data and the data status of the target historical abnormal data will be adjusted to an unrepairable state.

[0114] The final data status of abnormal data is determined by the number of times the abnormal data has been repaired, so as to avoid the abnormal data from occupying repair resources indefinitely due to reasons that cannot be repaired.

[0115] S105. Display the target's historical abnormal data, and then conduct the next round of audit on the target data with the abnormal data deleted. The next round of audit is the audit following the current round of audit.

[0116] To facilitate querying changes to target data, after the target data status is adjusted, the details of the target data status change can be stored and archived in the database. Specifically:

[0117] When the data status of the target data changes, or when the data status of the target's historical abnormal data changes, change information is generated. The system monitors the data status of the target data and the data status of the target's historical abnormal data. When these data statuses change, the system generates corresponding change information. Change information may include the time of the change, the data status before the change, and the data status after the change.

[0118] The change information is stored in the change information database so that it can be displayed when a query command is received.

[0119] The system stores the generated change information in a change information database. This database stores and manages all change information for later querying and display. When the change information database receives a query command, the system retrieves the relevant change information and displays it to the user. Users can use query commands to find change information within a specific time period or under specific data conditions.

[0120] Therefore, this embodiment can monitor the data status changes of target data and target historical abnormal data in real time and generate corresponding change information. Furthermore, by storing the change information in a change information database and displaying the change information upon receiving a query command, this embodiment can provide users with a convenient data status change query service.

[0121] The data processing method provided in this application identifies target data and its abnormal data. Based on the abnormal data, it identifies target historical abnormal data from historical abnormal data and then determines the number of times the target historical abnormal data has been repaired. If the number of times the target historical abnormal data has been repaired exceeds a preset number of repairs, the abnormal data is deleted from the target data, and the data status of the target historical abnormal data is adjusted to an unrepairable state. The target historical abnormal data is then displayed, and the target data with the deleted abnormal data is audited in the next round. This ensures that recurring abnormal data caused by objective reasons can be removed from the target data by the data processing system, and the data status of the corresponding target historical abnormal data is adjusted to an unrepairable state, avoiding repeated audits of the abnormal data by auditors in subsequent audits, thereby improving audit efficiency. Furthermore, setting a preset number of repairs for abnormal data helps avoid false audits and omissions, thus improving audit accuracy.

[0122] Figure 2 Another data processing method provided in this application embodiment can be executed by a computer; however, this embodiment does not impose any particular limitation here. Figure 2 As shown, the method may include:

[0123] S201 and Sqoop collect business data into Hive, and use Spark SQL based on the Spark framework to perform audit calculations on the data, obtain the audited abnormal data, and write the abnormal data into HDFS.

[0124] Sqoop is a tool for large-scale data migration between Hadoop and relational databases, allowing users to easily extract data from production databases and import it into Hive. Spark is a large-scale data processing framework that can efficiently process and analyze massive datasets. SparkSQL is a module of Spark that provides SQL query functionality, enabling users to query and perform calculations on data using SQL statements.

[0125] S202. Based on the Elasticsearch-Hadoop component, read historical abnormal data from Elasticsearch and associate the abnormal data with the historical abnormal data using the primary key.

[0126] Elasticsearch is a distributed search and analytics engine that provides powerful full-text search capabilities. Elasticsearch-Hadoop is a Spark integration plugin designed for Elasticsearch, allowing for easy data retrieval from Elasticsearch. A primary key is a field that uniquely identifies a record, enabling the association of two sets of data. In auditing scenarios, where data volumes can be very large, primary keys help auditors quickly locate the specific records requiring auditing, thereby accelerating the audit process and improving accuracy.

[0127] S203. Input the abnormal data and its associated historical abnormal data into a preset finite state machine. The finite state machine outputs the latest data state of the abnormal data.

[0128] A finite state machine (FSM) is a mathematical computational model that represents a finite number of states and the transitions and actions between these states. It has three characteristics: the total number of states is finite; at any given time, it is in only one state; and under certain conditions, it will transition from one state to another. For example, this embodiment divides the audited abnormal data into the following states: Abnormal, Back to Normal, Manually Confirmed Non-Abnormal, Manually Confirmed Repaired, Manually Confirmed Unrepairable, Manually Confirmed Processing, System Processing, Repaired (Manual), Repaired (System), System Repair Failed, Manually Confirmed Non-Abnormal Repaired. The specific definitions of these states are as follows:

[0129] "Abnormal" is the initial state of the audit results; "Restored to Normal" means the data has been restored to normal; "Manually Confirmed Non-Abnormal" means the auditor believes the audit logic is flawed and the audit result is not abnormal data; "Manually Confirmed Repaired" means the auditor notifies the audit system that the data has been repaired; "Manually Confirmed Unrepairable" means the data is known to have problems but cannot be repaired; "Manually Confirmed Processing" means the auditor notifies the audit system that data repair is in progress; "System Processing" means the audit system is automatically repairing the abnormal data identified during auditing; "Repaired (Manual)" means the data marked as "Manually Confirmed Repaired" has indeed been repaired; "Repaired (System)" means the data automatically repaired by the system has been repaired; "System Repair Failed" means the system performed automatic repair, but the repair failed; "Manually Confirmed Non-Abnormal Repaired" means the data marked as "Manually Confirmed Non-Abnormal" has been repaired.

[0130] The data processing system determines the new data status based on whether historical abnormal data exists, the abnormal status of business data, and whether it reappears in the audit results of the new day.

[0131] When abnormal data first appears, it is in an abnormal state. Auditors or the system can label the business data as having different states, including: manually confirmed as not abnormal, manually confirmed as repaired, manually confirmed as unrepairable, and manually confirmed as being processed.

[0132] If the abnormal data is not being displayed for the first time, the status will be adjusted according to the following rules: Abnormal data remains in the abnormal state; restored to normal and changed to abnormal; manually confirmed as non-abnormal remains manually confirmed as non-abnormal; manually confirmed as non-abnormal and repaired and changed to manually confirmed as non-abnormal; manually confirmed as repaired and changed to abnormal; repaired (manually) and changed to abnormal; manually confirmed as unable to be repaired remains manually confirmed as unable to be repaired; manually confirmed as being processed remains manually being processed; repaired (system) and changed to abnormal; system repair failed and changed to abnormal.

[0133] If previously abnormal data existed but no longer appears in the new day's abnormal results, the data is considered no longer abnormal and is adjusted according to the following rules: Abnormal changes to "Restored to Normal"; "Restored to Normal" remains in the "Restored to Normal" state; "Manually confirmed as non-abnormal" changes to "Manually confirmed as non-abnormal and repaired"; "Manually confirmed as non-abnormal and repaired" remains in the "Manually confirmed as non-abnormal and repaired" state; "Manually confirmed as non-abnormal and repaired" changes to "Repaired (Manual)"; "Repaired (Manual)" remains in the "Repaired (Manual)" state; "Manually confirmed as unable to repair" changes to "Restored to Normal"; "Manually confirmed as being processed" changes to "Repaired (Manual)"; "Repaired (System)" remains in the "Repaired (System)" state; "System repair failed" changes to "Restored to Normal".

[0134] The system can automatically repair abnormal data detected during audits, changing the data from an abnormal state to a state where the system is processing it. If the repair is successful, the data is changed from a state where the system is processing it to a state where it has been repaired; if the repair fails, the data is changed from a state where the system is processing it to a state where the system repair failed.

[0135] S204. Write the latest data status of the abnormal data to Elasticsearch.

[0136] To facilitate data status tracking, the system records the time of each data status change, the status before and after the change, the person making the change, the reason for the change, as well as the time of the first and last occurrence of abnormal data.

[0137] Figure 3 This is a schematic diagram of the structure of the data processing apparatus 300 provided in an embodiment of this application. Figure 3 As shown, the data processing device 300 includes: an acquisition module 310, a target historical abnormal data determination module 320, a repair count determination module 330, a processing module 340, and a display module 350. Wherein:

[0138] The acquisition module 310 is used to determine the target data and any abnormal data within the target data.

[0139] The target historical anomaly data determination module 320 is used to determine the target historical anomaly data from the historical anomaly data based on the anomaly data.

[0140] The repair count determination module 330 is used to determine the repair count of the target historical abnormal data;

[0141] Processing module 340 is used to adjust abnormal data based on the number of times the target historical abnormal data has been repaired;

[0142] The display module 350 is used to display the target's historical abnormal data and to conduct the next round of auditing on the target data with the abnormal data deleted.

[0143] In this embodiment of the application, the acquisition module 310 can also be specifically used for:

[0144] Obtain the data identifier of the abnormal data and the data identifier of the historical abnormal data; compare the data identifier of the abnormal data with the historical data identifier of the historical abnormal data; if there is a target historical data identifier corresponding to the data identifier in the historical data identifier, then determine the target historical abnormal data based on the target historical data identifier.

[0145] In this embodiment of the application, the acquisition module 310 can also be specifically used for:

[0146] If no target historical data identifier corresponding to the data identifier exists in the historical data identifier, the abnormal data is repaired to obtain the repair result; based on the repair result, the data status of the abnormal data is determined; the abnormal data and the data status of the abnormal data are stored in the target database, which is the database that stores historical abnormal data.

[0147] In this embodiment of the application, the acquisition module 310 can also be specifically used for:

[0148] The process involves identifying the target data, the previous auditable data, the previous abnormal data, and other data from the previous audit round. The previous auditable data refers to data whose status was normal after the previous audit round, the previous abnormal data refers to data whose status was abnormal after the previous audit round, and the other data refers to newly added data after the previous audit round. The previous auditable data is processed in the current round to obtain the first audit result. The previous abnormal data is stored in the target database so that after the previous abnormal data is repaired, it is processed in the current round to obtain the second audit result. The other data is processed in the current round to obtain the third audit result. Based on the first, second, and third audit results, the target audit result is obtained. Based on the target audit result, the abnormal data within the target data is identified.

[0149] In this embodiment of the application, the target historical anomaly data determination module 320 can also be specifically used for:

[0150] Obtain the data identifier for abnormal data, as well as the data identifier for historical abnormal data;

[0151] Compare the data identifiers of abnormal data with the historical data identifiers of historical abnormal data;

[0152] If a target historical data identifier exists in the historical data identifier, then the target historical abnormal data is determined based on the target historical data identifier.

[0153] In this embodiment of the application, the target historical anomaly data determination module 320 can also be specifically used for:

[0154] If no target historical data identifier corresponding to the data identifier exists in the historical data identifier, the abnormal data is repaired to obtain the repair result;

[0155] Based on the repair results, determine the data status of the abnormal data;

[0156] The abnormal data and its status are stored in the target database, which is a database that stores historical abnormal data.

[0157] In this embodiment of the application, the repair count determination module 330 can also be specifically used for:

[0158] If the number of times the target historical abnormal data has been repaired is less than or equal to the preset number of repairs, the abnormal data will be repaired to obtain the repair result, and the data status of the abnormal data will be adjusted to the repair processing status.

[0159] If the repair result indicates successful repair, the data status of the abnormal data will be changed from repair processing status to repair completed status, so that the next round of auditing can be carried out on the abnormal data in the target data whose data status is repair completed.

[0160] In this embodiment of the application, the repair count determination module can also be specifically used for:

[0161] If the repair result indicates that the repair has failed, the data status of the abnormal data will be maintained as the repair processing status, and the number of times the target historical abnormal data has been repaired will be updated to obtain the updated number of repairs.

[0162] Based on the number of times the update has been repaired, the abnormal data is repaired until the number of updates has been repaired exceeds the preset number of repairs, or the abnormal data has been repaired, and then the data status of the abnormal data is determined.

[0163] Based on the status of the abnormal data, the target data is updated in preparation for the next round of auditing.

[0164] In this embodiment of the application, the processing module 340 may also be specifically used for:

[0165] If the repair result indicates that the repair has failed, the data status of the abnormal data will be maintained as the repair processing status, and the number of times the target historical abnormal data has been repaired will be updated to obtain the updated number of repairs.

[0166] Based on the number of times the update has been repaired, the abnormal data is repaired until the number of updates has been repaired exceeds the preset number of repairs, or the abnormal data has been repaired, and then the data status of the abnormal data is determined.

[0167] Based on the status of the abnormal data, the target data is updated in preparation for the next round of auditing.

[0168] In this embodiment of the application, the processing module 340 may also be specifically used for:

[0169] Displays abnormal data and its status;

[0170] Responding to user feedback based on abnormal data and the data status of the abnormal data, feedback information is obtained;

[0171] Based on the feedback, the abnormal data is removed from the target data, and the data status of the target historical abnormal data corresponding to the abnormal data is adjusted.

[0172] In this embodiment of the application, the display module 350 can also be specifically used for:

[0173] When the data status of the target data changes, or when the data status of the target historical abnormal data changes, change information is generated;

[0174] The change information is stored in the change information database so that it can be displayed when a query command is received.

[0175] As can be seen from the above, the data processing device in this embodiment comprises an acquisition module 310 for determining target data and abnormal data of the target data; a target historical abnormal data determination module 320 for determining target historical abnormal data from historical abnormal data based on the abnormal data; a repair count determination module 330 for determining the number of times the target historical abnormal data has been repaired; a processing module 340 for adjusting the abnormal data based on the number of times the target historical abnormal data has been repaired; and a display module 350 for displaying the target historical abnormal data and performing the next round of auditing on the target data with the abnormal data deleted, thereby saving the auditing workload of the auditors and reducing auditing efficiency.

[0176] Figure 4 This is a structural schematic diagram of a vehicle provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 may include a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a communication component 403, and other components. The processor 401, memory 402, and communication component 403 are connected via a bus.

[0177] In the specific implementation process, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the data processing method described above.

[0178] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0179] In the above Figure 4 In the illustrated embodiments, it should be understood that the processor 401 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0180] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0181] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0182] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0183] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0184] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0185] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0186] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0187] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not contradict each other, they should be considered within the scope of this specification.

[0188] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0189] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, include: Identify target data and abnormal data of the target data, wherein the abnormal data is data identified during the current round of auditing of the target data; Based on the abnormal data, target historical abnormal data is determined from the historical abnormal data. The historical abnormal data is the historical data of the target data in the previous round of audit. The target historical abnormal data is the data corresponding to the abnormal data. The previous round of audit is the audit before the current round of audit. Determine the number of times the target historical anomaly data has been repaired; If the number of times the target historical abnormal data has been repaired is greater than the preset number of repairs, then the abnormal data will be deleted from the target data and the data status of the target historical abnormal data will be adjusted to an unrepairable state. Display the target's historical abnormal data, and then perform the next round of audit on the target data after deleting the abnormal data, wherein the next round of audit is the audit following the current round of audit; The determined target data and the abnormal data of the target data include: The target data, the previous audit data, the previous abnormal data, and other data of the target data in the previous audit are determined. The previous audit data is the data whose data status is normal after the previous audit. The previous abnormal data is the data whose data status is abnormal after the previous audit. The other data is the newly added data of the target data after the previous audit. The previous audit data is processed in the current round of auditing to obtain the first audit result; The previous abnormal data is stored in the target database. After the previous abnormal data is repaired, the previous abnormal data is subjected to the current round of audit processing to obtain the second audit result. The other data are subjected to the current round of audit processing to obtain the third audit result; the target audit result is obtained based on the first audit result, the second audit result, and the third audit result. Based on the audit results of the target, abnormal data in the target data are identified.

2. The method according to claim 1, characterized in that, After determining the number of times the target historical anomaly data has been repaired, the method includes: If the number of times the target historical abnormal data has been repaired is less than or equal to the preset number of repairs, then the abnormal data is repaired to obtain a repair result, and the data status of the abnormal data is adjusted to the repair processing status. If the repair result indicates successful repair, the data status of the abnormal data is adjusted from the repair processing status to the repair completed status, so as to conduct the next round of auditing on the abnormal data in the target data whose data status is repair completed.

3. The method according to claim 2, characterized in that, After the method proceeds to repair the abnormal data if the number of repairs to the target historical abnormal data is less than or equal to a preset number of repairs, obtains a repair result, and adjusts the data status of the abnormal data to a repair processing status, the method further includes: If the repair result indicates that the repair has failed, the data status of the abnormal data is kept in the repair processing status, and the number of times the target historical abnormal data has been repaired is updated to obtain the updated number of repairs. Based on the number of times the update has been repaired, the abnormal data is repaired until the number of times the update has been repaired is greater than the preset number of repairs, or the abnormal data is repaired, and then the data status of the abnormal data is determined. Based on the data status of the abnormal data, the target data is updated for the next round of auditing.

4. The method according to claim 3, characterized in that, After repairing the abnormal data and obtaining the repair result, the method further includes: Display the abnormal data and its data status; In response to the user's feedback operation based on the abnormal data and the data status of the abnormal data, feedback information is obtained; Based on the feedback information, the abnormal data is deleted from the target data, and the data status of the target historical abnormal data corresponding to the abnormal data is adjusted.

5. The method according to claim 1, characterized in that, The step of determining the target historical abnormal data from the historical abnormal data based on the abnormal data includes: Obtain the data identifier of the abnormal data and the data identifier of the historical abnormal data; Compare the data identifier of the abnormal data with the historical data identifier of the historical abnormal data; If a target historical data identifier corresponding to the data identifier exists in the historical data identifier, then the target historical abnormal data is determined based on the target historical data identifier.

6. The method according to claim 5, characterized in that, After comparing the data identifier of the abnormal data and the historical data identifier of the historical abnormal data, the method further includes: If no target historical data identifier corresponding to the data identifier exists in the historical data identifier, the abnormal data is repaired to obtain the repair result; Based on the repair process results, determine the data status of the abnormal data; The abnormal data and its data status are stored in a target database, wherein the target database is a database that stores the historical abnormal data.

7. The method according to claim 1, characterized in that, The method further includes: When the data status of the target data changes, or when the data status of the target historical abnormal data changes, change information is generated; The change information is stored in a change information database so that it can be displayed when the change information database receives a query command.

8. A data processing apparatus, characterized in that, include: The acquisition module is used to determine the target data and any abnormal data within the target data. The target historical anomaly data determination module is used to determine the target historical anomaly data from the historical anomaly data based on the anomaly data. The repair count determination module is used to determine the number of times the target historical abnormal data has been repaired; The processing module is used to adjust the abnormal data according to the number of times the target historical abnormal data has been repaired; The display module is used to display the target's historical abnormal data and to perform the next round of auditing on the target data after deleting the abnormal data; The acquisition module is specifically used for: The target data, the previous audit data, the previous abnormal data, and other data of the target data in the previous audit are determined. The previous audit data is the data whose data status is normal after the previous audit. The previous abnormal data is the data whose data status is abnormal after the previous audit. The other data is the newly added data of the target data after the previous audit. The previous audit data is processed in the current round of auditing to obtain the first audit result; The previous abnormal data is stored in the target database. After the previous abnormal data is repaired, the previous abnormal data is subjected to the current round of audit processing to obtain the second audit result. The other data are subjected to the current round of auditing to obtain the third audit result; Based on the first audit result, the second audit result, and the third audit result, the target audit result is obtained; Based on the audit results of the target, abnormal data in the target data are identified.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images