Data file exception detection method and device, electronic equipment and storage medium

Abstract

The application discloses an abnormality detection method and device of a data file, an electronic device and a storage medium, and relates to the technical field of big data or other related fields. The method comprises the following steps: receiving a data request and determining a data check template; performing sharding extraction on a target database table according to a data extraction range indicated by the data request to obtain batch sub-files; for each batch sub-file, performing data check according to the data check template to obtain a check sub-result; in the case that any check sub-result indicates a check error, generating an error report file, sending an abnormality prompt to a monitoring system, and waiting for a data correction instruction returned by the monitoring system; and in the case that all check sub-results indicate a check pass, merging the batch sub-files to obtain a batch data file, and sending the batch data file to a target business system. Through the application, the technical problem that the abnormality detection mechanism in the data transmission process is insufficient in the related art and the error rate of downstream business data is excessively high is solved.

Description

Technical Field

[0001] This invention relates to the field of big data technology or other related fields. Specifically, it relates to a method and apparatus for detecting anomalies in data files, electronic devices, and storage media. Background Technology

[0002] With the booming development of the financial industry and the continuous expansion of core businesses, the amount of data involved in data transmission and processing technologies is growing exponentially, and the requirements for the frequency and speed of data transmission are increasing. In particular, the transmission of batch files under a distributed architecture has become an important bridge connecting upstream data generation and downstream business processing. However, the anomaly detection mechanism for data transmission in this process is significantly inadequate.

[0003] Traditional methods often rely on simplified verification rules or manual sampling, which are not only inefficient, but also difficult to guarantee accuracy when faced with complex and ever-changing data formats and content. In particular, when the data volume is large, it is difficult to fully cover all data points, which significantly increases the risk of downstream business systems receiving and processing erroneous data.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, electronic device, and storage medium for detecting anomalies in data files, so as to at least solve the technical problem that the anomaly detection mechanism in the data transmission process is insufficient in the related art, resulting in an excessively high error rate of downstream business data.

[0006] To achieve the above objectives, according to one aspect of this application, an anomaly detection method for data files is provided. The method includes: receiving a data request and determining a data verification template based on the data request, wherein the data request is at least used to define the data extraction scope and the target business system; performing data segmentation and extraction on a target database table according to the data extraction scope indicated by the data request to obtain N batch sub-files, where N is a positive integer; for each batch sub-file, performing data verification on the batch sub-file according to the data verification template to obtain a verification sub-result; if any verification sub-result indicates a verification error, generating an error file and sending an anomaly prompt to a monitoring system, and waiting for the monitoring system to return a data correction instruction; if all verification sub-results indicate that the verification passed, merging the N batch sub-files to obtain a batch data file, and sending the batch data file to the target business system.

[0007] Further, the step of determining the data verification template based on the data request includes: determining the requesting department and data format requirements based on the sending source and request content of the data request, wherein the data format requirements include: field information, file format, and file name; retrieving M existing verification rules based on the requesting department and the target business system, wherein M is a positive integer; generating R incremental verification rules based on the data format requirements, wherein R is a positive integer; and integrating the existing verification rules and the incremental verification rules to obtain the data verification template.

[0008] Furthermore, after determining the data verification template based on the data request, the method further includes: upon receiving an update request for the data request, parsing the update request to obtain update information; adjusting the data verification template based on the update information to obtain an updated data request; and / or, if the update information includes an updated data extraction range, re-sharding and extracting the target database table based on the updated data range.

[0009] Further, the step of segmenting and extracting the target database table according to the data extraction range indicated by the data request to obtain N batch sub-files includes: determining N data extraction sub-ranges and extraction order based on the data extraction range; extracting data for each data extraction sub-range according to the extraction order; after each extraction, creating a corresponding sub-data file based on the extracted data, and naming the sub-data file according to the extraction order to obtain a sub-data file name containing a file number, wherein the file number is generated based on the extraction order or the identifier of the data extraction sub-range; and integrating all sub-data files according to the file number to obtain N batch sub-files.

[0010] Further, for each of the batch sub-files, the step of performing data verification on the batch sub-files according to the data verification template to obtain a verification sub-result includes: reading the data verification template to obtain the rule categories of the data verification rules, wherein the rule categories at least include: data format verification rules, special character verification rules, data content verification rules, and data integrity verification rules; for each rule category, extracting all the data verification rules corresponding to the rule category, verifying the data in the batch sub-files, and generating verification records during the verification process; if the verification records indicate that the batch sub-files conform to the data verification rules of all the rule categories, setting the verification sub-result as verification passed, otherwise setting it as verification error.

[0011] Furthermore, in the case where any of the verification sub-results indicates a verification error, the step of generating an error file includes: extracting the verification record; reading the verification record to obtain error information, wherein the error information includes: error code, sub-data file name, and error data location information; generating the error file based on the error information, and uploading it to the monitoring system.

[0012] Furthermore, upon receiving the data correction instruction, the method further includes: parsing the data correction instruction to obtain the data identifier, modification suggestion, and priority information of the abnormal data to be corrected; locating the batch sub-file where the abnormal data is located based on the priority information and the data identifier; after locating the batch sub-file, selecting the abnormal data based on the data identifier and modifying the abnormal data based on the modification suggestion; and re-verifying the data of the modified batch sub-file according to the data verification template, and updating the corresponding verification sub-result.

[0013] To achieve the above objectives, according to another aspect of this application, an anomaly detection device for data files is also provided. The device includes: a receiving unit, configured to receive a data request and determine a data verification template based on the data request, wherein the data request at least specifies the data extraction scope and the target business system; an extraction unit, configured to perform fragmented extraction on a target database table according to the data extraction scope indicated by the data request, obtaining N batch sub-files, where N is a positive integer; a verification unit, configured to perform data verification on each batch sub-file according to the data verification template, obtaining a verification sub-result; a generation unit, configured to generate an error file and send an anomaly prompt to a monitoring system when any verification sub-result indicates a verification error, and wait for the monitoring system to return a data correction instruction; and a sending unit, configured to merge the N batch sub-files to obtain a batch data file when all verification sub-results indicate successful verification, and send the batch data file to the target business system.

[0014] Further, the receiving unit includes: a first determining module, configured to determine the requesting department and data format requirements based on the sending source and request content of the data request, wherein the data format requirements include: field information, file format, and file name; a retrieving module, configured to retrieve M existing verification rules based on the requesting department and the target business system, wherein M is a positive integer; a generating module, configured to generate R incremental verification rules based on the data format requirements, wherein R is a positive integer; and a first integrating module, configured to integrate the existing verification rules and the incremental verification rules to obtain the data verification template.

[0015] Furthermore, the anomaly detection device for the data file further includes: a first parsing module, used to parse the update request to obtain update information when receiving an update request for the data request; an adjustment module, used to adjust the data verification template based on the update information to obtain an updated data request; and a first extraction module, used to re-shard and extract the target database table based on the updated data range when the update information includes an updated data extraction range.

[0016] Further, the extraction unit includes: a second determining module, used to determine N data extraction sub-ranges and extraction order based on the data extraction range; a second extraction module, used to extract data from each of the data extraction sub-ranges according to the extraction order; a naming module, used to create corresponding sub-data files based on the extracted data after each extraction, and name the sub-data files according to the extraction order to obtain sub-data file names containing file numbers, wherein the file numbers are generated based on the extraction order or the identifier of the data extraction sub-range; and a second integration module, used to integrate all sub-data files according to the file numbers to obtain N batch sub-files.

[0017] Further, the verification unit includes: a first reading module, used to read the data verification template to obtain the rule categories of the data verification rules, wherein the rule categories include at least: data format verification rules, special character verification rules, data content verification rules, and data integrity verification rules; a verification module, used to extract all the data verification rules corresponding to each rule category, verify the data in the batch sub-files, and generate verification records during the verification process; and a setting module, used to set the verification result as verified if the verification record indicates that the batch sub-files conform to the data verification rules of all the rule categories, otherwise set it as verified as verified as incorrect.

[0018] Furthermore, the generation unit includes: a third extraction module for extracting the verification record; a second reading module for reading the verification record to obtain error information, wherein the error information includes: an error code, a sub-data file name, and error data location information; and an upload module for generating the error file based on the error information and uploading it to the monitoring system.

[0019] Furthermore, the anomaly detection device for the data file further includes: a second parsing module, used to parse the data correction instruction upon receiving it, and obtain the data identifier, modification suggestion, and priority information of the abnormal data to be corrected; a positioning module, used to locate the batch sub-file where the abnormal data is located based on the priority information and the data identifier; a modification module, used to select the abnormal data based on the data identifier after locating the batch sub-file, and modify the abnormal data based on the modification suggestion; and an update module, used to re-verify the data of the modified batch sub-file according to the data verification template, and update the corresponding verification sub-results.

[0020] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the abnormal detection method for data files described in any one of the above claims.

[0021] To achieve the above objectives, according to another aspect of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the abnormal detection method for data files described in any of the above claims.

[0022] To achieve the above objectives, according to another aspect of this application, a computer program product is also provided, including computer instructions, wherein when the computer instructions are executed by a processor, they implement the steps of the abnormal detection method for data files described in any of the above claims.

[0023] This invention proposes an anomaly detection method for data files. First, a data request is received, and a data verification template is determined based on the data request. The data request at least clarifies the data extraction scope and the target business system. Then, the target database table is segmented and extracted according to the data extraction scope indicated by the data request, resulting in N batch sub-files, where N is a positive integer. For each batch sub-file, data verification is performed according to the data verification template, yielding verification sub-results. If any verification sub-result indicates a verification error, an error file is generated, and an anomaly alert is sent to the monitoring system. The system then waits for a data correction instruction from the monitoring system. Finally, if all verification sub-results indicate successful verification, the N batch sub-files are merged to obtain a batch data file, which is then sent to the target business system.

[0024] This invention employs an intelligent and dynamic data verification template customization method. Through deeply integrated distributed data processing and anomaly detection techniques, it achieves comprehensive data quality control before batch file transmission, thereby realizing the technical effect of accurately capturing and promptly correcting data anomalies. This solves the technical problem of insufficient anomaly detection mechanisms in the data transmission process in related technologies, which leads to excessively high error rates in downstream business data.

[0025] Specifically, this invention first strengthens the parsing and response of data requests at the source. By meticulously analyzing the key elements of the data request—the data extraction scope and the target business system—it intelligently generates a data verification template covering multiple dimensions of verification rules, including data format, special characters, content accuracy, and completeness, ensuring the comprehensiveness and relevance of subsequent data verification. Then, leveraging the advantages of a distributed architecture, it efficiently segments and extracts massive amounts of data from the target database tables, forming a series of batch sub-files, each of which is the object of data verification. Next, based on the previously customized data verification template, it performs fine-grained data verification on each batch sub-file, generating corresponding verification sub-results to detect and locate data anomalies as early as possible. Once a verification error is detected, this invention immediately generates a detailed error report file, sends an anomaly alert to the monitoring system, and initiates an automated data correction process. When all verification sub-results indicate no anomalies, the batch sub-files are seamlessly merged to form a high-quality batch data file, which is then securely transmitted to the target business system. This invention achieves comprehensive quality control before data transmission, effectively reducing the risk of downstream business system interruptions due to data anomalies, improving the reliability and efficiency of data processing, and providing a solid guarantee for data security and business continuity in the financial industry. Attached Figure Description

[0026] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0027] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing an anomaly detection method for data files is shown.

[0028] Figure 2 This is a flowchart of an optional data file anomaly detection method according to an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of an optional data file anomaly detection device according to an embodiment of the present invention;

[0030] Figure 4 This is a structural block diagram of an electronic device for an abnormal detection method of execution data files according to an embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:

[0034] Distributed architecture refers to a design pattern that distributes computing tasks across multiple computing nodes in a network for simultaneous processing. In this invention, distributed architecture is used for batch data sharding, extraction, and verification, improving the speed and efficiency of data processing.

[0035] In financial data processing, batch files refer to large-scale data sets processed at once. They are typically used for periodic data transmission or report generation and contain various types of data, such as customer information and transaction records. They need to be verified to ensure the accuracy and integrity of the data.

[0036] It should be noted that the data file anomaly detection method and apparatus in this application can be used in the field of big data technology for quality control before the transmission of batch data files, and can also be used in any field other than big data technology for quality control before the transmission of batch data files. This application does not limit the application field of the data file anomaly detection method and apparatus.

[0037] It should be noted that all relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, processing, transmission, provision, disclosure, use, and handling of such data comply with the laws, regulations, and standards of the relevant regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse access. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0038] The information collection (e.g., user voice, video, and text collection) and analysis operations involved in this application have provided users with corresponding operation entry points during execution, allowing users to choose to agree to or reject the automated decision results; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0039] The following embodiments of the present invention can be applied to various systems / applications / devices that require batch data file anomaly detection and pre-transmission quality control, enabling an intelligent data verification and correction system based on a distributed architecture. The present invention uses intelligently customized data verification templates to segment and extract batch data files and perform data verification. Then, when abnormal data is detected, an error file is generated and an anomaly alert is sent to the monitoring system. This allows for better capture of data anomalies and timely correction at an early stage, avoiding negative impacts of abnormal data on downstream business systems and ensuring the accuracy and security of data transmission and processing.

[0040] The present invention will now be described in detail with reference to various embodiments.

[0041] Example 1

[0042] According to an embodiment of the present invention, an embodiment of an anomaly detection method for data files is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] The data file anomaly detection method provided in Embodiment 1 of the present invention can be executed on a mobile terminal, computer terminal or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing an anomaly detection method for data files is shown. Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0044] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0045] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data file anomaly detection method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned data file anomaly detection method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0046] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0047] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0048] Under the above operating environment, the present invention provides, as follows: Figure 2 The data file anomaly detection method shown is implemented by a distributed data processing and verification system. It combines intelligent data verification template customization and deep verification rule application technology for quality control scenarios before batch data file transmission in the financial industry. In particular, it addresses the problem of insufficient anomaly detection mechanisms during data transmission, which leads to excessively high error rates in downstream business data. By integrating a distributed architecture and custom data verification strategies, the method is implemented through five key steps: receiving data requests, customizing data verification templates, extracting and verifying data in segments, capturing and correcting abnormal data, and finally merging and transmitting data files. This aims to intercept and correct data anomalies in advance, ensuring data transmission quality and the stable operation of downstream business systems.

[0049] The embodiments of the present invention will now be described in detail with reference to each specific step.

[0050] Figure 2 This is a flowchart of an optional data file anomaly detection method according to an embodiment of the present invention, such as... Figure 2 As shown, the method includes the following steps:

[0051] Step S201: Receive a data request and determine a data verification template based on the data request, wherein the data request is used to at least clarify the scope of data extraction and the target business system.

[0052] Specifically, a data request is a request initiated by a downstream business system to an upstream system in a batch data processing workflow to obtain specific types of data. This includes the data type, time range, data format, and other specific conditions, such as field filtering rules or requirements specific to a particular business scenario. A data validation template is a predefined data format and rule set designed based on the data extraction scope specified in the data request and the requirements of the target business system. It includes various validation rules for data fields, such as data type, data length, numerical range, special character restrictions, and logical relationships between data.

[0053] The data extraction scope refers to the time period, data type, and field selection information explicitly specified in the data request, defining the specific boundaries for data extraction from the source database. It serves as a crucial basis for developing data validation templates and implementing data validation. The target business system refers to the system where the data is ultimately transmitted and used; this can be an internal business processing system or an external data receiving platform from regulatory agencies or partners.

[0054] Optionally, in the data file anomaly detection method provided in this embodiment of the invention, the step of determining the data verification template based on the data request includes: determining the requesting department and data format requirements based on the sending source and request content of the data request, wherein the data format requirements include: field information, file format, and file name; retrieving M existing verification rules based on the requesting department and the target business system, wherein M is a positive integer; generating R incremental verification rules based on the data format requirements, wherein R is a positive integer; and integrating the existing verification rules and the incremental verification rules to obtain the data verification template.

[0055] The source of a data request is typically the business department requiring the data or an external partner. The request content details the specific requirements for the data, including but not limited to the time range, fields involved, data format (such as CSV (Comma-Separated Values), JSON (JavaScript Object Notation), etc.), and file naming conventions. Determining the source and request content ensures that data extraction and validation accurately match business needs, avoids the transmission of useless data, and facilitates the categorization and prioritization of data requests.

[0056] Furthermore, the requesting department refers to the internal or external entity that makes the data request, has a specific purpose for using the data, and has clear requirements for the data's format and content. Data format requirements specify in detail the structure and presentation of the data, including field types, lengths, formats, as well as file layout and naming conventions, ensuring that the data can be correctly parsed and used during transmission and reception.

[0057] Existing validation rules refer to the set of rules that already exist in the system. These rules can be defined based on historical data processing experience, business specifications, or industry standards. The purpose of retrieving existing validation rules is to leverage existing knowledge and experience to quickly build the basic framework for validation templates, ensuring that the formulation of new templates takes into account past validation logic and known abnormal data patterns.

[0058] Incremental validation rules refer to newly generated validation rules based on specific requirements and potential data anomaly patterns proposed in new data requests. The purpose of generating incremental validation rules is to adapt to the personalized needs of data requests and changes in the current business environment, ensuring that the validation template can fully cover new data formats and business rules.

[0059] Another point to note is that, in addition to field information, file format, and file name, data format requirements can also include field encoding methods, data compression formats (such as gzip, zip, etc.), encryption requirements, data delimiters (such as commas, tabs, etc.), and data row structure specifications.

[0060] Optionally, in the data file anomaly detection method provided in this embodiment of the invention, after determining the data verification template based on the data request, the method further includes: upon receiving an update request for the data request, parsing the update request to obtain update information; adjusting the data verification template based on the update information to obtain the updated data request; and / or, if the update information includes an updated data extraction range, re-sharding and extracting the target database table based on the updated data range.

[0061] It should be noted that an update request refers to a request from the requesting department to modify the data request again after the original data request has been issued and executed, due to changes in business needs, adjustments to data formats, or updates to the requirements of the target business system. Such update requests are common in dynamic data processing environments. When external regulatory rules change, internal business processes are adjusted, or new data usage scenarios are discovered, the original data request may need to be modified accordingly to adapt to new business or technical requirements.

[0062] The update information includes specific modifications to the original data request, which may include, but are not limited to, adjustments to the data extraction scope, additions or deletions of field information, changes to data format, updates to file naming conventions, and new validation rules. The purpose of the update information is to ensure that the data validation template can reflect the latest data formats and business requirements in real time, thereby guaranteeing the accuracy and integrity of the data.

[0063] Adjusting the data validation template based on updated information specifically refers to modifying the existing data validation template according to the specific requirements in the updated information after receiving an update request. This involves:

[0064] Validation rule updates: If the update information involves changes in data format, such as adjustments to field lengths or changes in the handling of special characters, the rules in the validation template need to be adjusted accordingly.

[0065] Adding or deleting field information: If changes in business requirements require the addition or deletion of certain fields in the data file, the relevant field information and validation logic in the validation template also need to be modified accordingly.

[0066] File naming and format adjustments: If the updated information involves changes to file naming rules or data formats, the descriptions of the corresponding sections in the verification template should also be updated accordingly.

[0067] Updating the data extraction scope refers to modifying the time period, data type, or other conditions for data extraction in an update request. For example, a business department might initially request data from the past month, but a subsequent update request might request expansion to the past six months or a change in the data extraction frequency, from daily to weekly. Updating the data extraction scope directly affects subsequent data processing and validation, potentially requiring adjustments to data validation templates and redefining data sharding strategies to ensure the extracted data meets the new business requirements.

[0068] Step S202: Extract data from the target database table in segments according to the data extraction range indicated by the data request, and obtain N batch sub-files, where N is a positive integer.

[0069] Sharding is a common data partitioning strategy in distributed data processing and big data environments, primarily aimed at parallel processing and optimizing data transmission efficiency. In the context of this invention, sharding refers to dividing the data in a target database table into multiple smaller subsets based on the data extraction range indicated in the data request. Each subset is called a batch sub-file, thus enabling parallel data processing on distributed computing nodes.

[0070] The process of extracting fragments generally follows these steps:

[0071] Determine the sharding strategy: Based on the data volume, data distribution characteristics, and hardware resource availability, decide how to divide the data into multiple subsets. Common sharding strategies include sharding by time, sharding by field value (such as by user ID), or sharding by data block size;

[0072] Data extraction: Following the determined sharding strategy, extract the corresponding data subsets from the target database tables. Each data subset should contain a complete data structure for subsequent independent data validation.

[0073] Generate batch subfiles: Convert the extracted data subsets into file formats suitable for transmission and processing, such as CSV and JSON, forming batch subfiles. The names of these batch subfiles typically include a unique identifier to distinguish different data subsets.

[0074] In this embodiment, batch sub-files typically represent a series of small data files, each containing a portion of the data requested in the data request. To facilitate management and tracking, these files are assigned unique naming rules, including information such as timestamps, data types, and fragment numbers. The design of batch sub-files aims to ensure data integrity and consistency while improving overall data processing efficiency through parallel processing.

[0075] Optionally, in the anomaly detection method for data files provided in this embodiment of the invention, the step of segmenting and extracting the target database table according to the data extraction range indicated by the data request to obtain N batch sub-files includes: determining N data extraction sub-ranges and extraction order based on the data extraction range; extracting data for each data extraction sub-range according to the extraction order; after each extraction, creating corresponding sub-data files based on the extracted data, and naming the sub-data files according to the extraction order to obtain sub-data file names containing file numbers, wherein the file numbers are generated based on the extraction order or the identifier of the data extraction sub-range; and integrating all sub-data files according to the file numbers to obtain N batch sub-files.

[0076] It should be noted that determining the data extraction sub-range and extraction order within the data extraction scope indicated by the data request directly affects the efficiency of subsequent sharding processing and data verification. The specific steps are as follows:

[0077] First, estimate the total amount of data within the data extraction range. Then, based on available computing resources (such as CPU, memory, disk I / O capabilities), and network bandwidth, plan a reasonable sub-range size for data extraction. The size of the sub-range should fully utilize computing resources without causing resource overload.

[0078] Based on the data's distribution characteristics, a data sharding strategy is designed. For time-series data, sharding can be done by date or time interval (hours, days, etc.); for data based on a category (such as region or user ID), sharding can be done according to the distribution of category values. This step determines how to divide the original data into N sub-ranges.

[0079] The extraction order is defined to ensure data continuity and verification accuracy. Typically, it is sorted chronologically, by category, or by other pre-defined logic. In a distributed processing environment, ordered extraction simplifies data integration and verification, ensuring the correctness of the results.

[0080] Once the data extraction sub-ranges and extraction order are determined, the next step is to extract data from the target database based on these sub-ranges and create sub-data files. The process is as follows: For each determined data extraction sub-range, a data query operation is performed, loading the extracted data into memory; the data in memory is then written to a separate file on disk, thus creating the sub-data file. When naming the file, a file number is typically included in the filename. This number is generated based on the extraction order or the identifier of the data extraction sub-range, for example, "File_010," where "010" represents the 10th data extraction or the 10th sub-range. This naming method clearly distinguishes files from different batches or sub-ranges, facilitating subsequent data management, verification, and integration.

[0081] After all data sub-ranges have been extracted and converted into sub-data files, the next integration step is to combine these independent files into the final N batch sub-files. The specific steps are as follows: Sort the sub-data files according to their file numbers to ensure data continuity; merge the data from the sorted sub-data files into a single file with the same structure, forming batch sub-files. During the merging process, ensure the correct order of data rows to prevent data disorder; after merging, perform a preliminary quality check on each batch sub-file to confirm data integrity and check for obvious formatting errors, such as missing fields; if abnormal data is found during the preliminary check, it should be marked in the batch sub-files, or the sub-data files containing abnormal data should be isolated for subsequent detailed verification and correction.

[0082] Step S203: For each batch sub-file, perform data verification on the batch sub-file according to the data verification template to obtain the verification sub-result.

[0083] The core of the above steps lies in performing data validation on each of the N batch sub-files obtained through fragmentation, ensuring that the data in each sub-file meets the predetermined data quality standards. The data validation template serves as the validation standard, containing all validation rules related to data quality, such as data format, field length, numerical range, and special character detection.

[0084] The purpose of performing data validation sequentially is to ensure data integrity, accuracy, and consistency. Since batch sub-files are extracted in chunks according to the data extraction range specified in the data request, each file may contain different data anomalies or errors. By validating each file individually, any potential problems can be identified and located promptly at the source. This prevents erroneous data from being passed to downstream systems, reducing the impact of abnormal data on business operations and regulatory reports. Furthermore, individual validation also facilitates subsequent error handling, as it allows for precise identification of which data items in which sub-file are abnormal, facilitating rapid repair and retries.

[0085] Optionally, in the anomaly detection method for data files provided in this embodiment of the invention, the step of performing data verification on the batch sub-files according to the data verification template to obtain a verification sub-result for each batch sub-file includes: reading the data verification template to obtain the rule categories of the data verification rules, wherein the rule categories include at least: data format verification rules, special character verification rules, data content verification rules, and data integrity verification rules; for each rule category, extracting all data verification rules corresponding to the rule category, verifying the data in the batch sub-files, and generating verification records during the verification process; if the verification records indicate that the batch sub-files conform to the data verification rules of all rule categories, setting the verification sub-result as verification passed, otherwise setting it as verification error.

[0086] The rule categories of data validation rules are classifications within the data validation template, used to indicate which aspects of the data should be checked. In this invention, the rule categories include at least the following four categories, but can be further expanded according to specific business needs and data characteristics.

[0087] The data format validation rules primarily check whether the data format meets expectations, such as whether the date format is YYYYMMDD, whether the phone number has a specific combination of digits, and whether the email address conforms to the standard email address format. Special character validation rules detect whether the data contains special characters that may affect data processing, such as delimiters and control characters, ensuring data readability and consistency. Data content validation rules check whether the specific content of the data is reasonable; for example, whether the amount field is positive, whether the account number follows the prescribed character sequence, and whether the name contains illegal characters. Data integrity validation rules ensure that no key fields are missing from the data file, and that all necessary data entries are fully recorded without any missing data.

[0088] For each batch of sub-files, the validation steps based on the data validation template are as follows: First, read the data validation template and extract the data validation rules under all rule categories. This step ensures that all necessary validation logic is applied during the data validation process. Then, apply the extracted rules to each row of data in the batch of sub-files, checking the data format, special characters, data content, and data integrity one by one. If data does not conform to a rule, record the anomaly, including the specific location of the data and details of the non-compliance. During the validation process, the system generates detailed validation records, recording whether all data items passed validation and the specific information of data items that failed validation. This provides a basis for subsequent error location and correction.

[0089] If the verification record indicates that a batch of sub-files conforms to all data verification rules of all rule categories, the verification result is set to "Verification Passed," indicating that the data quality in the batch of sub-files has reached the expected high standard and can be safely transmitted to downstream systems. Conversely, if the record indicates that any rule verification failed, the verification result is set to "Verification Error," indicating that there is data anomaly in the file, requiring manual review or automatic repair to avoid transmitting erroneous data downstream, causing business interruption or data inconsistency.

[0090] In step S204, if any verification result indicates a verification error, an error file is generated, an exception prompt is sent to the monitoring system, and the system waits for the monitoring system to send back a data correction instruction.

[0091] It should be noted that an error file is a document that records anomalies discovered during data validation. It includes a description of the specific error, the location information of the erroneous data (such as filename, line number, and field name), possible causes of the error, and suggested corrective measures. In this invention, the error file is generated after the data validation template is applied to batch sub-files and non-compliant data is detected. The purpose of generating the error file is to provide detailed error information for subsequent error analysis and data correction.

[0092] Anomaly alerts are warning messages that appear in monitoring systems to indicate any non-compliance found during data verification. Anomaly alerts typically include a brief description and the location of the problematic file, and sometimes the severity level of the error. Through anomaly alerts, monitoring systems can quickly identify which batches of sub-files have data problems, thus promptly notifying relevant personnel to handle the issue.

[0093] A monitoring system is a software platform or service that continuously monitors batch data processing workflows. It is responsible for collecting various status information and anomaly reports during data processing. By analyzing this information, the monitoring system can determine the health status of data processing, promptly detect and respond to data anomalies, such as triggering manual or automated data correction processes by sending anomaly alerts.

[0094] Data correction instructions are guiding messages issued by the monitoring system to the data processing system or manual operators after receiving an anomaly alert and analyzing the error file. The instructions include specific corrective measures for the abnormal data, such as manual correction, rerunning the data extraction process, or applying predefined data correction rules. Sending data correction instructions aims to quickly resolve problems discovered during the data verification phase, ensuring data accuracy and consistency. Optionally, in the anomaly detection method for data files provided in this embodiment of the invention, the step of generating an error file when any verification sub-result indicates a verification error includes: extracting verification records; reading the verification records to obtain error information, wherein the error information includes: error code, sub-data file name, and error data location information; generating an error file based on the error information and uploading it to the monitoring system.

[0095] It should be further explained that the verification log is an important document in the data verification process. It records the verification results of each data item in the batch sub-files, including whether the data passed the verification, the verification rules, and details of any possible verification anomalies. The verification log provides detailed data evidence for subsequent error analysis and data correction, ensuring the traceability and transparency of the data verification process.

[0096] Error messages are detailed reports generated after the data validation template is applied for data items that failed validation. In addition to the error code, sub-data file name, and error data location information mentioned earlier, error messages may also include the following: an error type description, specifying which validation rule the data violated, such as a format error or value exceeding the range; suggested corrective measures, providing possible corrective suggestions based on the error type, such as data format conversion or value correction; the original value of the erroneous data, recording the original value of the data item that caused the error for easier understanding of the specific error; the error code, used to quickly identify the error type and is the basis for automated error handling and error classification; the sub-data file name, helping to locate the specific batch of sub-files, making it easier to find and correct problematic data; and error data location information, such as the data's row number and field name, precisely indicating the location of the erroneous data within the file for quick error location and correction.

[0097] Error reports are documents that summarize all error information, detailing which data failed validation, the type and location of the error, and possible corrective measures. Uploading error reports to the monitoring system is a crucial step in the data anomaly detection process, with the following main functions: The monitoring system can collect error reports from different validation sub-results, centrally manage and analyze errors, and improve the efficiency of error handling; After receiving error reports, the monitoring system can immediately notify relevant personnel or automatically trigger the anomaly handling process, reducing the impact of problematic data on business processes; Uploading error reports provides a complete error log, facilitating the tracking of error handling progress, while also meeting the audit and compliance requirements of the financial industry for data processing processes.

[0098] Optionally, in the data file anomaly detection method provided in this embodiment of the invention, upon receiving a data correction instruction, the method further includes: parsing the data correction instruction to obtain the data identifier, modification suggestion, and priority information of the abnormal data to be corrected; locating the batch sub-files where the abnormal data is located based on the priority information and the data identifier; after locating the batch sub-files, selecting the abnormal data based on the data identifier, and modifying the abnormal data based on the modification suggestion; and re-verifying the modified batch sub-files according to the data verification template, and updating the corresponding verification sub-results.

[0099] Data identifiers are information used to uniquely identify specific data entries in a data file, typically including sub-data file names, data row numbers, field names, etc. The purpose of data identifiers is to quickly locate the abnormal data that needs correction when a data correction instruction is received, avoiding blind searching through large files and improving the efficiency and accuracy of corrections.

[0100] Modification suggestions are corrective strategies proposed by the data anomaly detection system for detected abnormal data. These suggestions may include specific modified values, correction algorithms, or data format adjustments. The purpose of modification suggestions is to provide operators or automated correction systems with clear directions for correction, ensuring that data correction is carried out according to predetermined rules and standards, and preventing the recurrence of errors.

[0101] Priority information refers to the processing order assigned to correction instructions based on the severity of abnormal data and its business impact. In scenarios where multiple abnormal data need to be corrected, priority information guides the system or operators on which data to correct first, ensuring that the most critical data issues are resolved first, thereby reducing the negative impact on business operations.

[0102] Locating the batch sub-file containing abnormal data is typically achieved by parsing the data identifier in the data correction instruction. The specific steps are as follows: extract the data identifier information from the data correction instruction, including the name of the sub-data file and the data location; use the data identifier to search and locate the specific batch sub-file containing the abnormal data in the file system; within the located batch sub-file, precisely pinpoint the exact location of the abnormal data based on the detailed information of the data identifier (such as line numbers and field names).

[0103] Once the abnormal data is located, the next step is to select and correct the data: based on the information in the data identifier, such as row number and field name, select the abnormal data entries that need to be corrected; based on the modification suggestions, correct the selected abnormal data. The correction process may include adjusting the data format, modifying the values, and supplementing missing data.

[0104] After the correction operation is completed, the modified batch sub-files need to be re-validated using the data validation template to ensure that the correction operation is effective and that no new data problems have been introduced: Apply the data validation template to check whether the modified data meets all validation rules. If the data validation passes, update the validation sub-result to pass; if there are still validation errors, further correction or regeneration of the error file is required.

[0105] In step S205, if all verification sub-results indicate that the verification has passed, merge the N batch sub-files to obtain a batch data file, and send the batch data file to the target business system.

[0106] Once each batch sub-file passes data validation, it means all data has been verified and confirmed to be in the correct format and content, without any anomalies. At this point, the system will perform a merge operation, integrating these batch sub-files into a single complete batch data file. This step ensures data integrity, enabling downstream systems to receive and process a unified dataset, rather than multiple scattered sub-files, thus simplifying the complexity of data reception and processing.

[0107] The merge operation may include the following specific steps:

[0108] Data consistency check: Recheck the data format and structure consistency of all batch sub-files to ensure that the merged file data is complete and in a uniform format.

[0109] File merging: Merge N batch sub-files according to preset rules (such as data sorting rules) to generate a single batch data file.

[0110] Metadata update: Update the metadata of the merged batch data files, such as filename, version number, creation time, etc., to ensure the accuracy of file information.

[0111] After the batch data files are merged, they need to be sent to the target business system for further processing or storage. To ensure data security and integrity, the system will take a series of security measures for data transmission, including but not limited to using encrypted communication protocols (such as SSL / TLS), data compression, data backup, and error detection and correction mechanisms during transmission.

[0112] Through the above steps S201 to S205, a data request can be received first, and a data verification template can be determined based on the data request. The data request is used to clarify the data extraction scope and the target business system. Then, the target database table is segmented and extracted according to the data extraction scope indicated by the data request to obtain N batch sub-files, where N is a positive integer. For each batch sub-file, data verification is performed on the batch sub-file according to the data verification template to obtain a verification sub-result. If any verification sub-result indicates a verification error, an error file is generated, and an exception prompt is sent to the monitoring system. The system waits for the monitoring system to send back a data correction instruction. Finally, if all verification sub-results indicate that the verification is passed, the N batch sub-files are merged to obtain a batch data file, and the batch data file is sent to the target business system.

[0113] In this embodiment of the invention, an intelligent and dynamic data verification template customization method is adopted. Through deeply integrated distributed data processing and anomaly detection methods, the goal of comprehensive data quality control before batch file transmission is achieved. This enables the technical effect of accurately capturing and timely correcting data anomalies, thereby solving the technical problem of insufficient anomaly detection mechanism in the data transmission process in related technologies, which leads to excessively high error rates in downstream business data.

[0114] Specifically, this invention first strengthens the parsing and response of data requests at the source. By meticulously analyzing the key elements of the data request—the data extraction scope and the target business system—it intelligently generates a data verification template covering multiple dimensions of verification rules, including data format, special characters, content accuracy, and completeness, ensuring the comprehensiveness and relevance of subsequent data verification. Then, leveraging the advantages of a distributed architecture, it efficiently segments and extracts massive amounts of data from the target database tables, forming a series of batch sub-files, each of which is the object of data verification. Next, based on the previously customized data verification template, it performs fine-grained data verification on each batch sub-file, generating corresponding verification sub-results to detect and locate data anomalies as early as possible. Once a verification error is detected, this invention immediately generates a detailed error report file, sends an anomaly alert to the monitoring system, and initiates an automated data correction process. When all verification sub-results indicate no anomalies, the batch sub-files are seamlessly merged to form a high-quality batch data file, which is then securely delivered to the target business system. The embodiments of the present invention achieve comprehensive quality control before data transmission, effectively reducing the risk of downstream business systems being interrupted due to data anomalies, improving the reliability and efficiency of data processing, and providing a solid guarantee for data security and business continuity in the financial industry.

[0115] The invention will now be described in conjunction with another alternative embodiment.

[0116] Example 2

[0117] This invention also provides an anomaly detection device for data files. It should be noted that the anomaly detection device for data files in this invention includes multiple implementation units, which can be used to execute the anomaly detection method for data files provided in the first embodiment above. Each implementation unit corresponds to each implementation step in the first embodiment above.

[0118] Figure 3 This is a schematic diagram of an optional data file anomaly detection device according to an embodiment of the present invention, such as... Figure 3 As shown, the device may include: a receiving unit 31, an extraction unit 32, a verification unit 33, a generation unit 34, and a sending unit 35.

[0119] The receiving unit 31 is used to receive data requests and determine a data verification template based on the data requests. The data requests are used to at least clarify the scope of data extraction and the target business system.

[0120] Extraction unit 32 is used to extract data from the target database table in segments according to the data extraction range indicated by the data request, and obtain N batch sub-files, where N is a positive integer.

[0121] The verification unit 33 is used to perform data verification on each batch sub-file according to the data verification template to obtain the verification result.

[0122] The generation unit 34 is used to generate an error file and send an exception prompt to the monitoring system when any verification sub-result indicates a verification error, and wait for the monitoring system to send back a data correction instruction.

[0123] The sending unit 35 is used to merge N batch sub-files to obtain a batch data file when all verification sub-results indicate that the verification has passed, and then send the batch data file to the target business system.

[0124] The aforementioned anomaly detection device for data files can first receive a data request through receiving unit 31 and determine a data verification template based on the data request. The data request is used to at least clarify the data extraction scope and the target business system. Then, extraction unit 32 extracts data from the target database table in segments according to the data extraction scope indicated by the data request, resulting in N batch sub-files, where N is a positive integer. Next, verification unit 33 performs data verification on each batch sub-file according to the data verification template, obtaining verification sub-results. Then, generation unit 34 generates an error file and sends an anomaly prompt to the monitoring system if any verification sub-result indicates a verification error, and waits for the monitoring system to send back a data correction instruction. Finally, sending unit 35 merges the N batch sub-files to obtain a batch data file if all verification sub-results indicate that the verification has passed, and sends the batch data file to the target business system.

[0125] In this embodiment of the invention, an intelligent and dynamic data verification template customization method is adopted. Through deeply integrated distributed data processing and anomaly detection methods, the goal of comprehensive data quality control before batch file transmission is achieved. This enables the technical effect of accurately capturing and timely correcting data anomalies, thereby solving the technical problem of insufficient anomaly detection mechanism in the data transmission process in related technologies, which leads to excessively high error rates in downstream business data.

[0126] Specifically, this invention first strengthens the parsing and response of data requests at the source. By meticulously analyzing the key elements of the data request—the data extraction scope and the target business system—it intelligently generates a data verification template covering multiple dimensions of verification rules, including data format, special characters, content accuracy, and completeness, ensuring the comprehensiveness and relevance of subsequent data verification. Then, leveraging the advantages of a distributed architecture, it efficiently segments and extracts massive amounts of data from the target database tables, forming a series of batch sub-files, each of which is the object of data verification. Next, based on the previously customized data verification template, it performs fine-grained data verification on each batch sub-file, generating corresponding verification sub-results to detect and locate data anomalies as early as possible. Once a verification error is detected, this invention immediately generates a detailed error report file, sends an anomaly alert to the monitoring system, and initiates an automated data correction process. When all verification sub-results indicate no anomalies, the batch sub-files are seamlessly merged to form a high-quality batch data file, which is then securely delivered to the target business system. The embodiments of the present invention achieve comprehensive quality control before data transmission, effectively reducing the risk of downstream business systems being interrupted due to data anomalies, improving the reliability and efficiency of data processing, and providing a solid guarantee for data security and business continuity in the financial industry.

[0127] Furthermore, the receiving unit includes: a first determining module, used to determine the requesting department and data format requirements based on the sending source and request content of the data request, wherein the data format requirements include: field information, file format, and file name; a retrieving module, used to retrieve M existing verification rules based on the requesting department and the target business system, wherein M is a positive integer; a generating module, used to generate R incremental verification rules based on the data format requirements, wherein R is a positive integer; and a first integrating module, used to integrate the existing verification rules and incremental verification rules to obtain a data verification template.

[0128] Furthermore, the anomaly detection device for data files also includes: a first parsing module, used to parse the update request to obtain update information when receiving an update request for a data request; an adjustment module, used to adjust the data verification template based on the update information to obtain the updated data request; and a first extraction module, used to re-shard and extract the target database table based on the updated data range when the update information includes an updated data extraction range.

[0129] Furthermore, the extraction unit includes: a second determining module, used to determine N data extraction sub-ranges and extraction order based on the data extraction range; a second extraction module, used to extract data from each data extraction sub-range according to the extraction order; a naming module, used to create corresponding sub-data files based on the extracted data after each extraction, and to name the sub-data files according to the extraction order to obtain sub-data file names containing file numbers, wherein the file numbers are generated based on the extraction order or the identifier of the data extraction sub-range; and a second integration module, used to integrate all sub-data files according to the file numbers to obtain N batch sub-files.

[0130] Furthermore, the verification unit includes: a first reading module, used to read the data verification template to obtain the rule categories of the data verification rules, wherein the rule categories include at least: data format verification rules, special character verification rules, data content verification rules, and data integrity verification rules; a verification module, used to extract all data verification rules corresponding to each rule category, verify the data in the batch sub-files, and generate verification records during the verification process; and a setting module, used to set the verification result as verified if the verification record indicates that the batch sub-files conform to the data verification rules of all rule categories, otherwise set it as verified as verified as incorrect.

[0131] Furthermore, the generation unit includes: a third extraction module for extracting verification records; a second reading module for reading verification records and obtaining error information, wherein the error information includes: error code, sub-data file name, and error data location information; and an upload module for generating an error file based on the error information and uploading it to the monitoring system.

[0132] Furthermore, the anomaly detection device for data files also includes: a second parsing module, used to parse the data correction instruction upon receiving it, and obtain the data identifier, modification suggestion, and priority information of the abnormal data to be corrected; a location module, used to locate the batch sub-files containing the abnormal data based on the priority information and the data identifier; a modification module, used to select the abnormal data based on the data identifier after locating the batch sub-files, and modify the abnormal data based on the modification suggestion; and an update module, used to re-verify the data in the modified batch sub-files according to the data verification template, and update the corresponding verification sub-results.

[0133] It should be noted that the receiving unit 31, extraction unit 32, verification unit 33, generation unit 34, and sending unit 35 mentioned above correspond to steps S201 to S205 in Embodiment 1. The instances and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules or units can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0134] The invention will now be described in conjunction with another alternative embodiment.

[0135] Example 3

[0136] The present invention can also provide an electronic device. Figure 4 This is a structural block diagram of an electronic device that implements an anomaly detection method for executing data files according to an embodiment of the present invention, such as... Figure 4 As shown, the electronic device may include: one or more ( Figure 4 (Only one is shown) Processor 702, memory 704, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0137] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the data file anomaly detection method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned data file anomaly detection method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0138] The processor can access information and applications stored in memory via a transmission device to perform the following steps: receiving a data request and determining a data verification template based on the data request, wherein the data request at least specifies the data extraction scope and the target business system; extracting data from the target database table in segments according to the data extraction scope indicated by the data request, resulting in N batch sub-files, where N is a positive integer; for each batch sub-file, performing data verification on the batch sub-file according to the data verification template, obtaining a verification sub-result; if any verification sub-result indicates a verification error, generating an error file, sending an exception prompt to the monitoring system, and waiting for the monitoring system to return a data correction instruction; if all verification sub-results indicate that the verification passed, merging the N batch sub-files to obtain a batch data file, and sending the batch data file to the target business system.

[0139] The processor can also invoke information and applications stored in the memory via the transmission device to perform the following steps: determine the requesting department and data format requirements based on the source and content of the data request, wherein the data format requirements include: field information, file format, and file name; retrieve M existing verification rules based on the requesting department and the target business system, where M is a positive integer; generate R incremental verification rules based on the data format requirements, where R is a positive integer; integrate the existing verification rules and incremental verification rules to obtain a data verification template.

[0140] The processor can also invoke information and applications stored in memory via a transmission device to perform the following steps: upon receiving an update request for a data request, parse the update request to obtain update information; adjust the data verification template based on the update information to obtain an updated data request; and / or, if the update information includes an updated data extraction range, re-shard and extract the target database table based on the updated data range.

[0141] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determine N data extraction sub-ranges and extraction order based on the data extraction range; extract data for each data extraction sub-range according to the extraction order; after each extraction, create corresponding sub-data files based on the extracted data, and name the sub-data files according to the extraction order to obtain sub-data file names containing file numbers, wherein the file numbers are generated based on the extraction order or the identifier of the data extraction sub-range; integrate all sub-data files according to the file numbers to obtain N batch sub-files.

[0142] The processor can also access information and applications stored in the memory via a transmission device to perform the following steps: read the data verification template to obtain the rule categories of the data verification rules, wherein the rule categories include at least: data format verification rules, special character verification rules, data content verification rules, and data integrity verification rules; for each rule category, extract all data verification rules corresponding to the rule category, verify the data in the batch sub-files, and generate verification records during the verification process; if the verification records indicate that the batch sub-files conform to the data verification rules of all rule categories, set the verification result as verification passed; otherwise, set it as verification error.

[0143] The processor can also access information and applications stored in the memory via a transmission device to perform the following steps: extracting verification records; reading the verification records to obtain error information, wherein the error information includes: error code, sub-data file name, and error data location information; generating an error file based on the error information and uploading it to the monitoring system.

[0144] The processor can also access information and applications stored in memory via a transmission device to perform the following steps: parsing data correction instructions to obtain the data identifier, modification suggestions, and priority information of the abnormal data to be corrected; locating the batch sub-file where the abnormal data is located based on the priority information and data identifier; after locating the batch sub-file, selecting the abnormal data based on the data identifier, and modifying the abnormal data based on the modification suggestions; re-verifying the data in the modified batch sub-file according to the data verification template, and updating the corresponding verification sub-results.

[0145] This invention provides a data file anomaly detection scheme. By employing an intelligent and dynamic data verification template customization method, and through deeply integrated distributed data processing and anomaly detection techniques, it achieves comprehensive data quality control before batch file transmission. This enables accurate capture and timely correction of data anomalies, thereby solving the technical problem of insufficient anomaly detection mechanisms during data transmission in related technologies, leading to excessively high error rates in downstream business data.

[0146] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 4 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0147] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0148] The invention will now be described in conjunction with another alternative embodiment.

[0149] Example 4

[0150] This invention also provides a computer-readable storage medium. Optionally, in this invention, the computer-readable storage medium can be used to store the program code executed by the anomaly detection method for data files provided in Embodiment 1.

[0151] Optionally, in this embodiment of the invention, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0152] This invention also provides a computer program product, which, when executed on a data processing device, is suitable for performing the steps of an anomaly detection method for data files: receiving a data request and determining a data verification template based on the data request, wherein the data request is at least used to clarify the data extraction scope and the target business system; performing fragmented extraction on the target database table according to the data extraction scope indicated by the data request to obtain N batch sub-files, where N is a positive integer; for each batch sub-file, performing data verification on the batch sub-file according to the data verification template to obtain a verification sub-result; if any verification sub-result indicates a verification error, generating an error file, sending an anomaly prompt to the monitoring system, and waiting for the monitoring system to return a data correction instruction; if all verification sub-results indicate that the verification passed, merging the N batch sub-files to obtain a batch data file, and sending the batch data file to the target business system.

[0153] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0154] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0156] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0158] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0159] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for detecting anomalies in data files, characterized in that, include: Receive a data request and determine a data verification template based on the data request, wherein the data request is at least used to clarify the data extraction scope and the target business system; The target database table is segmented and extracted according to the data extraction range indicated by the data request, resulting in N batch sub-files, where N is a positive integer; For each batch sub-file, data verification is performed on the batch sub-file according to the data verification template to obtain a verification sub-result; If any of the verification sub-results indicates a verification error, an error file is generated, an exception prompt is sent to the monitoring system, and the system waits for the monitoring system to send back a data correction instruction. If all the verification results indicate that the verification is successful, merge the N batch sub-files to obtain a batch data file, and send the batch data file to the target business system.

2. The anomaly detection method according to claim 1, characterized in that, The steps for determining the data verification template based on the data request include: The requesting department and data format requirements are determined by the source and content of the data request, wherein the data format requirements include: field information, file format, and file name; Based on the requesting department and the target business system, M existing verification rules are retrieved, where M is a positive integer; Based on the data format requirements, generate R incremental verification rules, where R is a positive integer; The data verification template is obtained by integrating the existing verification rules and the incremental verification rules.

3. The anomaly detection method according to claim 1, characterized in that, After determining the data verification template based on the data request, the process also includes: Upon receiving an update request for the data request, the update request is parsed to obtain update information; The data verification template is adjusted based on the updated information to obtain an updated data request; and / or, If the update information includes an updated data extraction range, the target database table is re-sharded and extracted based on the updated data range.

4. The anomaly detection method according to claim 1, characterized in that, The steps of partitioning and extracting data from the target database table according to the data extraction range indicated by the data request to obtain N batch sub-files include: Based on the aforementioned data extraction range, N data extraction sub-ranges and extraction order are determined; Data is extracted from each of the data extraction sub-ranges according to the extraction order; After each extraction, a corresponding sub-data file is created based on the extracted data, and the sub-data file is named according to the extraction order to obtain a sub-data file name containing a file number, wherein the file number is generated based on the extraction order or the identifier of the data extraction sub-range; All sub-data files are integrated according to the file number to obtain N batch sub-files.

5. The anomaly detection method according to claim 1, characterized in that, For each of the batch sub-files, the step of performing data validation on the batch sub-files according to the data validation template to obtain the validation result includes: Read the data verification template to obtain the rule categories of the data verification rules, wherein the rule categories include at least: data format verification rules, special character verification rules, data content verification rules, and data integrity verification rules; For each rule category, extract all the data verification rules corresponding to the rule category, verify the data in the batch sub-files, and generate verification records during the verification process; If the verification record indicates that the batch of sub-files conforms to the data verification rules of all the rule categories, the verification result is set to verification passed; otherwise, it is set to verification error.

6. The anomaly detection method according to claim 5, characterized in that, In the event that any of the aforementioned checksum results indicates a checksum error, the step of generating an error file includes: Extract the verification record; Read the verification record to obtain error information, wherein the error information includes: error code, sub-data file name, and error data location information; The error report file is generated based on the error information and uploaded to the monitoring system.

7. The anomaly detection method according to claim 1, characterized in that, Upon receiving the data correction instruction, the method further includes: Parse the data correction instruction to obtain the data identifier, modification suggestion, and priority information of the abnormal data to be corrected; The batch sub-file containing the abnormal data is located based on the priority information and the data identifier; After locating the batch of sub-files, the abnormal data is selected based on the data identifier, and the abnormal data is modified based on the modification suggestions; Based on the data verification template, the modified batch sub-files are re-verified, and the corresponding verification sub-results are updated.

8. An anomaly detection device for data files, characterized in that, include: A receiving unit is configured to receive a data request and determine a data verification template based on the data request, wherein the data request is at least used to clarify the data extraction scope and the target business system; The extraction unit is used to perform shard extraction on the target database table according to the data extraction range indicated by the data request, and obtain N batch sub-files, where N is a positive integer; The verification unit is used to perform data verification on each batch sub-file according to the data verification template to obtain a verification result. The generation unit is used to generate an error file and send an exception prompt to the monitoring system when any of the verification sub-results indicates a verification error, and wait for the monitoring system to send back a data correction instruction. The sending unit is configured to merge N batch sub-files to obtain a batch data file and send the batch data file to the target business system when all the verification sub-results indicate that the verification has passed.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the anomaly detection method for data files as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the anomaly detection method for data files according to any one of claims 1 to 7.

11. A computer program product, characterized in that, The method includes computer instructions, wherein when executed by a processor, the computer instructions implement the steps of the anomaly detection method for data files according to any one of claims 1 to 7.

Images