Quality detection method and system for online service data

Abstract

The invention relates to an online service data quality detection method and system, and the method comprises the following steps: collecting and analyzing service system data, and obtaining field attribute information; verifying the field attribute information based on a rule base to obtain a verification result; judging that the data is abnormal according to the verification result, adding a traceability identifier to the abnormal data, and generating an abnormal record; and distributing the data based on the abnormal record to obtain qualified and problem queues, and processing according to a preset strategy. The technical problems that when data exception is detected, a large amount of time and energy need to be consumed to trace the generation source of the abnormal data, the problem solving period is prolonged, and the quality problem of similar data repeatedly occurs due to the fact that the problem source cannot be positioned in time are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data quality detection, in particular to an online business data quality detection method and system. BACKGROUND

[0002] With the rapid development of digital economy, online business has become the core carrier for various enterprises to carry out business activities. From the transaction data of e-commerce platforms to the user credit information of financial institutions, and to the diagnosis and treatment records of Internet medical treatment, the scale of business data is growing at an exponential rate. These original business data not only serve as an important basis for enterprises to make strategic decisions and optimize service processes, but also are the key foundation to support the stable operation of business systems.

[0003] Under the trend of data-driven business development, enterprises have increasingly high demand for data value mining, and data quality directly determines the level of data value. The existing data quality detection system often lacks a traceability mechanism for abnormal data. When abnormal data is detected, staff need to spend a lot of time and effort to trace the source of abnormal data, which not only prolongs the problem solving period, but also causes the same data quality problems to recur due to the inability to locate the root cause of the problem in time. At the same time, for the detected qualified data and problem data, most enterprises still use manual classification processing methods, which not only increases labor costs, but also easily leads to inaccurate data processing results due to human operation errors, affecting the use efficiency of subsequent business data. The existence of these problems further highlights the importance and urgency of researching and optimizing the online business data quality detection method. SUMMARY

[0004] The technical problem solved by the present application is to provide an online business data quality detection method and system, which solves the technical problem that when abnormal data is detected, a lot of time and effort is needed to trace the source of abnormal data, which not only prolongs the problem solving period, but also causes the same data quality problems to recur due to the inability to locate the root cause of the problem in time.

[0005] To solve the above technical problems, an online business data quality detection method is adopted, which comprises the following steps: The original business data of the business system is collected and accessed, and the field attribute information is obtained by field splitting and parsing the original business data; The field attribute information is checked item by item based on a preset data quality rule library to obtain a field check result; Based on the field check result, it is determined whether the original business data is abnormal. If it is, an abnormal traceability identifier is attached to the original business data with abnormality, and a business abnormality record is generated; The original business data is processed based on the business exception record to obtain a qualified data queue and a problem data queue, and the qualified data queue and the problem data queue are processed by using a preset processing strategy.

[0006] Further, the original business data is field-split parsed to obtain field attribute information, including: The original business data is field-split parsed to obtain field attribute information, including: The format-compliant field is field-mapped and converted and attribute information is extracted based on a preset field mapping relationship to obtain field attribute information.

[0007] Further, the field check result is summarized and counted to obtain a single data quality score, including: The field type of each field in the field attribute information is identified to obtain a field type identifier, and a field check rule set is obtained by performing rule matching and extraction from the data quality rule library based on the field type identifier; The field check rule set is used to perform rule-based field value extraction on the field attribute information to obtain a single field check state, and the single field check state is summarized and integrated to obtain a field check result.

[0008] Further, the original business data with an exception is attached with an exception traceability identifier to generate a business exception record, including: The original business data with an exception is extracted to obtain an exception field set, and the exception field set is associated with a business link to obtain associated business link information; The exception field set is encoded based on the associated business link information to obtain an exception traceability identifier, and the original business data with an exception is recorded and integrated based on the exception traceability identifier to generate a business exception record.

[0009] Further, the exception field set is encoded based on the associated business link information to obtain an exception traceability identifier, including: The associated business link information is analyzed to obtain a business link level sequence, and each business link in the business link level sequence is uniquely encoded to obtain a business link code; Based on the business process code, each abnormal field in the abnormal field set is associated with a code and mapped. The abnormal field is bound to the corresponding business process code to obtain the abnormal field association code. The abnormal field association code is then integrated into a format and an abnormal source identification is generated according to the preset encoding format specification.

[0010] Furthermore, the process of dividing the original business data based on business anomaly records to obtain a qualified data queue and a problematic data queue includes: Each business anomaly record is analyzed, and the anomaly source identifier in each record is extracted. Based on the correspondence between historical anomaly source identifiers and preset anomaly types, the anomaly source identifiers are parsed to obtain anomaly type information. The anomaly type information is then classified and summarized to obtain an anomaly type distribution table. Based on the anomaly type distribution table, the original business data is diverted, and the original business data that does not have anomaly type information is determined to be qualified data and included in the qualified data queue. Original business data containing abnormal information is classified as unqualified data and categorized into the problem data queue.

[0011] Furthermore, the process of using a preset processing strategy to process the qualified data queue and the problematic data queue includes: The qualified data queue is output to the downstream business system using a preset processing strategy, and the problematic data queue is output to the abnormal data processing module.

[0012] The present invention also provides a quality inspection device for online business data, comprising: The data acquisition module is used to collect raw business data from the business system and to perform field splitting and parsing on the raw business data to obtain field attribute information. The verification module is used to verify the field attribute information item by item based on a preset data quality rule base, and obtain the field verification result. The judgment module is used to determine whether there is an anomaly in the original business data based on the field verification result. If there is an anomaly, an anomaly source identification is added to the original business data with an anomaly, and a business anomaly record is generated. The traffic splitting module is used to split the original business data based on the business exception records to obtain a qualified data queue and a problem data queue, and to process the qualified data queue and the problem data queue using a preset processing strategy.

[0013] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0014] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the above methods.

[0015] The above solution involves collecting and accessing the original business data from the business system, and then splitting and parsing the original business data to obtain field attribute information. Based on a preset data quality rule base, each field attribute information is verified to obtain field verification results. These results are then summarized and statistically analyzed to obtain a single data quality score. Based on the single data quality score, it is determined whether the original business data contains anomalies. If so, an anomaly source identification is added to the anomaly-containing original business data, generating a business anomaly record. Based on the business anomaly record, the original business data is split into qualified data queues and problematic data queues. Preset processing strategies are then used to process these queues. This solution solves the technical problem that when data anomalies are detected, a significant amount of time and effort is required to trace the source of the anomaly data, which not only prolongs the problem-solving cycle but also makes it difficult to promptly locate the root cause, leading to the recurrence of similar data quality problems. Starting from the technical process characteristics of data splitting, the solution automatically divides the original data into qualified data queues and problematic data queues based on the business anomaly record, and processes them separately according to preset strategies, avoiding the inefficiency caused by mixing qualified and problematic data. Qualified data can be quickly incorporated into subsequent business processes, ensuring efficient business operations; problematic data is processed centrally according to specific strategies, which not only prevents problematic data from interfering with normal business operations, but also achieves efficient utilization of data resources through classification, thereby improving the overall efficiency of business data flow and application. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the steps of a quality detection method for online business data in one embodiment of the present invention; Figure 2 This is a structural block diagram of an online business data quality detection device according to an embodiment of the present invention; Figure 3 This is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0018] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] Specifically, the online business data quality detection method in this embodiment includes the following steps: like Figure 1 As shown, Figure 1 This invention provides a method for quality inspection of online business data, comprising the following steps: Step S1: Collect and access the original business data of the business system, and perform field splitting and parsing on the original business data to obtain field attribute information.

[0021] Specifically, when collecting raw business data from business systems, conventional methods such as API interfaces and direct database connections are typically used in practice. Once the raw business data is obtained, the next step is to perform field splitting and parsing. For example, a piece of raw transaction data from an e-commerce platform might contain information such as order number, payment amount, order time, and shipping address. This information needs to be broken down piece by piece, and each field's type, length, and value range need to be parsed to ultimately extract the field attribute information. Throughout this process, the data collection phase must ensure stable data acquisition, and during the splitting and parsing phase, care must be taken not to miss any key fields, otherwise subsequent validation will be affected.

[0022] Step S2: Based on the preset data quality rule base, the field attribute information is verified item by item to obtain the field verification results.

[0023] Specifically, in practice, the field attribute information is validated item by item based on a pre-set data quality rule base. First, it's important to understand that this rule base isn't arbitrary; it contains pre-defined validation standards, such as non-empty field rules, data format rules (e.g., user phone numbers must conform to an 11-digit format, and dates must be in the format 'YYYY-MM-DD'), and numerical range rules (e.g., product inventory quantity cannot be negative, and discount ratios must be between 0 and 1). Taking the "phone number" field attribute information of a user registration record as an example, during item-by-item validation, the rule base is first checked to see if the field is empty. If it is empty, a non-empty rule exception is triggered; if it is not empty, it is then checked to see if it is an 11-digit number. If it doesn't, it is marked as a format error. If both of these checks pass, other related rules are checked, such as whether it is a duplicate of an existing user phone number (if the rule base has uniqueness rules). After all rules for a field have been validated, a corresponding field validation result is generated, such as "Validation passed" or "Format error (not 11 digits)"—a clear conclusion. It is important to note that the attribute information of each field must be checked one by one according to the entries in the rule base. You cannot skip steps, otherwise you may miss potential problems. Finally, after all fields have been validated, the results corresponding to each field are summarized to obtain the complete field validation results.

[0024] Step S3: Based on the field verification results, determine whether there is an anomaly in the original business data. If there is, attach an anomaly source identification to the original business data with an anomaly and generate a business anomaly record.

[0025] Specifically, the determination of whether original business data is abnormal is based on field validation results. The logic here is clear—it doesn't just look at a single field, but rather aggregates the validation results of all fields. If even one field's validation result is marked as "abnormal," such as "format error" or "value out of range," it indicates that the original business data is abnormal. For example, in the original business data of an e-commerce order, if the validation result of the "payment amount" field is "negative value (abnormal)," while all other fields "pass validation," then this data will still be judged as abnormal. If an abnormality is confirmed, an anomaly traceability identifier must be added to this abnormal original business data. This identifier must contain key information, such as the specific name of the abnormal field (e.g., "payment amount"), the anomaly type ("abnormal value"), the time the validation occurred, and the corresponding rule base entry number (e.g., the rule base entry number R005 for "payment amount must be greater than 0"). All this information is essential for tracing the root cause of the problem. It is important to note that the anomaly traceability identifier must correspond precisely to the original business data and cannot be confused. By binding this information together with the original business data, a complete business anomaly record can be generated. In addition to the above content, the record will also contain a unique identifier of this original business data, such as the order number, to facilitate subsequent retrieval and processing.

[0026] Step S4: Based on the business exception records, the original business data is split into qualified data queues and problematic data queues, and the qualified data queues and problematic data queues are processed using preset processing strategies.

[0027] Specifically, in practice, the raw business data is processed by tiering based on business anomaly records. This tiering isn't arbitrary; the core criterion is whether each piece of raw business data is associated with a corresponding business anomaly record. If a piece of raw business data is associated with a business anomaly record, regardless of whether the anomaly is due to an incorrect field format or an out-of-range value, it will be placed in the problematic data queue. If it's not associated with any business anomaly record, it means all field validations have passed, and it will be placed in the qualified data queue. This tiering quickly divides the raw business data into two parts: the qualified data queue and the problematic data queue. For example, in a batch of order data from an e-commerce platform, 100 orders without associated anomaly records go directly to the qualified data queue, while the other 20 orders associated with an anomaly record due to an "empty shipping address field" go to the problematic data queue. Then, a pre-defined processing strategy is used to process these two queues separately. The strategy for the qualified data queue is to directly push it to the subsequent business analysis system for calculating monthly sales. The strategy for the problematic data queue is usually to temporarily store it in a dedicated anomaly data pool while automatically sending a notification to technical personnel to remind them to investigate the cause of the anomaly. It is important to note that the processing strategies for the two types of queues should not be confused. If problematic data is mistakenly pushed to the business analysis system, it will greatly affect the subsequent data analysis results. Therefore, it is necessary to carefully check against the preset strategy when executing the data.

[0028] In a specific embodiment, the step of performing field splitting and parsing on the original business data to obtain field attribute information includes: The original business data is initially split into fields to obtain an initial field set, and each initial field in the initial field set is format-validated to obtain format-compliant fields; Based on the preset field mapping relationship, the format-compliant fields are mapped and converted and attribute information is extracted to obtain field attribute information.

[0029] Specifically, in practice, the first step in parsing and splitting raw business data to obtain field attribute information is preliminary field splitting. This splitting is not arbitrary; it is usually done according to delimiters agreed upon in advance during data transmission. For example, some raw data uses commas to separate different fields, while others use vertical bars. By splitting according to these delimiters, the raw business data can be broken down into independent initial fields. These fields together form the initial field set. Taking a user registration record as an example, "1001 | Zhang San | 13800138000 | 2025-12-17", after splitting by vertical bars, the initial field set consists of the initial fields "1001", "Zhang San", "13800138000", and "2025-12-17".

[0030] Next, each initial field in the initial field set needs to be format-validated. Each field must be checked against the preset format standards. For example, the initial field "13800138000" needs to be checked to see if it is an 11-digit mobile phone number, and "2025-12-17" needs to be checked to see if it is a date in the format "YYYY-MM-DD". Initial fields that conform to the format standards are considered compliant fields. If a field, such as "2025 / 12 / 17", does not conform to the date format, it will be excluded and not included in subsequent processing.

[0031] After all this is done, the format-compliant fields are then mapped and their attribute information extracted based on a pre-defined field mapping relationship. This field mapping relationship is pre-set; for example, "1001" corresponds to "Field Name: User ID, Field Type: String, Field Length: 20", and "Zhang San" corresponds to "Field Name: User Name, Field Type: String, Field Length: 50". Following this mapping relationship, the attribute information (such as field name, type, and length) for each format-compliant field is extracted, and necessary mapping conversions are performed (for example, some fields may need to be converted from text to numeric). The final integrated information is the field attribute information. It is important to note that the field types must match during mapping conversion; otherwise, errors in attribute information extraction can easily occur.

[0032] In a specific embodiment, the step of summarizing and statistically analyzing the field validation results to obtain a single data quality score includes: For each field in the field attribute information, the field type is identified to obtain the field type identifier, and rule matching is performed from the data quality rule base based on the field type identifier to obtain the field verification rule set; Based on the field validation rule set, the field values ​​in the field attribute information are executed one rule at a time to obtain the single field validation status. The single field validation status is then summarized and integrated to obtain the field validation result.

[0033] Specifically, in practice, the results of field validation are summarized and statistically analyzed to obtain a single data quality score. The first step is to identify the type of each field in the field attribute information. This is not a simple glance, but rather determining whether each field belongs to a specific type such as string, numeric, or date. For example, the "order amount" field will be identified as numeric, and the "order time" field will be identified as date. After identification, each field is marked with the corresponding field type identifier. Then, based on this field type identifier, rules are extracted from the data quality rule base for matching. This step is not random matching; rules must be found according to the type. For example, for the numeric "order amount" field, rules related to numeric values ​​are matched from the rule base, such as "numerical range rule (0-100000 yuan)" and "non-negative rule." These matched rules are integrated to form the field validation rule set. For the date-type "order time" field, rules such as "date format rule (YYYY-MM-DD)" and "date validity rule (cannot be later than the current time)" are matched to form the corresponding field validation rule set.

[0034] After this step is completed, the rules for each field value in the field attribute information are executed one by one based on the field validation rule set. This must be done without skipping any steps. Taking the value "Order Amount = -50 Yuan" as an example, the "Non-negative Rule" is executed first. Since the amount is negative, this rule is deemed invalid. Next, the "Number Range Rule" is executed; -50 Yuan is not within the range of 0-100000 Yuan, so this rule is also deemed invalid. Each rule execution will result in a "Pass" or "Fail" single-field validation status. Finally, all single-field validation statuses are summarized and integrated. For example, the "Order Amount" field fails both rules, while the "Order Time" field passes both rules. Organizing the validation status of each field gives the final field validation result. It is important to note that no rule can be omitted during execution. Even if a rule seems unimportant, skipping it will affect the accuracy of subsequent validation results, thus affecting the calculation of the individual data quality score.

[0035] In a specific embodiment, the step of attaching an anomaly source identification to the original business data containing anomalies and generating a business anomaly record includes: Abnormal fields are extracted from the original business data that are abnormal. Fields that cause data abnormalities are selected from the original business data to form an abnormal field set. The related business links are located for each abnormal field in the abnormal field set to obtain the related business link information. Based on the associated business process information, the abnormal field set is encoded with a source tracing identifier to obtain an abnormal source tracing identifier. Based on the abnormal source tracing identifier, the original business data with abnormalities is recorded and integrated to generate a business abnormal record.

[0036] Specifically, when processing abnormal data, anomaly traceability markers are added to the original business data containing anomalies to generate business anomaly records. The first step is to extract the abnormal fields. This step is not done by guesswork, but by comparing the results with previous field validation. For example, in a piece of original data, if the validation result of the "Payment Amount" field is "negative value (abnormal)" and the "Shipping Address" is "format error (abnormal)," while other fields are normal, then these two fields marked as "abnormal" are extracted. These extracted fields together constitute the abnormal field set. Taking order data from an e-commerce platform as an example, if the "Product Inventory" field of an order shows "-5" (abnormal) and the "Logistics Tracking Number" shows "empty value" (abnormal), then the abnormal field set consists of the "Product Inventory" and "Logistics Tracking Number" fields.

[0037] Next, we locate the associated business process for each abnormal field. Simply put, we see which step in the business process this abnormal field corresponds to. For example, if the "product inventory" field is abnormal, it can usually be located in the "inventory update step" - which is caused by the inventory not being deducted in time when the user places an order; if the "logistics tracking number" is abnormal, it corresponds to the "logistics information synchronization step", which is likely due to a problem with the transmission of the logistics system interface. After locating it, we can obtain the information of the associated business process.

[0038] Once this information is clear, the abnormal field set is coded with a source identification identifier based on the related business process information. The coding rules are generally predetermined, such as using a combination of "business process code + abnormality type code + occurrence timestamp," like "inventory update process (code K01) + numerical abnormality (code Y02) + 202512171430." This generates the abnormal source identification identifier. Then, based on this identifier, it is integrated with the original business data containing the abnormality—for example, binding the identifier "K01-Y02-202512171430" with the order data for "product inventory = -5," and adding an abnormal description, a complete business abnormality record is generated. It is crucial to ensure the related business process is correctly identified. If the "payment amount" abnormality is incorrectly identified as the "product listing process," subsequent source tracing will be misguided. Therefore, the identification must be repeatedly verified against the specific business process.

[0039] In a specific embodiment, the step of encoding the anomaly field set based on the associated business process information to obtain an anomaly source identifier includes: The related business process information is sorted out hierarchically to obtain a business process hierarchy sequence, and each business process in the business process hierarchy sequence is uniquely coded to obtain a business process code. Based on the business process code, each abnormal field in the abnormal field set is associated with a code and mapped. The abnormal field is bound to the corresponding business process code to obtain the abnormal field association code. The abnormal field association code is then integrated into a format and an abnormal source identification is generated according to the preset encoding format specification.

[0040] Specifically, in actual coding, the abnormal field set is coded with source identification based on the associated business process information to obtain the abnormal source identification. The first step is to sort out the business process hierarchy. This step is not arbitrary; it must be done according to the logic of the business process, usually divided into a hierarchy of "system - module - specific process". For example, the business process associated with a certain abnormal data is "payment system - transaction module - amount verification process". Following this hierarchy, a business process sequence from the top level to the specific process can be obtained. Next, a unique code is generated for each business process in this sequence. The coding rules are unified in advance, generally using a combination of "2-digit system code + 2-digit module code + 2-digit process code". For example, the code for "payment system" is 01, "transaction module" is 03, and "amount verification process" is 05, so the business process code for this process is 010305; if it is "logistics system - synchronization module - order number entry process", the system code is 02, the module code is 02, and the process code is 08, so the code is 020208.

[0041] Next, these business process codes are used to perform an association code mapping for each abnormal field in the abnormal field set. Simply put, this binds the abnormal field to its corresponding business process code. For example, if the abnormal field is "payment amount" and its associated business process code is 010305, then the resulting associated code is "payment amount - 010305". If the abnormal field is "logistics tracking number" with the corresponding code 020208, the associated code is "logistics tracking number - 020208". After this step, all the associated codes for abnormal fields are then formatted and integrated, strictly following the preset coding format specifications. For example, if the specification requires "abnormal field name | business process code | timestamp", then the previous "payment amount - 010305" will be integrated into "payment amount | 010305 | 202512171540". This generates the abnormal source identification. It's important to note that format integration must not deviate from the preset specifications. Even missing a single delimiter can cause problems when parsing identifiers later. Therefore, you must check the specifications line by line during integration.

[0042] In a specific embodiment, the step of performing traffic diversion processing on the original business data based on business anomaly records to obtain a qualified data queue and a problematic data queue includes: Each business anomaly record is analyzed, and the anomaly source identifier in each record is extracted. Based on the correspondence between historical anomaly source identifiers and preset anomaly types, the anomaly source identifiers are parsed to obtain anomaly type information. The anomaly type information is then classified and summarized to obtain an anomaly type distribution table. Based on the anomaly type distribution table, the original business data is diverted, and the original business data that does not have anomaly type information is determined to be qualified data and included in the qualified data queue. Original business data containing abnormal information is classified as unqualified data and categorized into the problem data queue.

[0043] Specifically, in actual data splitting, the original business data is processed based on business anomaly records to obtain qualified data queues and problematic data queues. The first step is to analyze each business anomaly record one by one. This isn't just a casual look; the anomaly traceability identifier in each record must be extracted. For example, an identifier like "Payment Amount | 010305 | 202512171540 | Y01" is used. Then, this identifier is parsed according to a pre-defined anomaly type correspondence. For instance, if the correspondence specifies "Y01 = Numerical Anomaly, Y02 = Format Anomaly, Y03 = Null Value Anomaly," finding "Y01" in the identifiers reveals the anomaly type as "Numerical Anomaly." Afterward, the anomaly type information of all business anomaly records is statistically analyzed by category. For example, if there are 20 "Numerical Anomalies," 15 "Format Anomalies," and 8 "Null Value Anomalies," an anomaly type distribution table is created. Taking a batch of e-commerce order data as an example, if there are 100 business anomaly records, the statistics will clearly show the quantity of each type of anomaly and the data involved.

[0044] Once the distribution table is created, it's used to distribute the raw business data. The core process is checking if each piece of raw business data is associated with any anomaly type information. For example, if a piece of raw data doesn't appear in the distribution table, it means it lacks corresponding anomaly type information, so it's considered qualified data and directly placed in the qualified data queue. If a piece of data has associated "format anomaly" information in the distribution table, it's considered unqualified data and placed in the problem data queue. It's crucial to distinguish between "no associated anomaly information" and "anomaly information not counted." Therefore, during distribution, each piece of data must be checked against the distribution table to ensure accurate placement. For instance, in that batch of e-commerce order data, 800 pieces without any associated anomaly type information would all be placed in the qualified data queue; the remaining 200 pieces with associated anomaly information would be placed in the problem data queue. This distribution process clearly distinguishes between the qualified and problem data queues.

[0045] In a specific embodiment, the process of using a preset processing strategy to process the qualified data queue and the problematic data queue includes: The qualified data queue is output to the downstream business system using a preset processing strategy, and the problematic data queue is output to the abnormal data processing module.

[0046] Specifically, during actual processing, a preset processing strategy is used to process the qualified data queue and the problematic data queue. The corresponding operation method is matched based on the different attributes of the two queues. For the qualified data queue, the preset processing strategy is not simply passed over; it is usually first adapted to the needs of downstream business systems. For example, if the downstream is a sales analysis system, fields such as "order amount" and "order time" in the qualified data are formatted according to the analysis system's requirements, and then output through the inter-system API interface, allowing the downstream system to directly use this data for statistics. Taking an e-commerce platform as an example, normal order data in the qualified data queue is pushed to the inventory update system and financial reconciliation system in this way to ensure timely inventory deduction and accurate account reconciliation.

[0047] In addition, the default strategy for handling the problem data queue prioritizes the synchronization of exception information. Generally, each piece of problem data is first appended with key information from previously generated business exception records, such as "exception type" and "exception fields," before the entire batch of data is output to the exception data processing module. Care must be taken during output; it's not a matter of simply sending data randomly. It must correspond to the module's receiving format—for example, if the module requires JSON format transmission, the problem data should be converted to JSON format before transmission to avoid the module being unable to recognize it. Taking e-commerce data as an example again, order data with "negative payment amount" in the problem data queue, accompanied by a "numerical anomaly (Y01)" description, will be sent to the exception data processing module, awaiting subsequent verification and correction by technical personnel.

[0048] Please see Figure 2 , Figure 2 This is a schematic diagram of a framework of an embodiment of the online business data quality inspection device of this application. Figure 2 As shown, the online business data quality inspection device includes a collection module 1, used to collect and access the original business data of the business system, and to perform field splitting and parsing on the original business data to obtain field attribute information; a verification module 2, used to verify the field attribute information item by item based on a preset data quality rule base to obtain field verification results; a judgment module 3, used to determine whether the original business data has any abnormalities based on the field verification results, and if so, to attach an abnormality source identification mark to the original business data with abnormalities and generate a business abnormality record; and a diversion processing module 4, used to divert the original business data based on the business abnormality record to obtain a qualified data queue and a problem data queue, and to process the qualified data queue and the problem data queue using a preset processing strategy.

[0049] Reference Figure 3 This invention also provides a computer device whose internal structure can be as follows: Figure 3 As shown, the computer device includes a processor, memory, display screen, input device, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores the data corresponding to this embodiment. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the above-described method.

[0050] Those skilled in the art will understand that Figure 3The structures shown are merely block diagrams of some structures related to the present invention and do not constitute a limitation on the computer devices on which the present invention is applied.

[0051] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0052] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the present invention and embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM, etc.

[0053] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0054] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0055] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0056] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0057] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0058] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0059] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A method for quality inspection of online business data, characterized in that, Includes the following steps: The system collects and accesses raw business data from the business system, and performs field splitting and parsing on the raw business data to obtain field attribute information. The field attribute information is verified item by item based on a preset data quality rule base to obtain the field verification results; Based on the field verification results, it is determined whether there is any abnormality in the original business data. If there is, an abnormality source identification is added to the original business data with abnormality, and a business abnormality record is generated. Based on the business anomaly records, the original business data is split into qualified data queues and problematic data queues, and then processed using preset processing strategies.

2. The method for quality inspection of online business data according to claim 1, characterized in that, The process of splitting and parsing the original business data to obtain field attribute information includes: The original business data is initially split into fields to obtain an initial field set, and each initial field in the initial field set is format-validated to obtain format-compliant fields; Based on the preset field mapping relationship, the format-compliant fields are mapped and converted and attribute information is extracted to obtain field attribute information.

3. The method for quality inspection of online business data according to claim 1, characterized in that, The process of summarizing and statistically analyzing the field validation results to obtain a single data quality score includes: For each field in the field attribute information, the field type is identified to obtain the field type identifier, and rule matching is performed from the data quality rule base based on the field type identifier to obtain the field verification rule set; Based on the field validation rule set, the field values ​​in the field attribute information are executed one rule at a time to obtain the single field validation status. The single field validation status is then summarized and integrated to obtain the field validation result.

4. The method for quality inspection of online business data according to claim 1, characterized in that, The process of attaching anomaly traceability identifiers to the original business data containing anomalies and generating business anomaly records includes: Abnormal fields are extracted from the original business data that are abnormal. Fields that cause data abnormalities are selected from the original business data to form an abnormal field set. The related business links are located for each abnormal field in the abnormal field set to obtain the related business link information. Based on the associated business process information, the abnormal field set is encoded with a source tracing identifier to obtain an abnormal source tracing identifier. Based on the abnormal source tracing identifier, the original business data with abnormalities is recorded and integrated to generate a business abnormal record.

5. The method for quality inspection of online business data according to claim 4, characterized in that, The step of encoding the anomaly field set based on the associated business process information to obtain an anomaly source identifier includes: The related business process information is sorted out hierarchically to obtain a business process hierarchy sequence, and each business process in the business process hierarchy sequence is uniquely coded to obtain a business process code. Based on the business process code, each abnormal field in the abnormal field set is associated with a code and mapped. The abnormal field is bound to the corresponding business process code to obtain the abnormal field association code. The abnormal field association code is then integrated into a format and an abnormal source identification is generated according to the preset encoding format specification.

6. The method for quality inspection of online business data according to claim 1, characterized in that, The process of dividing the original business data based on business anomaly records to obtain a qualified data queue and a problematic data queue includes: Each business anomaly record is analyzed, and the anomaly source identifier in each record is extracted. Based on the correspondence between historical anomaly source identifiers and preset anomaly types, the anomaly source identifiers are parsed to obtain anomaly type information. The anomaly type information is then classified and summarized to obtain an anomaly type distribution table. Based on the anomaly type distribution table, the original business data is diverted, and the original business data that does not have anomaly type information is determined to be qualified data and included in the qualified data queue. Original business data containing abnormal information is classified as unqualified data and categorized into the problem data queue.

7. The method for quality inspection of online business data according to claim 1, characterized in that, The process of using a preset processing strategy to process the qualified data queue and the problematic data queue includes: The qualified data queue is output to the downstream business system using a preset processing strategy, and the problematic data queue is output to the abnormal data processing module.

8. A quality inspection device for online business data, characterized in that, include: The data acquisition module is used to collect raw business data from the business system and to perform field splitting and parsing on the raw business data to obtain field attribute information. The verification module is used to verify the field attribute information item by item based on a preset data quality rule base, and obtain the field verification result. The judgment module is used to determine whether there is an anomaly in the original business data based on the field verification result. If there is an anomaly, an anomaly source identification is added to the original business data with an anomaly, and a business anomaly record is generated. The traffic splitting module is used to split the original business data based on the business exception records to obtain a qualified data queue and a problem data queue, and to process the qualified data queue and the problem data queue using a preset processing strategy.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, The processor executes the steps of any one of claims 1 to 7 when executing a computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When a computer program is executed by a processor, it implements the steps of the method of any one of claims 1 to 7.

Images