A method and system for end-to-end data verification based on priority sorting

By employing a priority-based end-to-end data verification method, combined with ant colony optimization and pheromone tagging, the problem of failing to detect intermediate processing steps in big data verification within the financial industry has been solved. This enables full lifecycle data verification and rapid defect discovery, thereby improving testing efficiency.

CN117009219BActive Publication Date: 2026-11-17CHINA CITIC BANK CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310796479.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-06-30
Publication Date
2026-11-17
Estimated Expiration
2043-06-30

AI Technical Summary

Technical Problem

Existing technologies cannot effectively detect problems in intermediate processing in big data verification in the financial industry, and are difficult to encapsulate into interfaces, making it impossible to quickly discover system defects. Testers also have high coding requirements.

Method used

A priority-based full-process data verification method is adopted. The test cases are prioritized using the ant colony algorithm. Combined with pheromone tags and a preset priority model, the correctness of the data throughout its entire lifecycle from generation to display is verified, and a positive feedback mechanism is formed.

Benefits of technology

It enables full lifecycle verification of data, quickly identifies defects, improves work efficiency, and facilitates the creation of front-end pages for testers, thus achieving a transformation from human control to machine control.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117009219B_ABST
    Figure CN117009219B_ABST
Patent Text Reader

Abstract

This invention relates to a method and system for end-to-end data verification based on priority sorting. It adds pheromone tags to data types and attribute values ​​and calculates priority parameters based on a preset model. A unique index is used to link the entire lifecycle of the data, simultaneously verifying the correctness of each stage from data generation to display to the user. This verifies the correctness of the data processing logic of each system, achieving end-to-end data verification. Furthermore, it incorporates an ant colony algorithm to prioritize test cases, prioritizing the verification of cases with high pheromone concentrations. The pheromone concentration of cases is continuously adjusted and trained based on test results, forming a positive feedback mechanism to ensure the freshness of case priority and facilitate rapid defect detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of big data processing and data verification technology, and in particular to a method and system for end-to-end data verification based on priority sorting. Background Technology

[0002] Financial big data involves the processing of data across multiple stages and nodes, including downstream business systems, upstream business systems, and intermediate data processing systems. During execution, data verification is crucial to ensure consistency and prevent defects in the data flow and usage process. Due to the diverse types of transaction details data, complex processing logic, and interconnected call relationships in the banking and financial industry, traditional manual verification is inefficient, difficult, and prone to errors, and is gradually being replaced by automated verification methods.

[0003] Existing technologies offer various solutions to the problem of big data verification in the financial industry. For example, consistency verification tests are performed on the target data pushed to the target application system and the target business source data to obtain consistency verification test results; alternatively, different types of data are divided into multiple test modules, and data verification scripts corresponding to each data test module are obtained from a preset configuration table.

[0004] While existing technologies can perform necessary verification of financial industry data to a certain extent, they still have shortcomings in practical use. Specifically, when performing consistency verification tests on target data and source data of the target application system, simply comparing the target and source data of upstream and downstream business systems fails to effectively identify problems in the intermediate data processing. Furthermore, the approach of dividing different types of data into multiple test modules does not effectively prioritize test cases, hindering the rapid identification of system issues. Additionally, existing verification methods are difficult to encapsulate into interfaces, making it impossible to provide testers with front-end pages and placing high demands on their coding skills. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a priority-based end-to-end data verification method and system. This method uses a unique index to link the entire lifecycle of data, simultaneously verifying the correctness of data at each stage from generation to display to the user. This verifies the correctness of the data processing logic of each system, achieving end-to-end data verification. Furthermore, it incorporates an ant colony algorithm to prioritize test cases, prioritizing the verification of cases with high "pheromone" concentrations. The "pheromone" concentration of cases is continuously adjusted and trained based on test results, forming a positive feedback mechanism to ensure the freshness of case priority and facilitate rapid defect detection.

[0006] To achieve the above objectives, the technical solution adopted by the present invention includes:

[0007] A method for end-to-end data verification based on priority sorting, characterized by comprising:

[0008] S1. Mark the data source and at least one set of target data, wherein the data source includes raw acquired data and / or verification expected benchmark data, and the target data includes non-raw data after data processing operations and / or transmission;

[0009] S2. Set the data type and attribute value corresponding to the validation operation to form a validation set;

[0010] S3. Add pheromone tags to each data type and attribute value in the verification set. The default value of the pheromone tags is 0.

[0011] S4. Using a preset priority model, assign values ​​to each pheromone label corresponding to the target data;

[0012] S5. Calculate the priority parameter of the target data based on the value of the pheromone tag, sort the target data to be executed using the priority parameter, and generate a verification sequence.

[0013] S6. Based on the verification sequence, perform verification operations on the target data in sequence to obtain the verification results.

[0014] Furthermore, the target data includes one or more first target data, one or more second target data, and one or more third target data;

[0015] The first target data includes data items after data processing operations by the downstream business system and the corresponding downstream business system interface messages;

[0016] The second target data includes data items transmitted via downstream business systems;

[0017] The third target data includes the front-end display data items of the upstream business system.

[0018] Furthermore, the method also includes:

[0019] S7. Perform a classification operation on the verification results based on the first target data, the second target data, and the third target data to obtain the corresponding first verification result, the second verification result, and the third verification result.

[0020] Further, step S4 includes:

[0021] Based on the preset priority model, determine whether the pheromone tag belongs to priority content;

[0022] When a pheromone tag is determined to be priority content, a value is assigned to the pheromone tag.

[0023] Furthermore, step S4 also includes:

[0024] Determine if the pheromone tag value is 0;

[0025] If the value of the pheromone tag is not 0, then it is further determined whether the assignment type of the pheromone tag is 0 or 1.

[0026] When the value of the pheromone tag is determined to be neither 0 nor 1, the value of the pheromone tag is normalized to convert it into a decimal between 0 and 1.

[0027] Further, step S5 includes:

[0028] Perform a summation operation on the pheromone tag values ​​corresponding to the target data to obtain the priority parameter of the target data.

[0029] This invention also relates to a priority-based end-to-end data verification system, characterized in that it includes:

[0030] The data acquisition module is used to tag the data source and the target data;

[0031] The validation collection module is used to set the data type and attribute value corresponding to the validation operation;

[0032] The pheromone management module is used to add pheromone tags to each data type and attribute value in the validation set, and to assign values ​​to the pheromone tags.

[0033] The priority calculation module is used to calculate the priority parameters of the target data based on the assigned values ​​of the pheromone tags, and then use the priority parameters to sort the target data to be executed and generate a verification sequence.

[0034] The verification execution module is used to perform verification operations on the target data sequentially according to the verification sequence and obtain the verification results.

[0035] The present invention also relates to a computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, implements the above-described method.

[0036] The present invention also relates to an electronic device, characterized in that it includes a processor and a memory;

[0037] The memory is used to store the data source and the target data;

[0038] The processor is used to execute the above method by invoking the data source and target data.

[0039] The present invention also relates to a computer program product, including a computer program and / or instructions, characterized in that the computer program and / or instructions, when executed by a processor, implement the steps of the above-described method.

[0040] The beneficial effects of this invention are as follows:

[0041] The priority-based end-to-end data verification method and system described in this invention uses a unique index to link the entire lifecycle of data, simultaneously verifying the correctness of data at each stage from generation to display to the user. This verifies the correctness of the data processing logic of each system, achieving end-to-end data verification. Furthermore, it incorporates an ant colony algorithm to prioritize test cases, prioritizing the verification of cases with high "pheromone" concentrations. The "pheromone" concentration of cases is continuously adjusted and trained based on test results, forming a positive feedback mechanism to ensure the freshness of case priority and rapid defect detection. Simultaneously, the method of this invention, by executing data verification scripts, easily creates a front-end page for testers. Testers can call the data verification script after submission to automatically verify data correctness, improving work efficiency and realizing the transformation of data verification from human-controlled to machine-controlled. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the process of the priority-based full-process data verification method of the present invention.

[0043] Figure 2 This is a schematic diagram of the structure of the priority-based end-to-end data verification system of the present invention. Detailed Implementation

[0044] To better understand the content of this invention, a detailed description will be provided in conjunction with the accompanying drawings and embodiments.

[0045] The first aspect of this invention relates to a process flow as follows: Figure 1 The priority-based end-to-end data validation method shown includes:

[0046] S1. Mark the data source and at least one set of target data, wherein the data source includes raw acquired data and / or verification expected benchmark data, and the target data includes non-raw data after data processing operations and / or transmission.

[0047] Preferably, the target data includes one or more first target data, one or more second target data, and one or more third target data. The first target data includes data items processed by downstream business systems and corresponding downstream business system interface messages; the second target data includes data items transmitted by downstream business systems; and the third target data includes front-end display data items from upstream business systems.

[0048] S2. Set the data type and attribute value corresponding to the validation operation to form a validation set.

[0049] S3. Add pheromone tags to each data type and attribute value in the verification set. The default value of the pheromone tags is 0.

[0050] S4. Using a preset priority model, assign values ​​to each pheromone label corresponding to the target data.

[0051] Specifically, based on a preset priority model, it is determined whether the pheromone tag belongs to priority content; when the pheromone tag is determined to belong to priority content, a value is assigned to the pheromone tag.

[0052] Preferably, during execution, the value of the pheromone tag is first determined to be 0; if the value of the pheromone tag is not 0, the assignment type of the pheromone tag is further determined to be 0 or 1; if the assignment type of the pheromone tag is not 0 or 1, the value of the pheromone tag is normalized to convert the value of the pheromone tag into a decimal between 0 and 1.

[0053] The priority model can be set by the testers according to actual needs, and may include any one or more combinations of the following conditions:

[0054] a1. Does the case involve single-field validation (0-1, 0 indicates single-field validation is involved; 1 indicates single-field validation is not involved)?

[0055] a2. Whether the query result is empty (1 indicates yes, 0 indicates no case where the query result is empty);

[0056] a3. Whether the query results exceed the limit (1 indicates yes, 0 indicates no, and the query results are empty);

[0057] a4. Does the case involve high-frequency or high-risk functions (0-10, 0 indicates no function, 10 indicates high-frequency or high-risk function);

[0058] a4. Does the case involve any changes in accounting records (1 indicates yes, 0 indicates no changes in accounting records);

[0059] a5. Whether the case is related to historical defects (0-10, 0 indicates no historical defects, 10 indicates many historical defects with high defect severity);

[0060] a6. Case execution cost (1-10, where 1 indicates extremely low cost and 10 indicates extremely high cost);

[0061] a7. Whether the functional modules corresponding to the case need to be modified in this case (0-10, 0 means no modification, 10 means modification and complex logic with high risk).

[0062] S5. Calculate the priority parameter of the target data based on the assigned value of the pheromone tag, sort the target data to be executed using the priority parameter, and generate a verification sequence.

[0063] After normalizing the pheromone tag values ​​that are not in the form of 0 / 1, the pheromone tag values ​​corresponding to the target data can be summed to obtain the priority parameter of the target data.

[0064] Preferably, the specific range of verification can be determined based on the calculated priority parameters. For example, target data with a priority parameter of 0 can be considered to have low importance and may not be verified under certain circumstances; while target data with a priority parameter exceeding a certain preset threshold can be used as key verification targets and additional auxiliary steps such as manual review can be added.

[0065] S6. Based on the verification sequence, perform verification operations on the target data in sequence to obtain the verification results.

[0066] Preferably, the verification process for the first target data, the second target data and the third target data that have been classified further includes step S7, which involves performing a classification operation on the verification results based on the first target data, the second target data and the third target data to obtain the corresponding first verification result, the second verification result and the third verification result.

[0067] When implementing the above method, it is preferable to integrate the relevant operation parts into a front-end operation page, where testers initiate data verification requests and the back-end returns the verification results to the front-end page for display.

[0068] For processes involving financial big data verification, the above method can be used. The data source is located as the expected result using the data type and attribute value as a unique index. The interface messages from system A, the data unloaded to system B, and the front-end data from system C are retrieved based on the data type and attribute value. These three target data sets are then compared with the data source. Inconsistencies indicate a flaw in the system's data processing logic. Finally, the comparison result is returned to the front-end for display. Specifically, this includes:

[0069] 1) Testers select the data type and attribute value on the front-end page and submit it;

[0070] 2) The big data verification backend locates the source data in the database table of downstream business system A as the expected result based on the data type and attribute value;

[0071] 3) The big data verification backend obtains the interface message of the downstream business system A as the target data 1 based on the data type and attribute value;

[0072] 4) The big data verification backend obtains data from the database table of system B of the data processing system as target data 2 based on the data type and attribute value;

[0073] 5) The big data verification backend obtains the front-end data of the upstream business system C as the target data 3 based on the data type and attribute value;

[0074] 6) Compare target data 1 with source data. If there is an inconsistency, it means that system A has a defect in processing the source data. Record the difference data 1.

[0075] 7) Compare target data 2 with source data. If there is an inconsistency, it means that there is a defect in the process of unloading the source data to system B. Record the difference data 2.

[0076] 8) Compare the target data 3 with the source data. If there is an inconsistency, it means that there is a defect in the processing of system B's data by system C. Record the difference data 3.

[0077] 9) Merge the difference data 1, 2, and 3 to obtain the verification result, and return the verification result to the front-end page to display to the testers.

[0078] Another aspect of this invention relates to a priority-based end-to-end data verification system, the structure of which is as follows: Figure 2 As shown, it includes:

[0079] The data acquisition module is used to tag the data source and the target data;

[0080] The validation collection module is used to set the data type and attribute value corresponding to the validation operation;

[0081] The pheromone management module is used to add pheromone tags to each data type and attribute value in the validation set, and to assign values ​​to the pheromone tags.

[0082] The priority calculation module is used to calculate the priority parameters of the target data based on the assigned values ​​of the pheromone tags, and then use the priority parameters to sort the target data to be executed and generate a verification sequence.

[0083] The verification execution module is used to perform verification operations on the target data sequentially according to the verification sequence and obtain the verification results.

[0084] By using this system, the aforementioned computational processing methods can be executed and the corresponding technical effects can be achieved.

[0085] Embodiments of the present invention also provide a computer-readable storage medium capable of implementing all the steps of the methods in the above embodiments, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements all the steps of the methods in the above embodiments.

[0086] Embodiments of the present invention also provide an electronic device for performing the above-described method. As an implementation device for the method, the electronic device has at least a processor and a memory. In particular, the memory stores data and related computer programs required for performing the method, such as data sources and target data. The processor calls the data and programs in the memory to execute all the steps of the method and obtain the corresponding technical effect.

[0087] Preferably, the electronic device may include a bus architecture, which may include any number of interconnected buses and bridges. The bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same element, i.e., a transceiver, providing a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor during operation.

[0088] Additionally, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operating control) used may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device; the memory may be one or more of a buffer, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices, which can store the aforementioned data information, and may also store programs for executing the information, and the processor can execute the program stored in the memory to achieve information storage or processing, etc.; the input unit is used to provide input to the processor, for example, it can be a button or touch input device; the power supply is used to provide power to the electronic device; the display is used to display images and text, for example, it can be an LCD display. The communication module is a transmitter / receiver that transmits and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, which can be the same as in conventional mobile communication terminals. Based on different communication technologies, multiple communication modules can be incorporated into the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) is also coupled to a speaker and microphone via an audio processor to provide audio output through the speaker and receive audio input from the microphone, thereby enabling typical telecommunications functions. The audio processor can include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor is coupled to a central processing unit, enabling on-device recording via the microphone and on-device playback of stored sound via the speaker.

[0089] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0093] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for end-to-end data verification based on priority sorting, characterized in that, include: S1. Mark the data source and at least one set of target data, wherein the data source includes raw acquired data and / or verification expected benchmark data, and the target data includes non-raw data after data processing operations and / or transmission; S2. Set the data type and attribute value corresponding to the validation operation to form a validation set; S3. Add pheromone tags to each data type and attribute value in the verification set. The default value of the pheromone tags is 0. S4. Using a preset priority model, assign values ​​to each pheromone label corresponding to the target data; S5. Calculate the priority parameter of the target data based on the value of the pheromone tag, sort the target data to be executed using the priority parameter, and generate a verification sequence. S6. Based on the verification sequence, perform verification operations on the target data in sequence to obtain the verification results.

2. The method as described in claim 1, characterized in that, The target data includes one or more first target data, one or more second target data, and one or more third target data; The first target data includes data items after data processing operations by the downstream business system and the corresponding downstream business system interface messages; The second target data includes data items transmitted via downstream business systems; The third target data includes the front-end display data items of the upstream business system.

3. The method as described in claim 2, characterized in that, The method further includes: S7. Perform a classification operation on the verification results based on the first target data, the second target data, and the third target data to obtain the corresponding first verification result, the second verification result, and the third verification result.

4. The method as described in claim 1, characterized in that, Step S4 includes: Based on the preset priority model, determine whether the pheromone tag belongs to priority content; When a pheromone tag is determined to be priority content, a value is assigned to the pheromone tag.

5. The method as described in claim 4, characterized in that, Step S4 further includes: Determine if the pheromone tag value is 0; If the value of the pheromone tag is not 0, then it is further determined whether the assignment type of the pheromone tag is 0 or 1. When the value of the pheromone tag is determined to be neither 0 nor 1, the value of the pheromone tag is normalized to convert it into a decimal between 0 and 1.

6. The method as described in claim 5, characterized in that, Step S5 includes: Perform a summation operation on the pheromone tag values ​​corresponding to the target data to obtain the priority parameter of the target data.

7. A priority-based end-to-end data verification system, characterized in that, include: A data acquisition module is used to mark data sources and target data. The data sources include raw acquired data and / or verification expected benchmark data. The target data includes non-raw data that has undergone data processing operations and / or transmission. The validation collection module is used to set the data type and attribute value corresponding to the validation operation; The pheromone management module is used to add pheromone tags to each data type and attribute value in the validation set, and to assign values ​​to the pheromone tags. The priority calculation module is used to calculate the priority parameters of the target data based on the assigned values ​​of the pheromone tags, and then use the priority parameters to sort the target data to be executed and generate a verification sequence. The verification execution module is used to perform verification operations on the target data sequentially according to the verification sequence and obtain the verification results.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 6.

9. An electronic device, characterized in that, Including processor and memory; The memory is used to store the data source and the target data; The processor is configured to execute the method of any one of claims 1 to 6 by invoking the data source and the target data.

10. A computer program product comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Regression test method for carrying out test case priority ranking based on ant colony algorithm

    CN110109822A

  • Database benchmark testing method and device based on transaction scene

    CN113064837A