A test data management and control method based on service data construction

Abstract

The application relates to a kind of test data management and control methods based on service data construction, comprising: establishing service basic data structure model;Establishing service basic test data link model;According to test data number generation service, generate original test data;Based on original test data, set key field identification sensitive information;Non-original test data declaration and declaration audit;Actual test data application control, actual test data is sequentially carried out sensitive information identification, original check and non-original check, and check through test data and check not through test data are determined;Based on service basic test data link model, check not through test data is carried out full-link tracing and destruction.Compared with prior art, the application can actively and accurately identify sensitive test data, and trace and process sensitive test data link to improve the persistence and availability of business test data.

Description

Technical Field

[0001] This invention relates to the field of test data management technology, and in particular to a test data control method based on business data. Background Technology

[0002] In financial business scenarios, most operations are characterized by "revolving credit, rich interaction channels, diverse customers, and frequent third-party interactions." Therefore, the software development and testing process demands a high degree of diversity and comprehensiveness in the business test data. Furthermore, basic information such as customer details, accounts, and transaction media constitutes critical business test data and directly impacts the quality of the final test delivery.

[0003] To ensure the diversity and comprehensiveness of test data, and to prevent sensitive data from entering the testing process, the current main approach is to import production data and transform it according to specified rules. However, in practice, identifying whether test data is sensitive is quite difficult. The following methods are generally used to manage test data:

[0004] (1) The system searches for elements such as card number, ID card number, and mobile phone number using a specified regular expression. After the corresponding string is detected, inspectors access the core system to conduct spot checks, examining the characteristics of the test data and the authenticity of each element's match. This inspection method has a certain degree of randomness and can only achieve inefficient spot checks with incomplete coverage.

[0005] In other words, regular expressions can only identify the characteristics of corresponding data such as card numbers, ID cards, and mobile phone numbers, but cannot identify whether the data is native data from the test environment, and there is a possibility of sensitive data being injected.

[0006] (2) Only certain non-real test data is allowed, such as ID card numbers starting with 111111 and mobile phone numbers starting with 999. Although this method replaces the identification of real and non-real data, the relevant test data is limited by its special structure and cannot pass the legality verification of customer acquisition, application submission, and pre-screening systems, making it difficult to use in end-to-end system links.

[0007] In other words, during the process of creating relevant test data, because it is necessary to include feature information of non-real data, it is impossible to perform legality verification, and therefore it cannot be used for end-to-end testing.

[0008] (3) Initializing test data by periodically deleting the database. This method has a significant impact on existing test data, especially pre-embedded scenarios. It directly renders scarce data for third-party interactions, historical data from long-term observations, and test data for recurring business processes (such as bills and credit limits) unusable. The cost of restoring the relevant test data is high, and it is impossible to return to the pre-initialization level in the short term.

[0009] In short, periodically initializing the test environment will directly lead to the stagnation of related testing requirements, and it will take considerable effort to restore it to the pre-initial state. This significantly limits the persistence and availability of test data, and related testing risks will become uncontrollable. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a test data management method based on business data construction, which can proactively and accurately identify sensitive test data and trace and process the sensitive test data link to improve the persistence and availability of business test data.

[0011] The objective of this invention can be achieved through the following technical solution: a test data management method based on business data, comprising the following steps:

[0012] S1. Establish a basic business data structure model;

[0013] S2. Establish a business-based test data link model;

[0014] S3. Generate native test data based on the test data generation service;

[0015] S4. Based on the original test data, set key fields to identify sensitive information;

[0016] S5. Submission and review of non-native test data;

[0017] S6. Actual test data application management: Sensitive information identification, native verification and non-native verification are performed on the actual test data in sequence to determine the test data that passes the verification and the test data that fails the verification.

[0018] S7. Based on the business-based test data link model, perform full-link tracing and destruction of test data that fails verification.

[0019] Furthermore, the business basic data structure model in step S1 is specifically structured around a three-layer relationship: customer layer, account layer, and card layer. The customer layer involves basic customer information, including customer name, ID number, mobile phone number, risk credit information, home address, and occupation information.

[0020] The account layer involves account information, including account type, account number, account layer limit, and billing date;

[0021] The card layer involves specific card media information for the customer and is subordinate to the account, including card product, card number, expiration date, security code (CVV), and magnetic stripe.

[0022] Furthermore, the data flow direction of the business basic test data link model in step S2 is specifically as follows: channel > application submission > approval > card verification > activation > card usage > loan approval > post-loan > big data. Among them, the channel includes online channels and offline channels, which are used to obtain basic customer data information.

[0023] The incoming documents are used for pre-screening between customer acquisition and pre-screening, as well as for routing and distribution.

[0024] The approval process is used to invoke external credit and risk control strategies, and, in conjunction with the set approval nodes and routing rules, complete the approval process.

[0025] The verification card is used to review the relevant information provided by the customer;

[0026] Big data is used to transmit relevant data tables to the big data system based on historical queries, regulatory reports, and data analysis needs.

[0027] Furthermore, step S3 specifically involves requesting and maintaining basic business test data by packaging the API, and formatting and packaging the relevant input and output parameters.

[0028] For key fields, the method of "randomly generating ID card numbers, setting fixed test identifier fields, and specifying test environment card number pools" is adopted to prevent sensitive data from entering the test environment, and key field information is recorded in the database.

[0029] Furthermore, step S4 specifically involves identifying and marking key sensitive field information in the business basic test data structure model.

[0030] Furthermore, the non-native test data in step S5 includes three categories:

[0031] Category 1: Test data service native data that needs to be modified for testing purposes and is not sensitive data;

[0032] Category 2: Test data that was not generated through the test data service due to special reasons, and which is not sensitive data;

[0033] Category 3: For special reasons, it is necessary to use the project employee information of non-external clients.

[0034] Furthermore, step S6 specifically includes the following steps:

[0035] S61. Actual test data application control: determine whether the current application field hits the set key field to identify sensitive information. If the determination is yes, proceed to step S62; otherwise, end the current process.

[0036] S62. Determine whether the current application field is the native test data provided by the data creation service. If it is, it means that the whitelist has been hit and the test data application is submitted. Otherwise, proceed to step S63.

[0037] S63. Determine whether the current application field is non-native test data and non-sensitive data. If it is within the scope of the reporting and review, it means that it has hit the whitelist and the test data application is submitted. Otherwise, end the current process and do not submit the test data application.

[0038] Furthermore, steps S62 and S63 specifically employ two modes for verification and judgment: precise recognition and fuzzy recognition. When both precise recognition and judgment fail the verification, fuzzy recognition is triggered.

[0039] Furthermore, the specific process of precise identification is as follows:

[0040] The first step is to identify the data by generating rules based on the data. If a rule is matched, the data is considered as test data.

[0041] The second step is to identify various test data combinations using a hash algorithm. When generating test data, the test data is combined into text and then a hash code is generated using a hash algorithm and stored as a data combination identifier key in the test database. During the identification process, the hash code of the target data combination is calculated and then the test database is scanned. If a match is found, it is determined to be test data.

[0042] The fuzzy recognition specifically employs a Word Embedding algorithm to construct text feature vectors for the test data, and then uses a word2vec AI model for recognition. The process of training the word2vec AI model is as follows:

[0043] The first step is to process the test dataset into sentences, then use the Jieba word segmentation tool to segment the words and remove stop words;

[0044] The second step is to use the processed test dataset as the training set for the word2vec model. The result is to map the word data in the dataset into word vectors, which contain contextual information.

[0045] The third step is to use the trained word2vec model to calculate the similarity between the target data and the expected data. The higher the similarity, the higher the probability that it is the test data.

[0046] The fourth step is to use the trained word2vec model to find and filter high-risk test data.

[0047] Furthermore, step S7 specifically targets the third type of non-native test data in step S5. After it is clear that the data has been used up, based on the business basic test data link, related test data is deleted in a cascading manner around the main data flow of "channel > application > approval > card issuance > activation > card use > loan acquisition > post-loan".

[0048] In addition, for the test data that failed the verification as determined in step S6, traceability and destruction are carried out based on the business basic test data link;

[0049] Further intelligent identification and inspection will be carried out on the existing test data.

[0050] Compared with existing technologies, this invention establishes a basic business test data structure model, which can identify important sensitive risk information in the test data during the business testing process. At the same time, it establishes a basic test data link model for credit card business, setting the data flow as "channel > application > approval > card issuance > activation > card use > loan approval > post-loan", which can reliably trace and process the relevant sensitive test data links.

[0051] When verifying test data subject to current application for control, this invention combines precise identification and fuzzy identification modes to achieve a full-process processing solution for the proactive identification, reporting, inspection, control, traceability, and destruction of sensitive test data through a model, solving problems such as inaccurate identification of sensitive test data and overly broad control methods.

[0052] Furthermore, this invention establishes a whitelist of non-sensitive test data and its combination relationship database through a business-based data generation service native test data management and reporting review mechanism. This serves as an important admission condition and control measure for determining whether sensitive data is involved during the testing process. It can greatly improve the adverse effects of the original fuzzy control mode on the persistence and availability of business test data while ensuring that sensitive data does not enter the test environment. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0054] Figure 2 This is a schematic diagram illustrating the application process of an example.

[0055] Figure 3 This is a schematic diagram of the business basic test data structure model in the embodiment;

[0056] Figure 4 This is a schematic diagram of the business basic test data link model in the embodiment. Detailed Implementation

[0057] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0058] Example

[0059] like Figure 1 As shown, a test data management method based on business data includes the following steps:

[0060] S1. Establish a basic business data structure model;

[0061] S2. Establish a business-based test data link model;

[0062] S3. Generate native test data based on the test data generation service;

[0063] S4. Based on the original test data, set key fields to identify sensitive information;

[0064] S5. Submission and review of non-native test data;

[0065] S6. Actual test data application management: Sensitive information identification, native verification and non-native verification are performed on the actual test data in sequence to determine the test data that passes the verification and the test data that fails the verification.

[0066] S7. Based on the business-based test data link model, perform full-link tracing and destruction of test data that fails verification.

[0067] This embodiment applies the above technical solution to manage and control test data for credit card business, such as... Figure 2 As shown, it mainly includes:

[0068] (1) Establish a basic test data structure model for credit card business

[0069] The basic data structure model for credit card business, such as Figure 3 As shown, the structure revolves around three layers: customer layer, account layer, and card layer. From top to bottom, the relationship is mainly one-to-many, but there are also many-to-many relationships between the account layer and the card layer.

[0070] The customer layer involves basic customer information, including "customer name, ID number, mobile phone number, credit risk information, home address, and occupation information." The customer number can be linked and interacted with the bank's ECIF system, enabling information synchronization and joint maintenance between the two systems.

[0071] The account layer involves credit card account information; an individual can link multiple accounts, including "account type, account number, account layer limit, billing date", etc.

[0072] The card layer involves specific card media information for the customer; it has a subordinate relationship with the account, generally a one-to-many or many-to-many relationship between accounts and cards. It includes "card product, card number, expiration date, security code (CVV), magnetic stripe," etc. The card dimension can be linked with third-party partner institutions to form co-branded cards and other similar forms. Cards can have extended forms such as supplementary cards and virtual cards.

[0073] Sensitive data generally refers to personal information that, if leaked, illegally provided, or misused, could endanger personal safety and property, and could easily lead to damage to personal reputation, mental and physical health, or discriminatory treatment. The model above primarily involves sensitive information fields such as "customer name, ID number, home address, mobile phone number, account number, card number, expiration date, various passwords, and magnetic stripe data," requiring the establishment of de-identification or identification, traceability, and control mechanisms.

[0074] (2) Establish a basic test data link model for credit card business

[0075] like Figure 4 As shown, a basic test data link model for credit card business is established based on the data flow throughout the entire lifecycle of the credit card business. The data flow involves the following:

[0076] Channels: These are online and offline channels for acquiring credit cards, and are the main links through which basic customer data information for related credit card businesses flows in;

[0077] Incoming documents: The pre-screening system between customer acquisition and pre-screening performs some routing and diversion functions, and calls the image system for image storage;

[0078] Card number pool: Allocating blank card numbers for various card products using the card BIN;

[0079] Approval: Credit card approval is conducted by calling external credit investigation and risk control strategies through approval nodes and routing rules;

[0080] Card approval: This is the process by which the core credit card system reviews the relevant information provided by the applicant. If the review is successful, the credit card is issued. Subsequent steps involve the entire lifecycle of the card, including activation, use, renewal, and replacement.

[0081] Loan access: Providing customers with credit card-based financial services, installment loan services, and other loan services;

[0082] Post-loan management: Primarily involves the collection and management of debts from clients under the list management system;

[0083] Big Data: Relevant data tables are entered into the big data system based on historical queries, regulatory reporting, and data analysis needs.

[0084] Based on the above data flow analysis, the main data flow for credit card business is "channel > application submission > approval > card issuance > activation > card usage > loan approval > post-loan management". The "channel" is the primary source of data. Test data can be traced and processed according to this data flow.

[0085] (3) Credit Card Business Basic Test Data Generation Service

[0086] This project encapsulates basic test data generation services for credit card business, covering categories such as "card issuance, customer information maintenance, and password maintenance." It drives related systems to apply for and maintain basic test data for credit card business by wrapping APIs, and formats and packages the relevant input and output parameters.

[0087] For key fields, sensitive data is prevented from entering the test environment by methods such as "randomly generating ID card numbers, setting fixed test identifier fields, and specifying test environment card number pools".

[0088] For the data generation service using basic test data from credit card business, the native test data is used. Key fields such as "card number, ID card number, mobile phone number, and name" are recorded in the database for easy retrieval when submitting subsequent applications.

[0089] (4) Set key fields to identify sensitive information

[0090] For the credit card business basic test data creation service packaged in step (3), the key sensitive information fields in the credit card business basic test data structure model are identified and marked.

[0091] (5) Submission and review of non-native test data

[0092] To facilitate the submission of native credit card business test data provided by the non-data creation service other than step (3), a non-native test data submission function is provided. Key information such as "card number, ID card number, mobile phone number, name" and other fields, as well as single relationships and their combinations, can be submitted.

[0093] The reasons for the application are as follows:

[0094] Reporting Option 1: Test data service native data, modified for testing purposes, and non-sensitive data;

[0095] Reporting Option 2: Test data that was not generated through the test data service due to special reasons, and which is not sensitive data;

[0096] Reporting Option 3: If it is necessary to use the project employee information of a non-external customer due to special reasons, delete the data and corresponding data link after use.

[0097] The relevant filings must undergo review and approval before they can be used.

[0098] (6) Management of Basic Test Data Applications for Credit Card Business

[0099] When applying for basic test data for credit card business through the corresponding data generation service, if the data involves key sensitive information fields marked in step (4), such as "card number, ID card number, mobile phone number, name", etc., the system will perform the following priority checks on such fields:

[0100] Verification 1: Is it the original credit card business test data provided by the data generation service in step (3)? If not, proceed to verification 2.

[0101] Verification 2: Whether it is non-native credit card business test data in step (5) and non-sensitive data, and within the scope of reporting and review; first, determine whether there are reporting fields such as "card number, ID card, mobile phone number, name"; second, if there is a combination relationship in the corresponding reporting relationship, then verify whether the combination relationship matches.

[0102] In the identification of sensitive test data, a combination of precise identification and fuzzy identification methods is mainly adopted.

[0103] Verification 1 and Verification 2 are based on the original test data records provided by the data creation service in step (3) and the data sources in the non-original test data declaration in step (5), and adopt the following "precise identification" mode.

[0104] The first step is to identify data based on rules generated from the data. For example, regular expressions are used to evaluate ID numbers, phone numbers, and other rule-generated data. If a rule is matched, the data is considered test data.

[0105] The second step involves identifying various combinations of test data using a hash algorithm. For example, when generating test data, the data is combined into text, and a hash code is generated using a hash algorithm and stored as a key in the test database to identify the data combination. During the identification process, only the hash code of the target data combination needs to be calculated, and then the test database is scanned. If a match is found, the data is determined to be test data. This algorithm is more efficient than the method of scanning and comparing individual fields.

[0106] If the "precise recognition" mode fails, the "fuzzy recognition" mode is triggered. That is, for data where the precise recognition method is unsuitable, such as names or addresses not generated according to rules, the Word Embedding algorithm is used to construct a text feature vector for the test data, and the word2vec AI model is trained for intelligent recognition. The implementation steps are as follows:

[0107] The first step is to process the test dataset into sentences, then use the Jieba word segmentation tool to segment the words and remove stop words.

[0108] In the second step, the processed dataset is used as the training set for the word2vec model. The result is that the word data in the dataset is mapped into word vectors, which contain context information. (For example, for test data such as names, "Pufa Test", "Xiaopu", "Zhang San", "Li Si", "Tester", etc., after passing through the word2vec model, they can be mapped into N-dimensional vectors).

[0109] In the third step, the trained word2vec model is used to calculate the similarity between the target data and the expected data. The higher the similarity, the higher the probability of being test data. For example, use the function model.similarity("Pufa Test", "Tester") to calculate the similarity between "Pufa Test" and "Tester". The closer the result is to 1, the more similar they are, and the closer to 0, the less relevant they are.

[0110] In the fourth step, the trained word2vec model is used to search for and screen high-risk test data. For example, model.most_similar("Bank Card", topn = 10), use this function to find the top 10 words in the thesaurus that are most relevant to "Bank Card".

[0111] In this embodiment, the similarity calculation adopts the cosine similarity calculation method. When the included angle between two word vectors is smaller, the similarity is higher, and vice versa.

[0112] When all the above validations fail, the system rejects the submission in the credit card business basic test data generation service, that is, such test data cannot flow into the relevant links, and such test data cannot be processed and extended in the data generation service.

[0113] (7) Trace and destroy test data and its entire link

[0114] For the reporting option 3 in point (5), after it is clearly used up. Based on the credit card business basic test data link, through the system that stores data in the data generation system, cascade delete the relevant test data around the main data flow of "Channel > Application Submission > Approval > Card Issuance > Activation > Card Usage > Loan Acquisition > Post-loan Management".

[0115] At the same time, relying on the precise identification and fuzzy identification modes in point (6), use the AI model to intelligently identify and check the existing stock test data, which together serve as an important data source for the traceability control of the entire link of test data.

[0116] Therefore, this embodiment applies the technical solution proposed in this invention, establishing a basic test data structure model for credit card business based on credit card business. This model identifies important sensitive risk information in the test data during the credit card business testing process. Simultaneously, a basic test data link model for credit card business is established, identifying the data flow as "channel > application > approval > card issuance > activation > card use > loan acquisition > post-loan," and implementing a mechanism for tracing and processing relevant sensitive test data links. This achieves a complete solution for proactively identifying, reporting, checking, controlling, tracing, and destroying sensitive test data during the credit card business testing process, relying on precise and fuzzy identification modes and the model.

Claims

1. A test data management method based on business data, characterized in that, Includes the following steps: S1. Establish a basic business data structure model; S2. Establish a business basic test data link model. The specific data flow of the business basic test data link model is as follows: Channel > Application > Approval > Card Issuance > Activation > Card Use > Loan Obtaining > Post-Loan > Big Data. Among them, the channels include online channels and offline channels, which are used to obtain basic customer data information. The incoming documents are used for pre-screening between customer acquisition and pre-screening, as well as for routing and distribution. The approval process is used to invoke external credit and risk control strategies, and, in conjunction with the set approval nodes and routing rules, complete the approval process. The verification card is used to review the relevant information provided by the customer; Big data is used to transmit relevant data tables to the big data system based on historical queries, regulatory reports, and data analysis needs. S3. Generate native test data based on the test data generation service; S4. Based on the original test data, set key fields to identify sensitive information; S5. Submission and review of non-native test data; S6. Actual test data application management: Sensitive information identification, native verification and non-native verification are performed on the actual test data in sequence to determine the test data that passes the verification and the test data that fails the verification. S7. Based on the business-based test data link model, perform full-link tracing and destruction of test data that fails verification.

2. The test data management method based on business data construction according to claim 1, characterized in that, The business basic data structure model in step S1 is mainly based on the relationship between three layers: customer layer, account layer, and card layer. The customer layer involves basic information of the customer dimension, including customer name, ID number, mobile phone number, risk credit, home address, and occupation information. The account layer involves account information, including account type, account number, account layer limit, and billing date; The card layer involves specific card media information for the customer and is subordinate to the account, including card product, card number, expiration date, security code, and magnetic stripe.

3. The test data management method based on business data construction according to claim 1, characterized in that, Specifically, step S3 involves requesting and maintaining basic business test data by packaging the API, and formatting and packaging the relevant input and output parameters. For key fields, the method of "randomly generating ID card numbers, setting fixed test identifier fields, and specifying test environment card number pools" is adopted to prevent sensitive data from entering the test environment, and key field information is recorded in the database.

4. The test data management method based on business data construction according to claim 2, characterized in that, Step S4 specifically involves identifying and marking key sensitive field information in the business basic test data structure model.

5. The test data management method based on business data construction according to claim 1, characterized in that, The non-native test data in step S5 includes three categories: Category 1: Test data service native data that needs to be modified for testing purposes and is not sensitive data; Category 2: Test data that was not generated through the test data service due to special reasons, and which is not sensitive data; Category 3: For special reasons, it is necessary to use the project employee information of non-external clients.

6. The test data management method based on business data construction according to claim 1, characterized in that, Step S6 specifically includes the following steps: S61. Actual test data application control: determine whether the current application field hits the set key field to identify sensitive information. If the determination is yes, proceed to step S62; otherwise, end the current process. S62. Determine whether the current application field is the native test data provided by the data creation service. If it is, it means that the whitelist has been hit and the test data application is submitted. Otherwise, proceed to step S63. S63. Determine whether the current application field is non-native test data and non-sensitive data. If it is within the scope of the reporting and review, it means that it has hit the whitelist and the test data application is submitted. Otherwise, end the current process and do not submit the test data application.

7. The test data management method based on business data construction according to claim 6, characterized in that, Specifically, steps S62 and S63 employ two modes for verification and judgment: precise recognition and fuzzy recognition. If the precise recognition fails the verification, fuzzy recognition is triggered.

8. The test data management method based on business data construction according to claim 7, characterized in that, The specific process of accurate identification is as follows: The first step is to identify the data by generating rules based on the data. If a rule is matched, the data is considered as test data. The second step is to identify multiple combinations of test data using a hash algorithm. When generating test data, the test data is combined into text and then a hash code is generated using a hash algorithm and stored as a data combination identifier key in the test database. During the identification process, the hash code of the target data combination is calculated, and then the test database is scanned. If a match is found, it is determined to be test data. The fuzzy recognition specifically employs a Word Embedding algorithm to construct text feature vectors for the test data, and then uses a word2vec AI model for recognition. The process of training the word2vec AI model is as follows: The first step is to process the test dataset into sentences, then use the Jieba word segmentation tool to segment the words and remove stop words; The second step is to use the processed test dataset as the training set for the word2vec model. The result is to map the word data in the dataset into word vectors, which contain contextual information. The third step is to use the trained word2vec model to calculate the similarity between the target data and the expected data. The higher the similarity, the higher the probability that it is the test data. The fourth step is to use the trained word2vec model to find and filter high-risk test data.

9. The test data management method based on business data construction according to claim 5, characterized in that, Specifically, step S7 involves deleting related test data based on the business-based test data link after the use of the third type of non-native test data in step S5, following the main data flow of "channel > application > approval > card issuance > activation > card use > loan acquisition > post-loan". In addition, for the test data that failed the verification as determined in step S6, traceability and destruction are carried out based on the business basic test data link; Further intelligent identification and inspection will be carried out on the existing test data.

Images