Data testing methods, apparatus, equipment and computer storage media

Abstract

The embodiments of the present application provide a data testing method, device, electronic device and computer storage medium; the method comprises: for each big data business demand, determining a corresponding customer-level data model, the customer-level data model comprises at least one table in a data warehouse tool corresponding to the each big data business demand, and a query keyword for each table in the at least one table, the query keyword being used to characterize information of at least one customer identity; for the customer-level data model corresponding to each big data business demand, performing data backup according to the customer identity to obtain a backup result, the backup result comprising backup data of each customer identity in the at least one customer identity; after obtaining a data test request, processing the backup result to obtain data to be tested; and performing data testing on the data to be tested.

Description

Technical Field

[0001] The present application relates to big data testing technology for financial technology (Fintech), and is related to but not limited to a data testing method, device, electronic device and computer storage medium. Background Art

[0002] With the development of computer technology, more and more technologies are being applied in the financial field. Traditional finance is gradually transforming into financial technology. However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on technology.

[0003] Currently, data warehouse tools such as Hive can be used to capture and back up data for big data testing. However, in related technologies, the use of Hive to capture and back up data is an operation implemented for a partition, and it is impossible to capture data related to a specific customer, which makes it difficult to meet the actual needs of big data testing tasks. Summary of the invention

[0004] The embodiments of the present application provide a data testing method, device, electronic device and computer storage medium, which can realize data backup for each customer identity, so that when performing data testing, test data corresponding to each customer identity can be obtained, which can meet the actual needs of big data testing tasks.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a data testing method, the method comprising:

[0007] For each big data business requirement, determine a corresponding customer-level data model, wherein the customer-level data model includes at least one table in a data warehouse tool corresponding to each big data business requirement, and a query keyword for each table in the at least one table, wherein the query keyword is used to characterize information of at least one customer identity;

[0008] For the customer-level data model corresponding to each big data business demand, data is backed up according to customer identity to obtain a backup result, wherein the backup result includes backup data of each customer identity in the at least one customer identity;

[0009] After obtaining the data test request, the backup result is processed to obtain the data to be tested; and a data test is performed on the data to be tested.

[0010] In some embodiments of the present application, determining the corresponding customer-level data model for each big data business requirement includes:

[0011] Determine at least one big data task corresponding to each big data business requirement; analyze the at least one big data task to determine a task dependency tree for the at least one big data task, wherein the task dependency tree represents a task dependency relationship for each big data task in the at least one big data task;

[0012] Analyze each task in the task dependency tree to determine a result table in the data warehouse tool corresponding to each task in the task dependency tree; use the result table as at least one table in the data warehouse tool corresponding to each big data business requirement;

[0013] The customer-level data model corresponding to each big data business requirement is determined according to at least one table in the data warehouse tool corresponding to each big data business requirement and a query keyword of each table in the at least one table.

[0014] In some embodiments of the present application, analyzing each task in the task dependency tree to determine a result table in a data warehouse tool corresponding to each task in the task dependency tree includes:

[0015] When the task in the task dependency tree is a data extraction task, reading a table that the task in the task dependency tree depends on, and using the read table as a result table in a data warehouse tool corresponding to the task in the task dependency tree;

[0016] When the task in the task dependency tree is a processing task, determine the script storage path corresponding to the task in the task dependency tree, read the script from the script storage path, execute the read script, and obtain the result table in the data warehouse tool corresponding to the task in the task dependency tree.

[0017] In some embodiments of the present application, determining the customer-level data model corresponding to each big data business requirement according to at least one table in the data warehouse tool corresponding to each big data business requirement and a query keyword of each table in the at least one table includes:

[0018] identifying a specific field in each of the at least one table, the specific field comprising at least one of a difference field and a date field, the difference field representing: when the at least one table comprises a plurality of tables, a field in the plurality of tables having the same field name and different field values;

[0019] According to the query keywords of each table in the at least one table, the customer-level data model corresponding to each big data business demand is obtained, and the customer-level data model also includes: the specific fields in each table of the at least one table identified.

[0020] In some embodiments of the present application, identifying the difference fields in each table of the at least one table includes:

[0021] Obtain a specified field, and use the specified field in each table of the at least one table as the difference field;

[0022] The number of enumeration values ​​appearing in each other field in the at least one table is counted, and the difference fields in each table of the at least one table are determined according to the number of enumeration values ​​appearing in each other field in the at least one table; the other fields represent fields other than the specified fields.

[0023] In some embodiments of the present application, determining the difference fields in each table of the at least one table according to the number of enumeration values ​​that appear in each other field of the at least one table includes:

[0024] When the number of enumeration values ​​of the target field in the at least one table is greater than or equal to a first set threshold, the target field is determined to be a difference field; the target field represents any other field in the at least one table.

[0025] In some embodiments of the present application, the first set threshold is the product of the total number of records of each other field in the at least one table and a set ratio, and the set ratio is a positive number less than 1.

[0026] In some embodiments of the present application, the step of processing the backup result to obtain the data to be tested includes:

[0027] The specific fields in each table in the backup result are processed to obtain data to be tested.

[0028] In some embodiments of the present application, the processing of the specific field in each table in the backup result to obtain the data to be tested includes:

[0029] When the specific field includes a difference field, the difference field in each table in the backup result is replaced to obtain the data to be tested;

[0030] When the specific field includes a date field, the date field in each table in the backup result is time-series shifted to obtain the data to be tested.

[0031] In some embodiments of the present application, the performing data testing on the data to be tested includes:

[0032] A mock test is performed on at least part of the data to be tested to obtain a test result corresponding to at least part of the data.

[0033] The present application provides a data testing device, characterized in that the device comprises:

[0034] A determination module, configured to determine a corresponding customer-level data model for each big data business requirement, wherein the customer-level data model includes at least one table in a data warehouse tool corresponding to each big data business requirement, and a query keyword for each table in the at least one table, wherein the query keyword is used to characterize information of at least one customer identity;

[0035] A backup module, configured to perform data backup according to customer identities for the customer-level data model corresponding to each big data business requirement, and obtain a backup result, wherein the backup result includes backup data for each customer identity of the at least one customer identity;

[0036] The test module is used to process the backup result after obtaining the data test request to obtain the data to be tested; and perform data test on the data to be tested.

[0037] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0038] A memory for storing executable instructions;

[0039] The processor is used to implement any one of the above data testing methods when executing the executable instructions stored in the memory.

[0040] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for implementing any of the above-mentioned data testing methods when executed by a processor.

[0041] In an embodiment of the present application, for each big data business demand, a corresponding customer-level data model is determined; for the customer-level data model corresponding to each big data business demand, data is backed up according to customer identity to obtain a backup result; after obtaining a data test request, the backup result is processed to obtain data to be tested; and data testing is performed on the data to be tested.

[0042] It can be seen that in the embodiment of the present application, data backup can be implemented for each customer identity, so that when performing data testing, test data corresponding to each customer identity can be obtained, which can meet the actual needs of big data testing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flow chart of a data testing method according to an embodiment of the present application;

[0044] Figure 2 A flowchart of another data testing method according to an embodiment of the present application;

[0045] Figure 3 A schematic diagram of a task dependency tree and a task-table relationship tree in an embodiment of the present application;

[0046] Figure 4 An example diagram of a requirement-task-table relationship tree established through static text analysis in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of the implementation process of data backup and data cloning in an embodiment of the present application;

[0048] Figure 6 This is an example diagram of performing time-series shifting on a date field in an embodiment of the present application;

[0049] Figure 7 For the embodiment of this application Figure 4 A schematic diagram of mock testing for Task 3 based on the above;

[0050] Figure 8 This is a schematic diagram of an optional structure of a data testing device according to an embodiment of the present application;

[0051] Fig. 9 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] In the related art, a data warehouse tool refers to a tool used to extract, transform and load data, and can store, query and analyze large-scale data; illustratively, the data warehouse tool can be hive or other data warehouse tools.

[0053] The table in the data warehouse tool belongs to a type of data model in the data warehouse tool. For example, each table in Hive has a corresponding directory to store data.

[0054] For example, in Hive, Hive Select queries generally scan the entire table content in Hive, which consumes a lot of time to complete unnecessary work. However, sometimes you only need to scan a portion of the data you are interested in in the table, so the concept of partition is introduced when building a table in Hive. Generally, the date is used as the partition. For a T-day partition, the partition of the previous day can be called the T-1 day partition, where T is an integer greater than 1.

[0055] In the related art, for big data testing tasks, if you want to reuse data during testing or implement automated testing in regression, you either have to test based on the original hive partition data or use hive commands to move partitions, and you cannot operate on the data of a specific customer; using hive to capture and back up data is an operation implemented for one partition, and each partition is usually a slice of data for a date. Each partition represents a limited number of actual scenarios. If you want to test data for multiple scenarios, you need to run data capture on different partitions, which is very costly. In addition, the related art cannot aggregate data from different scenarios in different partitions into one target partition.

[0056] In the related technology, artificial data can also be directly inserted into each table of hive, which will increase the cost and difficulty of data preparation. For example, for the big data of loan products, it is very difficult to artificially create hive data, because in the big data of loan products, the relationship between tables is strongly coupled, and the number of tables is large. When inserting data manually, it is necessary to clearly analyze the dependency of the tables, and it is necessary to insert reasonable data into these tables, otherwise the script operation cannot be associated with the data and the script logic cannot be tested.

[0057] Big data testing mainly tests the processing logic of big data, and verifying the correctness of script logic depends on the richness of hive data. In the loan products of banks, there are many divergent dimensions of scenarios, such as loan amount, overdue days, repayment method, etc., which will lead to many scenarios, and the processing scripts are generally associated with multiple tables. During testing, it is often because of incomplete scenarios or the associated tables are not extracted. The problem of problems such as problems such as the lack of data, resulting in a very long time for testing a single case of big data. It often takes one or two days to obtain the first batch of data. However, after each use, when you want to retest the task next time, first, the previous test data is in multiple partitions and cannot be grouped together for testing, and second, the partitions may have been cleaned up, and there are many tables associated with the task, and the tables are strongly coupled. If you insert data analysis directly in hive manually, the cost is very high. Therefore, it is necessary to re-capture data and test it in the order of doing transactions -> running batches -> running big data tasks. Here, running batches means daily cutting and scheduling batch tasks of the day. How to allow testers to create data once and use it repeatedly is a technical problem that needs to be solved urgently.

[0058] In view of the technical problems existing in the related technologies, a technical solution of the embodiments of the present application is proposed.

[0059] The present application will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the embodiments provided herein are only used to explain the present application and are not intended to limit the present application. In addition, the embodiments provided below are partial embodiments for implementing the present application, rather than providing all embodiments for implementing the present application. In the absence of conflict, the technical solutions recorded in the embodiments of the present application can be implemented in any combination.

[0060] It should be noted that, in the embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or device including a series of elements includes not only the elements explicitly recorded, but also includes other elements not explicitly listed, or also includes elements inherent to the implementation of the method or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other related elements (such as steps in a method or units in a device, for example, a unit may be a part of a circuit, a part of a processor, a part of a program or software, etc.) in the method or device including the element.

[0061] For example, the data testing method provided in the embodiment of the present application includes a series of steps, but the data testing method provided in the embodiment of the present application is not limited to the recorded steps. Similarly, the data testing device provided in the embodiment of the present application includes a series of modules, but the device provided in the embodiment of the present application is not limited to including the modules explicitly recorded, and may also include modules that need to be set up to obtain relevant information or perform processing based on information.

[0062] The embodiments of the present application can be applied to terminals and / or servers, where the terminals can be thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, programmable consumer electronics, network personal computers, small computer systems, etc. The servers can be small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0063] Electronic devices such as servers may include program modules that execute computer instructions. Typically, program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks. Computer systems / servers may be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

[0064] Figure 1 A flow chart of a data testing method according to an embodiment of the present application is shown in FIG. Figure 1 As shown, the process may include:

[0065] Step 101: For each big data business requirement, determine the corresponding customer-level data model, where the customer-level data model includes at least one table in the data warehouse tool corresponding to each big data business requirement, and query keywords for each table in the at least one table, where the query keywords are used to characterize information about at least one customer identity.

[0066] Here, each big data business requirement represents a function point. In order to realize a function point, multiple big data tasks may need to be executed. For example, a big data task belongs to a shell script. The logic of the big data task is to extract data from the database (DataBase, DB) to hive or process hive data. Therefore, the big data task can be divided into data extraction (Extract-Transform-Load, ETL) tasks and processing tasks.

[0067] It is understandable that each big data business requirement can be achieved by executing multiple big data tasks, and the essence of executing big data tasks is to read multiple tables in the data warehouse tool, perform associated processing through structured query language (SQL) statements in the script, and finally store the processed results in the result table of the data warehouse tool. Therefore, for each big data business requirement, at least one table corresponding to each big data business requirement can be determined.

[0068] In actual applications, the definition of the customer search method may be different in different business scenarios. Here, according to the business scenario corresponding to the big data business needs, the query keyword representing the customer identity information can be manually configured. For example, for the big data business needs corresponding to the loan business, the query keyword may include at least one of the following: customer ID card id_no, customer number cust_id, account number acct_no, bank internal unified number ecifno. In the embodiment of the present application, the query keyword may be called a search keyword (key).

[0069] Step 102: For each customer-level data model corresponding to each big data business requirement, data is backed up according to customer identities to obtain a backup result, where the backup result includes backup data for each customer identity in at least one customer identity.

[0070] Here, since the query keyword is used to characterize the information of at least one customer identity, for each customer identity, a query can be performed in each table of at least one table according to the query keyword of at least one table in the customer-level data model to obtain the corresponding query results of at least one table, and the corresponding query results of at least one table are used as backup data corresponding to each customer identity.

[0071] It can be seen that in the embodiment of the present application, in the customer-level data model, corresponding data backup can be performed for each customer identity according to the customer identity information, so as to facilitate the subsequent data cloning (DATA CLONE) according to the backup data corresponding to each customer identity. In the embodiment of the present application, data cloning means the replication of data form (DATA FORM), and data form means data with a complex life cycle, and each stage of the data life cycle corresponds to a data form.

[0072] Step 103: After obtaining the data test request, the backup result is processed to obtain the data to be tested; and a data test is performed on the data to be tested.

[0073] In the embodiment of the present application, after obtaining the data test request, the backup result can be processed according to the identity information of the target customer to obtain the data to be tested. For example, when the data warehouse tool is hive, the hive cloning technology can be used to clone the backed up morphological data to a new target empty account; here, during the hive cloning process, the backed up morphological data can be modified.

[0074] In practical applications, steps 101 to 103 can be implemented based on a processor of an electronic device, and the processor can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that the electronic device that implements the above-mentioned processor function can also be other, and the embodiments of the present application are not limited.

[0075] It can be seen that in the embodiment of the present application, data backup can be implemented for each customer identity, so that when performing data testing, test data corresponding to each customer identity can be obtained, which can meet the actual needs of big data testing tasks.

[0076] In the related art, supervised training can be used to obtain data models. The process of supervised training is: use pre-selected samples to train (TRAINING) the data model, that is, use known data forms as samples, and automatically extract the data form model through these samples; however, in hive data, because from making transactions, to running batches, to running big data tasks into hive tables, the entire process is relatively long and it is difficult to obtain standard samples. Therefore, for hive data, it is not easy to use supervised training to obtain data models, and the implementation complexity is relatively high.

[0077] In response to the above technical problems, in some embodiments of the present application, unsupervised training (UNSUPERVISED TRAINING) can be used to obtain a customer-level data model. The difference between unsupervised training and supervised training is that there are no training samples in advance, and the data can be modeled directly.

[0078] The following is an exemplary description of the implementation method of unsupervised training.

[0079] Exemplarily, for each big data business requirement, determining the implementation method of the corresponding customer-level data model may include:

[0080] Determine at least one big data task corresponding to each big data business requirement; analyze the at least one big data task to determine a task dependency tree for the at least one big data task, wherein the task dependency tree represents a task dependency relationship between each big data task in the at least one big data task;

[0081] Analyze each task in the task dependency tree to determine a result table in the data warehouse tool corresponding to each task in the task dependency tree; use the result table as at least one table in the data warehouse tool corresponding to each big data business requirement;

[0082] The customer-level data model corresponding to each big data business demand is determined according to at least one table in the data warehouse tool corresponding to each big data business demand and a query keyword of each table in the at least one table.

[0083] In practical applications, if the table in the data warehouse tool is a table obtained by processing other tasks, then in the big data task operation mechanism, the relationship between the task and the table can be determined through the task dependency. For example, all dependent tasks of each big data task can be analyzed through the big data lineage analysis tool, that is, the task dependency tree of at least one big data task can be obtained.

[0084] It can be understood that the embodiments of the present application can establish a customer-level data model based on the correspondence between big data business needs and big data tasks, and the correspondence between big data tasks and tables in data warehouse tools. Since there is no need to prepare any training samples, unsupervised training can be used to obtain the customer-level data model, which is easy to implement.

[0085] In some embodiments of the present application, the above analysis of each task in the task dependency tree to determine the result table in the data warehouse tool corresponding to each task in the task dependency tree may include:

[0086] When the task in the task dependency tree is a data extraction task, a table that the task in the task dependency tree depends on is read, and the read table is used as a result table in a data warehouse tool corresponding to the task in the task dependency tree;

[0087] When the task in the task dependency tree is a processing task, determine the script storage path corresponding to the task in the task dependency tree, read the script from the script storage path, execute the read script, and obtain the result table in the data warehouse tool corresponding to the task in the task dependency tree.

[0088] In an embodiment of the present application, text analysis can be performed on each task in the task dependency tree in turn; when a task in the task dependency tree is a data extraction task, it means that the task belongs to a leaf node in the task dependency tree. In this case, the preset ETL extraction task configuration table can be directly read to obtain the table on which the task depends, and then the task analysis process of the task is ended. Here, the preset ETL extraction task configuration table is used to represent the dependency relationship between the preset task and the table.

[0089] When the task in the task dependency tree is a processing task, the script storage path corresponding to the task can be read through the corresponding script configuration information, thereby reading the script from the script storage path and executing the read script. Here, the script configuration information can represent the corresponding relationship between the preset task and the script storage path.

[0090] It can be seen that the embodiment of the present application can process the task accordingly according to the type of task in the task dependency tree, thereby obtaining a result table in the data warehouse tool corresponding to the task. The embodiment of the present application can obtain the result table in the data warehouse tool for a variety of different types of tasks, and has a wide range of applications.

[0091] In some embodiments of the present application, determining the customer-level data model corresponding to each big data business requirement according to at least one table in the data warehouse tool corresponding to each big data business requirement and the query keyword of each table in the at least one table may include:

[0092] identifying a specific field in each table of at least one table, the specific field comprising at least one of a difference field and a date field, the difference field representing: when the at least one table comprises a plurality of tables, a field in the plurality of tables having the same field name and different field values;

[0093] The customer-level data model corresponding to each big data business requirement is obtained, and the customer-level data model also includes: a specific field in each table of the at least one identified table.

[0094] In the embodiment of the present application, after backing up data according to customer identity, if the backed-up data is to be applied to a new customer identity, it is necessary to identify the fields in the table that do not need to be replaced and the fields that need to be replaced. Here, the fields in the table that need to be replaced are difference fields, and the fields that do not need to be replaced can be recorded as the same fields. In actual application scenarios, for some fields in the table, even if the business scenarios are the same, because different customer accounts are used, the values ​​of the fields are different, and these fields are difference fields.

[0095] In the related art, when using supervised training to obtain a data model, it is necessary to construct two samples of exactly the same shape, capture the fields of the same name in all tables of the two samples, and then compare the table field values. If the values ​​are different, it indicates that the field is a field that needs to be replaced when the backed-up data is applied to the new customer identity, that is, the field is a difference field; however, in actual scenarios, it is very difficult to construct two samples of the same shape; to address this problem, in the embodiments of the present application, unsupervised training can be used to obtain a customer-level data model. In the process of using unsupervised training to obtain a customer-level data model, statistical analysis can be performed on the fields in the table, so as to determine the difference fields in the table based on the statistical analysis results.

[0096] Exemplarily, a field that can be formatted as a date can be identified, and the identified field that can be formatted as a date is recorded as a date field, so that date fields in different formats can be identified.

[0097] It can be understood that the embodiment of the present application can identify the difference fields and date fields in each table of at least one table, so that when the backup results are subsequently processed, the difference fields and date fields in the table can be adaptively modified in a more targeted manner.

[0098] It can be understood that, according to the definition of the difference field, the generation source of the difference field in the table is often a random number, an auto-increment column, or a customer inherent attribute such as the above-mentioned query keyword, etc. The characteristic of these generation sources is that the enumeration value increases continuously as the number of customers increases or the number of records increases. Therefore, in one implementation, identifying the difference field in each table of the at least one table may include:

[0099] Obtain a specified field, and use the specified field in each table of at least one table as the difference field; count the number of enumeration values ​​that appear in each other field of at least one table, and determine the difference field in each table of at least one table according to the number of enumeration values ​​that appear in each other field of at least one table; other fields represent fields other than the specified field.

[0100] In actual applications, designated fields can be set manually, and the manually set designated fields are the default difference fields. In this way, a part of the difference fields can be first identified according to the names of the designated fields. After the designated fields in each table of at least one table are used as difference fields, other fields can also be analyzed to determine the difference fields in other fields.

[0101] Exemplarily, for the big data business of loan products, the query keyword field and the business date (BUSINESS DATE) field can be set as the default difference fields; here, for the big data business of loan products, the field in the year-month-day format represents the business date field; a unified date field will be set in the big data business system involved in the loan products, and the value of this date field is automatically scrolled according to the current time. This date field is the business date field, and all transaction dates involving loan products are based on the value of this date field.

[0102] For example, for the big data business of loan products, fields with decimals represent amount-related fields. If such fields have the same form, their values ​​will be the same in two scenarios. Therefore, based on this feature, the amount-related fields are determined by default to be the same fields.

[0103] It can be seen that the embodiment of the present application can give priority to the designated field as the difference field, and then determine other difference fields based on the analysis of the values ​​of other fields, so as to accurately and comprehensively determine the difference fields in the table.

[0104] In some embodiments of the present application, determining the difference fields in each table of at least one table according to the number of enumeration values ​​that appear in each other field in at least one table may include:

[0105] When the number of enumeration values ​​of the target field appearing in at least one table is greater than or equal to a first set threshold, the target field is determined to be a difference field; the target field represents any other field in the at least one table.

[0106] Exemplarily, the first set threshold may be a fixed value, for example, the first set threshold may be 200, 300 or 400; exemplary, the first set threshold may also be the product of the total number of records in each other field in at least one table and a first set ratio, wherein the first set ratio is a positive number less than 1, for example, the set ratio may be 15%, 20% or 25%.

[0107] In an embodiment of the present application, after determining the number of enumeration values ​​of the target field in at least one table, the relationship between the number of enumeration values ​​of the target field in at least one table and the first set threshold value can be determined; if the number of enumeration values ​​of the target field in at least one table is less than the first set threshold value, it means that the target field is the same field or an unknown field, and the unknown field means that the field type cannot be determined at present; after a period of data accumulation, the type of the unknown field can be determined, that is, it can be determined whether the unknown field is the same field or a difference field. If the number of enumeration values ​​of the target field in at least one table is greater than or equal to the first set threshold value, it means that the target field is a difference field.

[0108] Exemplarily, when the total number of records in each field in at least one table is greater than or equal to the second set threshold, the relationship between the number of enumeration values ​​appearing in the target field in at least one table and the first set threshold can be determined; if the total number of records in each field in at least one table is less than the second set threshold, the target field can be considered to be an unknown field.

[0109] Here, the second set threshold value may be set based on experience, for example, the second set threshold value is 800, 1000 or 1200.

[0110] For example, the type of the target field in at least one table can be determined according to the following steps A1 to A4:

[0111] Step A1: performing deduplication processing on records in at least one table, and performing statistical analysis on the number of enumeration values ​​appearing in each field of the word in the deduplication-processed table to obtain statistical analysis results.

[0112] Step A2: If the total number of records of each field in at least one table is greater than 1000, and the number of enumeration values ​​of the target field in at least one table meets the first set condition, it is determined that the target field is the same field.

[0113] Here, the first setting condition is: the number of enumeration values ​​appearing in the target field in at least one table is less than 5% of the total number of records of each field in at least one table, and the number of enumeration values ​​appearing in the target field in at least one table is less than 300.

[0114] Step A3: If the total number of records of each field in at least one table is greater than 1000, and the number of enumeration values ​​appearing in the target field in at least one table meets the second set condition, the target field is determined to be a difference field.

[0115] Here, the second setting condition is: the number of enumeration values ​​appearing in the target field in at least one table is greater than or equal to 20% of the total number of records of each field in at least one table, or the number of enumeration values ​​appearing in the target field in at least one table is greater than or equal to 300.

[0116] Step A4: If the total number of records in each field in at least one table is less than or equal to 1000, or the number of enumeration values ​​appearing in the target field in at least one table does not satisfy both the first set condition and the second set condition, the target field is determined to be an unknown field. At this time, data accumulation can be performed until the total number of records in each field in at least one table is greater than 1000. The type of the target field can be determined by re-executing steps A2 to A4.

[0117] In an embodiment of the present application, if the total number of records in each field in at least one table is less than or equal to 1000, it means that the total number of records in each field in at least one table is small. At this time, the credibility of determining the type of the target field is low. Therefore, the target field can be first determined as an unknown field, and the type of the unknown field can be accurately determined again in the subsequent process.

[0118] When the total number of records in each field of at least one table is greater than 1000, let A represent the number of enumeration values ​​of the target field in at least one table, and let B represent the ratio of the number of enumeration values ​​of the target field in at least one table to the total number of records in each field of at least one table. When B∈[0,5%] and A<300, the target field is determined to be the same field. When B∈(5%,20%), the target field is determined to be an unknown field. When B∈[20%,1] or A≥300, the target field is determined to be a difference field.

[0119] In the embodiment of the present application, the execution order of step A2 to step A4 is not limited, and step A2 to step A4 can be executed simultaneously.

[0120] Exemplarily, the first set threshold, the second set threshold, the first set ratio, etc. may be determined according to the characteristics of the application product of the big data service.

[0121] It can be seen that in the embodiment of the present application, the type of the target field in the table can be determined, and as the data accumulates, the number of unknown fields in each table will become fewer and fewer.

[0122] For example, for unknown fields in the table, when the backup results are subsequently processed, the values ​​of the unknown fields may be kept unchanged in the same manner as the same fields, so that the processing of the backup results will not be affected by the unknown fields.

[0123] It can be seen that the embodiment of the present application can accurately determine the unknown fields in at least one table based on the relationship between the number of enumeration values ​​of the target field in at least one table and the first set threshold, thereby improving the accuracy and reliability of subsequent processing of the backup results.

[0124] In some embodiments of the present application, the implementation method of processing the backup result to obtain the data to be tested may include: processing a specific field in each table in the backup result to obtain the data to be tested.

[0125] Exemplarily, when the specific field includes a difference field, the difference field in each table in the backup result is replaced to obtain the data to be tested; when the specific field includes a date field, the date field in each table in the backup result is time-shifted to obtain the data to be tested.

[0126] Here, the time shift process refers to the process of moving the date from the source date to the target date according to a reference standard. For example, in an embodiment of the present application, for the big data business of the loan product, the first billing date can be used as a reference because the loan product requires a fixed monthly billing date.

[0127] It can be seen that by making corresponding modifications to the difference fields and date fields in each table in the backup results, it is helpful to obtain the test data that meets the actual needs.

[0128] In some embodiments of the present application, the implementation method of performing data testing on the test data may include: performing a mock test on at least part of the test data to obtain a test result corresponding to at least part of the data.

[0129] In actual applications, if a task in the task dependency tree cannot be actually tested, you can set up a mock test for the task, that is, directly use the result table in the backup result corresponding to the task, without running the task to implement the big data test related to the task, but continue to execute big data tests related to other tasks.

[0130] It can be seen that the embodiment of the present application is conducive to achieving comprehensive testing of the data to be tested and improving the accuracy of the test results by performing mock testing on at least part of the data to be tested.

[0131] The data testing method of the embodiment of the present application is further exemplified below in conjunction with the accompanying drawings.

[0132] Reference Figure 2 , the data testing method of the embodiment of the present application may include:

[0133] Step 201: Data modeling.

[0134] Here, step 201 may be implemented by determining a corresponding customer-level data model for each big data business requirement. For example, a requirement-task-table relationship tree may be established through static text analysis, and query keywords for each hive table may be obtained, and finally, the difference fields and date fields of each hive table may be extracted through statistical analysis methods.

[0135] Step 202: Data backup.

[0136] Here, for the customer-level data model corresponding to each big data business demand, data backup can be performed according to customer identity to obtain backup results; illustratively, the backup results can be stored in a data warehouse.

[0137] Step 203: Data cloning.

[0138] Here, step 203 may be implemented by: after obtaining a data test request, processing the backup result to obtain data to be tested; and performing a data test on the data to be tested.

[0139] It should be noted that the data cloning process in the embodiment of the present application does not directly use and test the backup results, but performs data testing after adaptively modifying the difference fields and date fields in the backup results.

[0140] The embodiment of the present application can apply the backup results in the data warehouse to the target test environment, support the application of data captured in different environments at different times to the same environment and the same partition, and support the running of some tasks in the form of mock.

[0141] Exemplarily, the implementation of step 201 may include:

[0142] Step B1: Establish a requirement-task-table relationship tree through static text analysis.

[0143] First, the top tasks corresponding to the big data business needs can be manually configured. Figure 3 In the figure, Task 1 is the top task. After the top task is determined, the task dependency tree of the top task can be obtained through the big data lineage analysis tool (that is, by analyzing the pre-dependencies of the tasks, analyzing layer by layer to obtain all dependencies).

[0144] Figure 3 In the example, task 1 depends on tasks 2, 3, and 4; task 2 depends on tasks 5 and 6; and task 3 depends on tasks 7 and 8.

[0145] After obtaining the task dependency tree, text analysis can be performed on each task in the task dependency tree in turn to obtain the type of the task in the task dependency tree, so that the task can be processed accordingly according to the type of the task.

[0146] For example, refer to Figure 3 , the common syntax format of big data script operations on tables can be used to parse out the tables that tasks depend on or the result tables of tasks, thereby obtaining a relationship tree between tasks and tables. For example, the select syntax is used to parse out the tables that tasks depend on, the insert syntax is used to parse out the result tables generated for tasks, and the drop syntax is used to parse out the result tables produced by tasks. Here, the result table produced by tasks using the drop syntax is an intermediate table.

[0147] Figure 3 In the table, task 1 depends on table A, table B, and table C, task 2 depends on table E and table F, and the result table of task 2 is table A. Task 3 depends on table F and table H. The result table of task 3 is table B, the result table of task 4 is table C, the result table of task 5 is table E, the result table of task 6 is table F, and the result table of task 8 is table H.

[0148] By executing step B1, a relationship tree between tasks and tables can be established for each big data business requirement; Figure 4 An example diagram of a requirement-task-table relationship tree established through static text analysis, in which Task 1, Task 12 and Task 13 are three different top tasks corresponding to Requirement 1.

[0149] Step B2: According to the query keyword in the hive table, query is performed in the hive table to obtain the corresponding query result of at least one table, and the corresponding query result of at least one table is used as the backup data corresponding to each customer identity.

[0150] The embodiment of the present application can realize hive data capture for specified customers. For example, by inputting the identity information of a certain customer, the table related to the requirement of this identity information can be captured; after establishing the relationship tree of requirement-task-table, the query keyword can be found for each hive table, so as to query in the hive table according to the query keyword in the hive table.

[0151] In actual applications, an SQL statement for obtaining query keywords may be established for the big data business of each product. Exemplarily, only one configuration is required for each product.

[0152] After determining the query keyword, the hive table can be searched according to the query keyword of the hive table, and the field name of each query keyword can be matched with the field of the hive table one by one using the field matching method. If the match is successful, it means that the table hits the corresponding query keyword, that is, the query keyword can be classified as the query keyword of the table; if multiple query keywords are hit, multiple query keywords can be classified as the query keywords of the table. In the embodiment of the present application, even if the value of each field in the hive table is empty, the corresponding query keyword can be retrieved.

[0153] After the query keyword of the hive table is retrieved, if there are still tables that do not match the query keyword, manual intervention can be performed. If it is determined that the field meanings of the two fields are the same but the field names are different, the mapping field of the query keyword can be reconfigured. If the analysis shows that there are new query keywords, query keywords can also be added manually; after manual intervention, the above process of searching in the hive table can be re-executed until the query keyword is retrieved from each table except the parameter-related table.

[0154] By executing step 201, an unsupervised training can be performed for each big data business requirement to obtain a customer-level data model corresponding to the big data business requirement. The key points of the customer-level data model include the relationship tree of requirements-tasks-tables and the query keywords of each table. Step 202 can be executed based on the relationship tree of requirements-tasks-tables and the query keywords of each table.

[0155] Exemplarily, the implementation of step 202 may include:

[0156] When testing a big data requirement, after the test verification meets expectations, the tester uses the hive data backup tool to make a backup. The parameters collected in the hive data backup tool may include: requirement name, environment, customer identity information id_no, partition date; based on the requirement name and the relationship tree between requirement-task-table, the list of tables that need to be backed up for the requirement can be determined, and all query keywords are retrieved in each table based on the customer identity information id_no; then, based on the query keywords of each table in the data model, the data of each table of the corresponding customer is obtained, and the data of each table of the corresponding customer is backed up to the hive data warehouse.

[0157] Reference Figure 5The source customer identity information includes at least one type of customer identity information. For each type of customer identity information in the source customer identity information, corresponding data backup can be performed in the customer-level data model according to the customer identity information. A corresponding form of data can be generated for each type of customer identity information. The identity identification number (Identity Document, ID) of the form data is a unique identifier of the backup data corresponding to each customer identity.

[0158] After obtaining the morphological data corresponding to each type of customer identity information, the morphological data corresponding to each type of customer identity information can be permanently stored in a data warehouse to achieve data backup of the morphological data corresponding to each type of customer identity information.

[0159] In the embodiment of the present application, the insert statement is used to reuse data when cloning data. In actual application, refer to Figure 5 , the target identity information includes at least one customer identity information; for each customer identity information in the target identity information, the difference field in the backup result can be replaced; and because the cloned target partition is different from the source partition, the date field in the backup result can be time-shifted. After the difference field and date field in the backup result are adaptively modified, the insert statement is actually executed into the hive table, thereby generating new morphological data in the target environment. Here, the new morphological data refers to the morphological data obtained after the difference field and date field in the morphological data of the data warehouse are adaptively modified.

[0160] For example, refer to Figure 6 , the date field in the source morphological data can be time-shifted to obtain new morphological data that can be applied in the target environment. Here, the current business date of the target environment is different from the business date in the source morphological data.

[0161] Exemplarily, for the data cloning process, the embodiment of the present application also supports running some tasks in the form of mock, that is, Figure 7 For Figure 4 Schematic diagram of mock testing for Task 3 based on .

[0162] In summary, the embodiment of the present application can adopt an unsupervised training method based on data cloning technology and big data lineage analysis technology to establish a corresponding customer-level data model for each big data business demand, and use the customer-level data model for data backup and data cloning. The key points of the embodiment of the present application include: the text analysis of the shell script and the task dependency tree to establish a demand-task-table relationship tree, and then train the difference fields of each table through statistical analysis of the test environment data, and find query keywords for each table, so as to establish a customer-level data model for the loan product through an unsupervised training method. At the application level, the embodiment of the present application can perform customer-level data backup and data cloning on hive data, and mock testing can be enabled during data cloning. The embodiment of the present application can improve the efficiency of big data number creation and the efficiency of data preparation for the automation of big data logical functions.

[0163] Based on the data testing method proposed in the above embodiment, the embodiment of the present application further proposes a data testing device; Figure 8 This is an optional structural diagram of the data testing device of the embodiment of the present application, such as Figure 8 As shown, the data testing device 800 may include:

[0164] A determination module 801 is used to determine a corresponding customer-level data model for each big data business requirement, wherein the customer-level data model includes at least one table in a data warehouse tool corresponding to each big data business requirement, and a query keyword for each table in the at least one table, wherein the query keyword is used to represent information of at least one customer identity;

[0165] A backup module 802 is used to perform data backup according to customer identities for the customer-level data model corresponding to each big data business demand, and obtain a backup result, wherein the backup result includes backup data of each customer identity in the at least one customer identity;

[0166] The testing module 803 is used to process the backup result after obtaining the data testing request to obtain the data to be tested; and perform data testing on the data to be tested.

[0167] In some embodiments of the present application, the determining module 801 is specifically used to:

[0168] Determine at least one big data task corresponding to each big data business requirement; analyze the at least one big data task to determine a task dependency tree for the at least one big data task, wherein the task dependency tree represents a task dependency relationship for each big data task in the at least one big data task;

[0169] Analyze each task in the task dependency tree to determine a result table in the data warehouse tool corresponding to each task in the task dependency tree; use the result table as at least one table in the data warehouse tool corresponding to each big data business requirement;

[0170] The customer-level data model corresponding to each big data business requirement is determined according to at least one table in the data warehouse tool corresponding to each big data business requirement and a query keyword of each table in the at least one table.

[0171] In some embodiments of the present application, the determining module 801 is specifically used to:

[0172] When the task in the task dependency tree is a data extraction task, reading a table that the task in the task dependency tree depends on, and using the read table as a result table in a data warehouse tool corresponding to the task in the task dependency tree;

[0173] When the task in the task dependency tree is a processing task, determine the script storage path corresponding to the task in the task dependency tree, read the script from the script storage path, execute the read script, and obtain the result table in the data warehouse tool corresponding to the task in the task dependency tree.

[0174] In some embodiments of the present application, the determining module 801 is specifically used to:

[0175] identifying a specific field in each of the at least one table, the specific field comprising at least one of a difference field and a date field, the difference field representing: when the at least one table comprises a plurality of tables, a field in the plurality of tables having the same field name and different field values;

[0176] According to the query keywords of each table in the at least one table, the customer-level data model corresponding to each big data business demand is obtained, and the customer-level data model also includes: the specific fields in each table of the at least one table identified.

[0177] In some embodiments of the present application, the determining module 801 is specifically used to:

[0178] Obtain a specified field, and use the specified field in each table of the at least one table as the difference field;

[0179] The number of enumeration values ​​appearing in each other field in the at least one table is counted, and the difference fields in each table of the at least one table are determined according to the number of enumeration values ​​appearing in each other field in the at least one table; the other fields represent fields other than the specified fields.

[0180] In some embodiments of the present application, the determining module 801 is specifically used to:

[0181] When the number of enumeration values ​​of the target field in the at least one table is greater than or equal to a first set threshold, the target field is determined to be a difference field; the target field represents any other field in the at least one table.

[0182] In some embodiments of the present application, the first set threshold is the product of the total number of records of each other field in the at least one table and a set ratio, and the set ratio is a positive number less than 1.

[0183] In some embodiments of the present application, the testing module 803 is specifically used to:

[0184] The specific fields in each table in the backup result are processed to obtain data to be tested.

[0185] In some embodiments of the present application, the testing module 803 is specifically used to:

[0186] When the specific field includes a difference field, the difference field in each table in the backup result is replaced to obtain the data to be tested;

[0187] When the specific field includes a date field, the date field in each table in the backup result is time-series shifted to obtain the data to be tested.

[0188] In some embodiments of the present application, the testing module 803 is specifically used to:

[0189] A mock test is performed on at least part of the data to be tested to obtain a test result corresponding to at least part of the data.

[0190] In practical applications, the determination module 801, the backup module 802 and the test module 803 can all be implemented using a processor of an electronic device, and the processor can be at least one of an ASIC, a DSP, a DSPD, a PLD, an FPGA, a CPU, a controller, a microcontroller, and a microprocessor. It can be understood that the electronic device that implements the functions of the processor can also be other, and the embodiments of the present application are not limited thereto.

[0191] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0192] It should be noted that in the embodiments of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a terminal, a server, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0193] Correspondingly, an embodiment of the present application further provides a computer program product, which includes computer executable instructions, and the computer executable instructions are used to implement any data testing method provided in the embodiment of the present application.

[0194] Accordingly, an embodiment of the present application further provides a computer storage medium, on which computer executable instructions are stored, and the computer executable instructions are used to implement any one of the data testing methods provided in the above embodiments.

[0195] The present application also provides an electronic device, Fig. 9 A schematic diagram of an optional structure of an electronic device provided in an embodiment of the present application, such as Fig. 9 As shown, the electronic device 90 includes:

[0196] Memory 901, used to store executable instructions;

[0197] The processor 902 is used to implement any one of the above data testing methods when executing the executable instructions stored in the memory 901.

[0198] The processor 902 may be at least one of an ASIC, a DSP, a DSPD, a PLD, a FPGA, a CPU, a controller, a microcontroller, and a microprocessor.

[0199] The above-mentioned computer-readable storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0200] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0201] It should be understood that the "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiments are included in at least one embodiment of the present application. Therefore, the "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0202] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0203] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0204] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0205] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0206] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for the automatic test line of the device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0207] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0208] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0209] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data testing method, characterized in that: The method comprises: For each big data business requirement, determine a corresponding customer-level data model, wherein the customer-level data model includes at least one table in a data warehouse tool corresponding to each big data business requirement, and a query keyword for each table in the at least one table, wherein the query keyword is used to characterize information of at least one customer identity; For the customer-level data model corresponding to each big data business demand, data is backed up according to customer identity to obtain a backup result, wherein the backup result includes backup data of each customer identity in the at least one customer identity; After obtaining the data test request, the backup result is processed to obtain the data to be tested; the data to be tested is tested; and the corresponding customer-level data model is determined for each big data business requirement, including: Determine at least one big data task corresponding to each big data business requirement; analyze the at least one big data task to determine a task dependency tree for the at least one big data task, wherein the task dependency tree represents a task dependency relationship for each big data task in the at least one big data task; Analyze each task in the task dependency tree to determine a result table in the data warehouse tool corresponding to each task in the task dependency tree; use the result table as at least one table in the data warehouse tool corresponding to each big data business requirement; The customer-level data model corresponding to each big data business requirement is determined according to at least one table in the data warehouse tool corresponding to each big data business requirement and a query keyword of each table in the at least one table.

2. The method according to claim 1, characterized in that The step of analyzing each task in the task dependency tree to determine a result table in a data warehouse tool corresponding to each task in the task dependency tree includes: When the task in the task dependency tree is a data extraction task, reading a table that the task in the task dependency tree depends on, and using the read table as a result table in a data warehouse tool corresponding to the task in the task dependency tree; When the task in the task dependency tree is a processing task, determine the script storage path corresponding to the task in the task dependency tree, read the script from the script storage path, execute the read script, and obtain the result table in the data warehouse tool corresponding to the task in the task dependency tree.

3. The method according to claim 1, characterized in that Determining the customer-level data model corresponding to each big data business requirement according to at least one table in the data warehouse tool corresponding to each big data business requirement and a query keyword of each table in the at least one table includes: identifying a specific field in each of the at least one table, the specific field comprising at least one of a difference field and a date field, the difference field representing: when the at least one table comprises a plurality of tables, a field in the plurality of tables having the same field name and different field values; According to the query keywords of each table in the at least one table, the customer-level data model corresponding to each big data business demand is obtained, and the customer-level data model also includes: the specific fields in each table of the at least one table identified.

4. The method according to claim 3, characterized in that: Identifying a difference field in each of the at least one table includes: Obtain a specified field, and use the specified field in each table of the at least one table as the difference field; The number of enumeration values ​​appearing in each other field in the at least one table is counted, and the difference fields in each table of the at least one table are determined according to the number of enumeration values ​​appearing in each other field in the at least one table; the other fields represent fields other than the specified fields.

5. The method according to claim 4, characterized in that Determining the difference fields in each table of the at least one table according to the number of enumeration values ​​of each other field in the at least one table comprises: When the number of enumeration values ​​of the target field in the at least one table is greater than or equal to a first set threshold, the target field is determined to be a difference field; the target field represents any other field in the at least one table.

6. The method according to claim 5, characterized in that The first set threshold is the product of the total number of records of each other field in the at least one table and a set ratio, and the set ratio is a positive number less than 1.

7. The method according to claim 4, characterized in that The step of processing the backup result to obtain the data to be tested includes: The specific fields in each table in the backup result are processed to obtain data to be tested.

8. The method according to claim 7, characterized in that The processing of the specific field in each table in the backup result to obtain the data to be tested includes: When the specific field includes a difference field, the difference field in each table in the backup result is replaced to obtain the data to be tested; When the specific field includes a date field, the date field in each table in the backup result is time-series shifted to obtain the data to be tested.

9. The method according to claim 1, characterized in that: The performing data testing on the data to be tested includes: A mock test is performed on at least part of the data to be tested to obtain a test result corresponding to at least part of the data.

10. A data testing device, characterized in that: The device comprises: A determination module, configured to determine a corresponding customer-level data model for each big data business requirement, wherein the customer-level data model includes at least one table in a data warehouse tool corresponding to each big data business requirement, and a query keyword for each table in the at least one table, wherein the query keyword is used to characterize information of at least one customer identity; A backup module, configured to perform data backup according to customer identities for the customer-level data model corresponding to each big data business requirement, and obtain a backup result, wherein the backup result includes backup data for each customer identity of the at least one customer identity; The test module is used to process the backup result after obtaining the data test request to obtain the data to be tested; and perform data testing on the data to be tested; The determination module is specifically used for: Determine at least one big data task corresponding to each big data business requirement; analyze the at least one big data task to determine a task dependency tree for the at least one big data task, wherein the task dependency tree represents a task dependency relationship for each big data task in the at least one big data task; Analyze each task in the task dependency tree to determine a result table in the data warehouse tool corresponding to each task in the task dependency tree; use the result table as at least one table in the data warehouse tool corresponding to each big data business requirement; The customer-level data model corresponding to each big data business requirement is determined according to at least one table in the data warehouse tool corresponding to each big data business requirement and a query keyword of each table in the at least one table.

11. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, configured to implement the data testing method according to any one of claims 1 to 9 when executing the executable instructions stored in the memory.

12. A computer-readable storage medium, characterized in that: Executable instructions are stored, and when executed by a processor, the data testing method according to any one of claims 1 to 9 is implemented.

Images