Method, device and equipment for multi-data-source data test verification
By acquiring and associating multiple business data tables and judging their field consistency, the problems of long data verification time and insufficient coverage are solved, and fast and efficient data verification and more comprehensive data coverage are achieved.
Patent Information
- Application Number
- CN202411110978.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-26
AI Technical Summary
During the data accuracy verification process, testers need to check defects step by step, resulting in long-term and incomplete data coverage.
By obtaining multiple business data tables, determining statistical dimensions, performing correlation processing, judging the field consistency of the output table and business source data table of the data application layer, and improving the efficiency and coverage of data verification.
Quickly carry out data verification from multiple data sources, improve the testing efficiency of testers, improve the coverage of data verification, and ensure the accuracy and completeness of data.
Smart Images

Figure CN119025563A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of automated data processing, and in particular to a method, device and equipment for testing and verifying data from multiple data sources. Background Art
[0002] Software as a Service (SaaS) allows users to use cloud-based applications through an Internet connection. The travel SaaS platform is a system that serves multiple companies. Companies that rent systems are called tenants. A SaaS travel system has many tenants, each of which involves business modules such as basic operations, assets, finance, transactions, and clients. Each business module has a large amount of data and most of them are stored in separate databases and tables. In the process of verifying the accuracy of the data, testers are required to query and troubleshoot defects step by step, which takes a long time and does not cover all the data. Summary of the invention
[0003] In view of this, the present invention provides a method, device and equipment for testing and verifying data from multiple data sources, so as to solve the problem that in the process of verifying the accuracy of data, testers need to query and troubleshoot defects step by step, which takes a long time and has incomplete data coverage.
[0004] In a first aspect, the present invention provides a method for testing and verifying data from multiple data sources, the method comprising: acquiring and accommodating multiple business data tables to obtain multiple business source data tables; determining a statistical dimension based on data analysis requirements; performing a first association processing on all fields of the data application layer output table based on the fields of the statistical dimension to obtain a first association field; aggregating multiple business source data tables based on the statistical dimension to obtain a first table, performing a second association processing on the first table based on the fields of the statistical dimension to obtain a second association field; associating the first association field with the second association field to determine whether the fields of the data application layer output table and the first table are consistent.
[0005] In an optional embodiment, performing a first association processing on all fields of the data application layer output table based on the fields of the statistical dimension to obtain the first association field includes: performing encryption processing based on the spliced data application layer output table to obtain the first association field; and performing a second association processing on the first table based on the fields of the statistical dimension to obtain the second association field includes: performing the encryption processing based on the spliced first table to obtain the second association field.
[0006] In an optional embodiment, based on the first association field and the second association field, it is determined whether the fields of the data application layer output table and the business source data table are consistent, including: if the values of the target fields of the first association field and the second association field are inconsistent, it is determined that the fields of the data application layer output table and the business source data table are inconsistent; if the target field in the first association field is empty and the target field in the second association field is not empty, it is determined that the fields of the data application layer output table and the business source data table are inconsistent; if the target field in the first association field is not empty and the target field in the second association field is empty, it is determined that the fields of the data application layer output table and the business source data table are inconsistent.
[0007] In an optional embodiment, the aforementioned method for testing and verifying data from multiple data sources also includes: after associating the multiple business data tables and the accommodated business source data tables based on the statistical dimensions, determining whether the values of the fields of the business source data tables and the business data tables are consistent.
[0008] In an optional implementation, the aforementioned method for testing and verifying data from multiple data sources further includes: obtaining an enumeration value for each field in the data application layer output table, and if the enumeration value does not meet a preset condition, determining that the testing and verification of data from multiple data sources fails.
[0009] In an optional implementation, obtaining the enumeration value of each field in the data application layer output table includes: obtaining the enumeration value of each field in the data application layer output table; deduplicating the enumeration values, and determining whether the preset conditions are met based on the deduplicated enumeration values.
[0010] In an optional embodiment, the method also includes: synchronizing the data application layer output table to the real-time data warehouse; splicing all the fields of the data application layer output table synchronized to the real-time data warehouse based on the statistical dimension field to obtain a third associated field; based on the second associated field and the third associated field, determining whether the fields of the business source data table and the data application layer output table synchronized to the real-time data warehouse are consistent.
[0011] In a second aspect, the present invention provides a device for testing and verifying data from multiple data sources, the device comprising: a first acquisition module, used to acquire and store multiple business data tables to obtain multiple business source data tables; a second acquisition module, used to determine the statistical dimension based on data analysis requirements; a third acquisition module, used to perform a first association processing on all fields of the data application layer output table based on the fields of the statistical dimension to obtain a first association field; a fourth acquisition module, used to aggregate the multiple business source data tables based on the statistical dimension to obtain a first table, and perform a second association processing on the first table based on the fields of the statistical dimension to obtain a second association field; a judgment module, used to associate the first association field with the second association field to determine whether the fields of the data application layer output table and the first table are consistent.
[0012] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method for testing and verifying data from multiple data sources according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0013] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for testing and verifying data from multiple data sources according to the first aspect or any corresponding embodiment thereof.
[0014] Determine the statistical dimensions based on data analysis requirements, associate the business source data table with the business source data table after accommodation based on the statistical dimensions, and judge the consistency of the business source data table with the business source data table after accommodation. This can quickly verify the data source, improve the testing efficiency of testers, and increase the coverage of data verification. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0016] Figure 1 A schematic diagram of a process for testing and verifying data from multiple data sources provided by an embodiment of the present invention is shown;
[0017] Figure 2 A schematic diagram of the process of data development and data testing and verification in an embodiment of the present invention is shown;
[0018] Figure 3 A schematic diagram of the structure of a device for testing and verifying data from multiple data sources provided by an embodiment of the present invention is shown;
[0019] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0021] According to an embodiment of the present invention, a method embodiment for testing and verifying data from multiple data sources is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0022] In this embodiment, a method for testing and verifying data from multiple data sources is provided. Figure 1 A schematic diagram of a process for testing and verifying data from multiple data sources provided by an embodiment of the present invention is shown. Figure 1 As shown, the process includes the following steps:
[0023] Step S101, acquiring and storing multiple business data tables to obtain multiple business source data tables.
[0024] In this step, data collection refers to the collection and processing of the received business source data table, based on the Open Data Processing Service (ODPS) to collect the business data table, and obtain the collected business source data table corresponding to the business data table, that is, S table. Among them, ODPS is a massive data offline processing service.
[0025] The collection method of business source data tables can be set based on the difference in data volume. For business source data tables with small data volume, they can be collected on a daily basis, and for business source data tables with large data volume, they can be collected on a minute or hour basis. The specific collection method can be determined based on actual business needs. For business source data tables with small data volume, the latest snapshot data of the business party's relational database management system (MySQL) table or distributed column-oriented open source database (HBase) table can be collected regularly after midnight every day; for business source data tables with large data volume, snapshot data within the corresponding time range can be collected on a minute or hour basis based on business logs.
[0026] The business data table may include: business party table A, business party table B and business party table C. Among them, business party table A may include: a first column (column_a), a second column (column_b), a third column (column_c) and a fourth column (column_d). Business party table B may include: a first column (column_a), a second column (column_b), a fifth column (column_e) and a sixth column (column_f). Business party table C may include: a third column (column_c), a seventh column (column_g) and an eighth column (column_h). The number and type of specific business data tables are determined based on actual business needs. The above is only an exemplary description of the business data table in the embodiment of the present invention and does not constitute a limitation to the embodiment of the present invention.
[0027] Step S103, determining the statistical dimension based on data analysis requirements.
[0028] In this step, the output table of the data application layer (Application Data Store, ADS for short) stores the statistical indicator data or report data of the data product. It is mainly used for data products and data analysis. It is usually divided into traffic, orders or users according to business needs, and generates a wide table with more fields for subsequent business query or data distribution.
[0029] In terms of data granularity, ADS data is summary-level data and also includes some detailed data. In terms of the time span of data, it is usually part of the DW layer, and its main purpose is to meet the needs of user analysis. From the perspective of analysis, users usually only need to analyze data from the past few years. In terms of the breadth of data, it still covers all business data. In a data warehouse, a dimension is a description of a specific aspect of data, and is usually used to classify and organize data. Dimensions can include various attributes, such as time, location, product, or user, etc.
[0030] Data analysis requirements are determined based on specific business scenarios. For example, for the driver dimension table, the statistical dimensions obtained are the latest snapshot data of the driver identifier and date; for the order daily increment table, the statistical dimensions obtained are the latest snapshot data of the order identifier, driver identifier, and date.
[0031] The statistical dimensions of the data application layer output table can be determined based on the data analysis requirements and indicator caliber documents. Indicator caliber refers to the metrics and calculation methods used in data analysis and data reporting. For example, indicator caliber can be statistical caliber, business caliber or technical caliber. Among them, statistical caliber refers to the statistical methods and calculation formulas used in data analysis. Business caliber refers to the business indicators and calculation methods used in data analysis. Technical caliber refers to the technical indicators and calculation methods used in data analysis.
[0032] Step S105, performing a first association process on all fields of the data application layer output table based on the fields of the statistical dimension to obtain a first associated field.
[0033] In this step, all fields of the data application layer output table (table a) are queried and concatenated based on the statistical dimension fields to obtain the first associated field a.str.
[0034] Step S107, aggregating multiple business source data tables based on the statistical dimension to obtain a first table, performing a second association process on the fields of the first table based on the statistical dimension to obtain a second associated field.
[0035] After querying ODPS, table S is aggregated according to the statistical dimension to obtain the first table (table b). The fields of the first table are spliced based on the statistical dimension to obtain the second associated field b.str.
[0036] Step S109, associating the first associated field with the second associated field, and determining whether the fields of the data application layer output table and the first table are consistent.
[0037] The first associated field and the second associated field are associated through FULLJOIN, the association condition is ON a.str=b.str, and the where condition is: one of the first associated field and the second associated field is null and inconsistent, or the values of the target fields in the first associated field and the second associated field are inconsistent.
[0038] For example, where=(NVL(a.str,'default')<>NVL(b.str,'default'))OR NVL(a.clumn_d,'default')<>NVL(b.clumn_d,'default')).
[0039] The method for testing and verifying data from multiple data sources provided in this embodiment determines the statistical dimensions based on data analysis requirements, associates the business source data table with the contained business source data table based on the statistical dimensions, and then determines the consistency of the business source data table with the contained business source data table. This allows for rapid data verification of all data sources, improves the testing efficiency of testers, and increases the coverage of data verification.
[0040] Figure 2 FIG. 2 shows a flow chart of data development and data testing in an embodiment of the present invention. Figure 2 As shown in the figure, ① after the data is collected, the S table is used to write the associated SQL calculation logic statements according to the relationship between the business tables to obtain the first table; ② data testing: confirm the statistical dimensions, main tables and the relationship between tables through the requirements document and indicator caliber, and write SQL statements that conform to the fields of the data application layer output table; ③ the tables obtained by the data test and the data application layer output table are associated through the statistical dimension fields to compare the total number and the data in each field; ⑤ ODPS data is synchronized with hologress, and data synchronization test verification is performed, ODPS data is synchronized with clickhouse, and data synchronization test verification is performed.
[0041] like Figure 2 As shown, data development includes: after data collection, cleaning and aggregation processing, the data application layer output table is obtained, and then it is synchronized with hologress or clickhouse based on the specific usage scenario. Service development queries through the interface and then returns it to the business party.
[0042] In this way, executing the SQL statements written above can verify the accuracy of each field in the data application layer output table and statistical dimension.
[0043] In some optional implementations, all fields of the data application layer output table are associated based on the fields of the statistical dimension to obtain a first associated field, including: encryption processing based on the spliced data application layer output table to obtain the first associated field; and, the first table is second associated based on the fields of the statistical dimension to obtain a second associated field, including: encryption processing based on the spliced first table to obtain the second associated field.
[0044] In this embodiment, the function used for encryption processing may be an MD5 function, a SHA1 function, or a SHA2 function. The spliced data application layer output table may be encrypted using the MD5 function, and the first associated field obtained is the encrypted field a.md5_str. The spliced first table is encrypted using the MD5 function, and the second associated field obtained is the encrypted field b.md5_str. Among them, MD5 is a hash function used to convert a given string into an MD5 hash value. The MD5 hash algorithm converts input data of any length into a 128-bit hash value, usually expressed as 32 hexadecimal digits.
[0045] The encrypted first associated field and the second associated field are associated through FULL JOIN, with an associated condition ONa.md5_str=b.md5_str, and a where condition is: one of the first associated field and the second associated field is null and inconsistent, or the values of the target fields in the first associated field and the second associated field are inconsistent.
[0046] For example, where = (NVL (a.md5_str, 'default') <> NVL
[0047] (b.md5_str,'default'))OR NVL(a.clumn_d,'default')<>NVL(b.clumn_d,'default')).
[0048] In this way, plain text data is not queried or displayed as much as possible, and online data can be kept confidential.
[0049] In some optional embodiments, based on the first associated field and the second associated field, it is determined whether the fields of the data application layer output table and the business source data table are consistent, including: if the values of the target fields of the first associated field and the second associated field are inconsistent, it is determined that the fields of the data application layer output table and the business source data table are inconsistent; if the target field in the first associated field is empty and the target field in the second associated field is not empty, it is determined that the fields of the data application layer output table and the business source data table are inconsistent; if the target field in the first associated field is not empty and the target field in the second associated field is empty, it is determined that the fields of the data application layer output table and the business source data table are inconsistent.
[0050] In this implementation, the number of inconsistent query results is expected to be zero. If the number of inconsistent query results is not zero, if either the data application layer output table or the business source data table is NULL, there may be a problem with the calculation of the data development personnel, or there may be a problem with the calculation logic of the test case. The test personnel can first troubleshoot the problem based on the difference. If there is no result, the data development personnel can assist in troubleshooting the problem until the cause is found, and the party with the problem can modify the sql calculation logic.
[0051] For example, the relevant SQL statements are as follows:
[0052]
[0053]
[0054]
[0055] By finding the rows that generate NULL values, you can find the differences between the data application layer output table and the business source data table.
[0056] In some optional embodiments, the aforementioned method for testing and verifying data from multiple data sources further includes: after associating multiple business data tables with the accommodated business source data table based on statistical dimensions, determining whether the values of the fields of the business source data table and the business data table are consistent.
[0057] In this embodiment, the S table and the business source data table can be connected through a script, and after the statistical dimension field is associated, it is determined whether the values of each field in the S table and the business source data table are consistent. For example, the ODPS and the relational database management system (mysql) are connected through a script, and the S table and the business source data table after ODPS are received are queried. After the statistical dimension field is associated, the values of each field in the two tables are compared to see if they are consistent.
[0058] In some optional implementations, the aforementioned method for multi-data source data verification further includes: obtaining an enumeration value for each field in the data application layer output table, and if the enumeration value does not meet a preset condition, determining that the multi-data source data test verification fails.
[0059] In this implementation, the enumeration value of each field can be queried through the distinct function. Distinct is a keyword used in SQL to return a unique result set that does not repeat. It specifies one or more columns to remove duplicates and returns a unique result. It can ensure the legitimacy of the enumeration values in the output table of the data application layer, thereby improving the accuracy and reliability of the data.
[0060] In some optional implementations, obtaining the enumeration value of each field in the data application layer output table includes: obtaining the enumeration value of each field in the data application layer output table; deduplicating the enumeration value, and determining whether the preset condition is met based on the deduplicated enumeration value.
[0061] In this embodiment, the preset condition can be set to that the enumeration value is an integer greater than or equal to zero, or, for the enumeration value of the amount type, it is generally two decimal places according to the indicator caliber document. For example, the enumeration value of the completed order quantity indicator must be an integer greater than or equal to zero. If null or decimals appear, the data test verification fails. Those skilled in the art can set the preset conditions according to the attributes of the field. Based on the distinct deduplication logic to verify whether the enumeration value that appears in the field meets expectations, the enumeration value can be tested efficiently and accurately.
[0062] In some optional embodiments, the aforementioned method for multi-data source data verification also includes: synchronizing the data application layer output table to the real-time data warehouse; splicing all fields of the data application layer output table synchronized to the real-time data warehouse based on the statistical dimension field to obtain a third associated field; based on the second associated field and the third associated field, determining whether the fields of the business source data table and the data application layer output table synchronized to the real-time data warehouse are consistent.
[0063] In this embodiment, the data application layer output table is synchronized to hologress or clickhouse. ODPS and hologress are connected through automated scripts, or ODPS and clickhouse are connected through automated scripts, and the tables in ODPS and hologress are associated based on the statistical dimension fields, and the fields in the tables in ODPS and hologress are compared to see if they are consistent. The relevant SQL statements are shown above and will not be repeated here.
[0064] In this way, the consistency of the data in the test data warehouse and the data in ClickHouse, or the consistency of the data in the data warehouse and the data in HoloLogress can be verified.
[0065] In this embodiment, a device for multi-data source data testing and verification is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0066] This embodiment provides a device for testing and verifying data from multiple data sources, such as Figure 3 As shown, including:
[0067] The first acquisition module 501 is used to acquire and store multiple business data tables to obtain multiple business source data tables.
[0068] The second acquisition module 502 is used to determine the statistical dimension based on the data analysis requirement.
[0069] The third acquisition module 503 is used to perform a first association process on all fields of the data application layer output table based on the fields of the statistical dimension to obtain a first associated field.
[0070] The fourth acquisition module 504 is used to aggregate multiple business source data tables based on the statistical dimension to obtain a first table, and perform a second association process on the fields of the first table based on the statistical dimension to obtain a second association field.
[0071] The judgment module 505 is used to associate the first associated field with the second associated field to determine whether the fields of the data application layer output table and the first table are consistent.
[0072] In some optional implementations, the third acquisition module 503 includes:
[0073] The first unit of the third acquisition module is used to perform encryption processing based on the spliced data application layer output table to obtain a first associated field.
[0074] In some optional implementations, the fourth acquisition module 504 includes:
[0075] The first unit of the fourth acquisition module is used to perform encryption processing based on the spliced first table to obtain a second associated field.
[0076] In some optional implementations, the determination module 505 includes:
[0077] The first unit of the judgment module is used to determine that the fields of the data application layer output table and the business source data table are inconsistent if the values of the target fields of the first associated field and the second associated field are inconsistent.
[0078] The second unit of the judgment module is used to determine that the fields of the data application layer output table and the business source data table are inconsistent if the target field in the first association field is empty and the target field in the second association field is not empty.
[0079] The third unit of the judgment module is used to determine that the fields of the data application layer output table and the business source data table are inconsistent if the target field in the first association field is not empty and the target field in the second association field is empty.
[0080] In some optional implementations, the aforementioned apparatus for testing and verifying data from multiple data sources further includes:
[0081] The second judgment module is used to associate the multiple business data tables with the contained business source data table based on the statistical dimension, and then judge whether the values of the fields in the business source data table and the business data table are consistent.
[0082] In some optional implementations, the aforementioned apparatus for testing and verifying data from multiple data sources further includes:
[0083] The sixth acquisition module is used to obtain the enumeration value of each field in the data application layer output table. If the enumeration value does not meet the preset conditions, it is determined that the multi-data source data test verification fails.
[0084] In some optional implementations, the sixth acquisition module includes:
[0085] The first unit of the sixth acquisition module is used to obtain the enumeration value of each field in the data application layer output table; deduplicate the enumeration value, and determine whether it meets the preset conditions based on the deduplicated enumeration value.
[0086] In some optional implementations, the aforementioned apparatus for testing and verifying data from multiple data sources further includes:
[0087] The seventh acquisition module is used to synchronize the data application layer output table to the real-time data warehouse; splice all the fields of the data application layer output table synchronized to the real-time data warehouse based on the statistical dimension field to obtain the third associated field; based on the second associated field and the third associated field, determine whether the fields of the business source data table and the data application layer output table synchronized to the real-time data warehouse are consistent.
[0088] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0089] The device for multi-data source data verification provided by the embodiment of the present invention determines the statistical dimension based on the data analysis requirement, associates the business source data table with the contained business source data table based on the statistical dimension, and then judges the consistency of the business source data table with the contained business source data table. This can quickly perform data verification on all data sources, improve the testing efficiency of testers, and increase the coverage of data verification.
[0090] The device for multi-data source data verification in this embodiment is presented in the form of a functional unit, where the unit refers to an application specific integrated circuit (ASIC) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0091] The embodiment of the present invention also provides a computer device having the above Figure 3The device shown is used for data verification from multiple data sources.
[0092] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a graphical user interface on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.
[0093] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0094] The aforementioned memory 20 stores instructions executable by at least one processor 10, so that the aforementioned at least one processor 10 executes the method shown in the above embodiment.
[0095] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0096] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0097] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.
[0098] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (such as a light emitting diode) and a tactile feedback device (such as a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.
[0099] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0100] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.
[0101] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A method for testing and verifying data from multiple data sources, characterized in that: The method comprises: Acquire and store multiple business data tables to obtain multiple business source data tables; Determine statistical dimensions based on data analysis requirements; Performing a first association process on all fields of the data application layer output table based on the fields of the statistical dimension to obtain a first association field; Aggregating the plurality of business source data tables based on the statistical dimension to obtain a first table, and performing a second association process on the fields of the first table based on the statistical dimension to obtain a second association field; The first associated field and the second associated field are associated to determine whether the fields of the data application layer output table and the first table are consistent.
2. The method according to claim 1, characterized in that The step of performing a first association process on all fields of the data application layer output table based on the fields of the statistical dimension to obtain a first associated field includes: performing encryption processing based on the spliced data application layer output table to obtain the first associated field; and The performing a second association process on the field of the statistical dimension of the first table to obtain a second association field includes: performing the encryption process based on the spliced first table to obtain the second association field.
3. The method according to claim 1 or 2, characterized in that: Based on the first associated field and the second associated field, determining whether the fields of the data application layer output table and the business source data table are consistent includes: If the values of the target fields of the first association field and the second association field are inconsistent, it is determined that the fields of the data application layer output table and the service source data table are inconsistent; If the target field in the first associated field is empty, and the target field in the second associated field is not empty, it is determined that the fields of the data application layer output table and the business source data table are inconsistent; If the target field in the first association field is not empty and the target field in the second association field is empty, it is determined that the fields of the data application layer output table and the business source data table are inconsistent.
4. The method according to claim 1, characterized in that: The method further comprises: After associating the plurality of business data tables with the contained business source data table based on the statistical dimension, it is determined whether the values of the fields of the business source data table and the business data table are consistent.
5. The method according to claim 1, characterized in that The method further comprises: The enumeration value of each field in the data application layer output table is obtained, and if the enumeration value does not meet the preset conditions, it is determined that the multi-data source data test verification fails.
6. The method according to claim 5, characterized in that Obtain the enumeration value of each field in the data application layer output table, including: Obtaining the enumeration value of each field in the data application layer output table; The enumerated values are deduplicated, and whether the preset condition is met is determined based on the deduplicated enumerated values.
7. The method according to any one of claims 4 to 6, characterized in that The method further comprises: Synchronize the data application layer output table to the real-time data warehouse; All fields of the data application layer output table synchronized to the real-time data warehouse are spliced based on the statistical dimension field to obtain a third associated field; Based on the second associated field and the third associated field, it is determined whether the fields of the business source data table and the data application layer output table synchronized to the real-time data warehouse are consistent.
8. A device for testing and verifying data from multiple data sources, characterized in that: The device comprises: A first acquisition module is used to acquire and store multiple business data tables to obtain multiple business source data tables; The second acquisition module is used to determine the statistical dimension based on the data analysis requirements; A third acquisition module is used to perform a first association process on all fields of the data application layer output table based on the fields of the statistical dimension to obtain a first associated field; A fourth acquisition module, configured to aggregate the plurality of business source data tables based on the statistical dimension to obtain a first table, and perform a second association process on the fields of the first table based on the statistical dimension to obtain a second association field; A judgment module is used to associate the first associated field with the second associated field to determine whether the fields of the data application layer output table and the first table are consistent.
9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for multi-data source data testing and verification as described in any one of claims 1 to 7 by executing the computer instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for testing and verifying data from multiple data sources according to any one of claims 1 to 7.
Citation Information
Patent Citations
Data consistency verification method and device, equipment, medium and program product
CN113312338A
Analysis of streaming data using deltas and snapshots
US20210004352A1
