Method, device, system and storage medium for verifying data processing quality
By connecting to the database through an interactive computing environment, and employing multiple verification logics to directly verify the quality of the data, the challenges of big data testing are solved, testing accuracy and efficiency are improved, and labor costs are reduced.
Patent Information
- Application Number
- CN202311265515.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-27
- Publication Date
- 2025-11-18
- Estimated Expiration
- 2043-09-27
AI Technical Summary
In existing technologies, it is impossible to test effectively when the amount of data is too large. Traditional functional testing methods result in large differences in sample sampling, high risk of error, and high labor costs.
By connecting to the database through an interactive computing environment, large amounts of data can be directly verified based on business needs. Multiple verification logics, such as regular expression checks, quantity checks, enumeration checks, and business logic checks, are used to automatically judge data quality and reduce manual sampling tests.
It enables accurate verification of large amounts of data, improves testing quality and efficiency, reduces labor costs, and avoids sampling errors.
Smart Images
Figure CN117312285B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and specifically to a method, apparatus, system, and storage medium for verifying the quality of data processing. Background Technology
[0002] Currently, with the increasing prevalence of system visualization, to achieve the desired effect, a large amount of data undergoes processing such as cleaning, collection, and storage during the view design process. The processed data is then stored in a new database table and statistically displayed through views. This requires testers to verify the data quality during testing. However, most companies primarily employ functional testers, whose scope is relatively limited and suitable for simpler requirements. When logical verification of large amounts of data is required, only partial data can be extracted for verification. When the amount of data to be verified reaches hundreds of thousands or even millions, the number and method of sampling for functional testing directly impact the test results. If the sample size is within a reasonable range, the sample is usable; if the sample size only represents a portion of the characteristics, the sampled test results will differ significantly from the actual results. The number of samples and the sampling method need to be specifically evaluated and calculated. When the data volume is too large, storing it directly in the database without a front-end page makes testing difficult and carries a high risk of error. Summary of the Invention
[0003] The purpose of this application is to provide a method, apparatus, system, and storage medium for verifying data processing quality, in order to solve the problems of being unable to test when the amount of test data is too large and the test accuracy is low.
[0004] To achieve the above objectives, the first aspect of this application provides a method for verifying data processing quality, applied to a system for verifying data processing quality. The system includes an apparatus for verifying data processing quality, multiple databases, and a test terminal. The apparatus for verifying data processing quality communicates with the multiple databases and the test terminal respectively. The method includes:
[0005] Obtain business requirements;
[0006] Write business requirements into the interactive computing environment;
[0007] Connects a target database across multiple databases with an interactive computing environment;
[0008] Based on the business requirements in the interactive computing environment, determine at least one processing verification result of the data in the target database;
[0009] At least one processing verification result is sent to the test terminal so that the test terminal can determine whether the data meets the data quality requirements.
[0010] In this embodiment of the application, determining at least one processing verification result of data in the target database based on business requirements in an interactive computing environment includes:
[0011] Multiple verification function points are determined based on business requirements;
[0012] Convert multiple verification function points into multiple data processing logics;
[0013] Perform at least one data processing logic on the data in the target database to determine at least one processing verification result of the data in the target database.
[0014] In this embodiment of the application, performing at least one data processing logic on the data in the target database to determine at least one processing verification result of the data in the target database includes:
[0015] Acquire data query methods and multiple data processing logics;
[0016] Execute at least one data processing logic based on the data query method;
[0017] The query methods include row-based query, column-based query, whole-table query, direct query, and encapsulated function query.
[0018] In this embodiment of the application, the data processing logic includes at least one of the following:
[0019] Regular expression checks, quantity checks, enumeration checks, and business logic checks.
[0020] In this embodiment of the application, the method further includes:
[0021] For any data, if at least one processing verification result does not meet the preset result, new data processing logic is added through the test terminal.
[0022] Execute new data processing logic on arbitrary data to obtain new processing verification results;
[0023] The new processing verification results are sent to the test terminal so that the test terminal can determine whether any data meets the data quality requirements.
[0024] In this embodiment of the application, connecting the target database and the interactive computing environment includes:
[0025] Set up the Python environment and install the dependencies for the interactive computing environment;
[0026] Execute the target command in the Python environment to open the interactive computing environment;
[0027] Import the target database into the interactive computing environment to connect to the target database.
[0028] A second aspect of this application provides an apparatus for verifying the quality of data processing, comprising:
[0029] The memory is configured to store instructions; and
[0030] The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the method for verifying data processing quality as described above.
[0031] A third aspect of this application provides a system for verifying the quality of data processing, comprising:
[0032] The apparatus for verifying the quality of data processing described above;
[0033] Multiple databases, communicating with a device for verifying data processing quality, are configured to store processed data; and
[0034] The test terminal communicates with the device for verifying the quality of data processing and is configured to receive the verification results and determine whether the data meets the data quality requirements.
[0035] A fourth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned method for verifying data processing quality.
[0036] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for verifying data processing quality.
[0037] The above technical solution first obtains business requirements, writes them into an interactive computing environment, then connects a target database from multiple databases to the interactive computing environment, and based on the business requirements in the interactive computing environment, determines at least one processing verification result for the data in the target database. Finally, at least one processing verification result is sent to a test terminal so that the test terminal can determine whether the data meets the data quality requirements. This application directly verifies large amounts of data without sampling and without errors, effectively improving test quality and efficiency while reducing labor costs.
[0038] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description
[0039] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:
[0040] Figure 1A flowchart illustrating a method for verifying data processing quality according to an embodiment of this application is shown schematically.
[0041] Figure 2 This schematic diagram illustrates a structural block diagram of an apparatus for verifying the quality of data processing according to an embodiment of this application;
[0042] Figure 3 The diagram illustrates the internal structure of a computer device according to an embodiment of this application. Detailed Implementation
[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0044] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.
[0045] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.
[0046] Figure 1 A flowchart illustrating a method for verifying data processing quality according to an embodiment of this application is shown schematically. Figure 1 As shown in the embodiments of this application, a method for verifying data processing quality is provided, applied to a system for verifying data processing quality. The system includes a device for verifying data processing quality, multiple databases, and a test terminal. The device for verifying data processing quality communicates with the multiple databases and the test terminal respectively. The method may include the following steps:
[0047] Step 102: Obtain business requirements;
[0048] Step 104: Write the business requirements into the interactive computing environment;
[0049] Step 106: Connect the target database and the interactive computing environment across multiple databases;
[0050] Step 108: Based on the business requirements in the interactive computing environment, determine at least one processing verification result of the data in the target database;
[0051] Step 110: Send at least one processing verification result to the test terminal so that the test terminal can determine whether the data meets the data quality requirements.
[0052] During the design of views, large amounts of data undergo cleaning, collection, and storage processes before being stored in new database tables and statistically displayed through views. This necessitates testers verifying the quality of the processed data during testing. In existing technologies, when testing large amounts of data, traditional functional testing methods primarily rely on manual sampling, which involves selecting a subset of data for testing and verification. Manual sampling is prone to significant sampling variability and strong subjectivity; the number and method of sampling directly impact the test results, leading to vastly different outcomes. Therefore, this application proposes a method for verifying data processing quality that eliminates the need for sampling. It directly connects to the database to analyze and verify large amounts of data, avoiding inaccurate test results caused by sampling and solving the problem of testers being unable to verify large datasets. Simultaneously, it effectively improves testing efficiency, ensures test quality, and reduces labor costs.
[0053] In this embodiment, Python is primarily used to establish a connection between the interactive computing environment and the database. Business requirements are then written into the interactive computing environment, which is executed to directly determine whether the data in the database meets the business requirements. The determination result is then marked and printed out, achieving customized output, and returned to the test terminal. Preferably, the interactive computing environment can be JupyterLab. JupyterLab is web-based, supports data analysis and visualization, offers an interactive environment supporting over forty languages, and has a flexible user-configurable interface. Specifically, the device for verifying data processing quality first obtains the business requirements, which are the specific requirements of the actual project, and then writes them into JupyterLab. Next, it identifies the database containing the data to be tested from multiple databases, designating it as the target database, and then connects the target database to JupyterLab. While different Jupyter codes are used to connect different databases or tables, the methods are similar; only slight modifications to the Jupyter code are needed to connect different databases or tables. After connecting the target database to JupyterLab, the data in the target database can be quality verified based on the business requirements in JupyterLab. In practical applications, business requirements are first defined as specific needs, then these specific needs are broken down into verification function points, and finally, these verification function points are converted into Jupyter verification logic. JupyterLab can then be executed directly to obtain judgment results, which are used to verify the quality of the data. It should be noted that the system for verifying data processing quality in this embodiment supports multiple verification logics simultaneously. Therefore, for any data, there is at least one processing verification result. For any data, when there are multiple verification logics, the system will run them sequentially from top to bottom, and then print the result of each verification logic. Automatic judgment replaces manual sampling testing, performing logical judgments on all data. The next verification logic is executed only after the previous one has been completed. After execution, the results are automatically saved and can be repeated. The result of each verification logic is based on the preset verification logic, which includes the result output format. In other words, the system returns the result to the test terminal according to the pre-defined verification logic.
[0054] The above technical solution first obtains business requirements, writes them into an interactive computing environment, then connects a target database from multiple databases to the interactive computing environment, and based on the business requirements in the interactive computing environment, determines at least one processing verification result for the data in the target database. Finally, at least one processing verification result is sent to a test terminal so that the test terminal can determine whether the data meets the data quality requirements. This application directly verifies large amounts of data without sampling and without errors, effectively improving test quality and efficiency while reducing labor costs.
[0055] In this embodiment of the application, step 106, connecting the target database and the interactive computing environment, may include:
[0056] Set up the Python environment and install the interactive computing environment dependencies;
[0057] Execute the target command in the Python environment to open the interactive computing environment;
[0058] Import the target database into the interactive computing environment to connect to the target database.
[0059] In this embodiment, JupyterLab is used as the interactive computing environment. The device for verifying data processing quality first sets up a Python environment and installs the JupyterLab dependency package. Then, the target command `jupyter notebook` is executed within the Python environment to open JupyterLab. After opening JupyterLab, the target database to be tested can be imported and connected. The target database can be MongoDB, PostgreSQL, or MySQL, etc. The database connected depends on the needs. Different Jupyter codes are used to connect to different databases or tables, but the methods are similar; only slight modifications to the Jupyter code are required to connect to different databases or tables. After connecting to the database, the data can be edited and queried for verification.
[0060] In this embodiment of the application, step 108, determining at least one processing verification result of the data in the target database based on the business requirements in the interactive computing environment, may include:
[0061] Multiple verification function points are determined based on business requirements;
[0062] Convert multiple verification function points into multiple data processing logics;
[0063] Perform at least one data processing logic on the data in the target database to determine at least one processing verification result of the data in the target database.
[0064] Specifically, the business requirements are first defined as specific needs, then broken down into multiple verification function points, and finally converted into multiple data processing logics, i.e., Jupyter verification logics. JupyterLab can then be directly executed to obtain judgment results, thereby verifying data quality based on these results. It should be noted that the system for verifying data processing quality in this embodiment supports multiple verification logics simultaneously; therefore, for any data, there is at least one processing verification result. For any data, when there are multiple verification logics, the system will run them sequentially from top to bottom, and then print the result of each verification logic. Automatic judgment replaces manual sampling testing, performing logical judgments on all data. The next verification logic is executed only after the previous one has been completed. After execution, the results are automatically saved and can be repeated. The result of each verification logic is based on the preset verification logic, which includes the result output format. In other words, the system returns the result to the test terminal according to the pre-defined verification logic.
[0065] In this embodiment of the application, the data processing logic may include at least one of the following:
[0066] Regular expression checks, quantity checks, enumeration checks, and business logic checks.
[0067] Specifically, regular expression checks validate the data format, such as whether it is purely English, purely Chinese, or a string starting with "CH". Quantity checks compare data from different databases, such as MySQL and MongoDB, to count data that meets certain conditions. Enumeration checks validate data that satisfies enumeration fields. Business logic checks validate data based on actual business needs; for example, if the business requires that both the handler and the processing group cannot be null, then validation logic is added to ensure that both the handler and the processing group are not null, and to check if `sum()` is 0. It should be noted that the Jupyter validation logic in this embodiment includes not only regular expression checks, quantity checks, enumeration checks, and business logic checks, but also other validation logic based on business requirements.
[0068] In this application embodiment, performing at least one data processing logic on data in the target database to determine at least one processing verification result of the data in the target database may include:
[0069] Acquire data query methods and multiple data processing logics;
[0070] Execute at least one data processing logic based on the data query method;
[0071] The query methods include row-based query, column-based query, whole-table query, direct query, and encapsulated function query.
[0072] In this embodiment, after connecting the target database to JupyterLab, the data can be edited and queried for verification. Specifically, after connecting to the test database or table, the system first obtains the verification logic and query method. The query method can be a row-by-row query, a column-by-column query, or a full table query. When querying fields, verification can be performed directly or by defining a function module. In practical applications, simpler verification logic can be directly queried for verification, while more complex or frequently used verification logic can be encapsulated into functions for query verification. In one example, it is necessary to determine whether there are null or None values in the table. Since null value judgment is frequently used, a function can be used. The function defined using an enumeration type is as follows:
[0073] The function name is defined as: def chkNull(data):
[0074] Judgment logic: Mark null values as 1 and non-null values as 0. By reading the fields in the table, count the values that satisfy the null value of 1, and calculate the total number of null values for each field.
[0075] Output logic:
[0076] Use loc[:,:].applymap(chkNUull).sum() to perform summation on all empty fields in the table and then print the results.
[0077] In this embodiment of the application, the method may further include:
[0078] For any data, if at least one processing verification result does not meet the preset result, new data processing logic is added through the test terminal.
[0079] Execute new data processing logic on arbitrary data to obtain new processing verification results;
[0080] The new processing verification results are sent to the test terminal so that the test terminal can determine whether any data meets the data quality requirements.
[0081] In this embodiment, for any data, when multiple verification logics exist, the result will be returned to the test terminal regardless of whether the verification logic is satisfied or not. The test terminal determines whether the data meets the processing quality requirements based on the returned result and the actual needs. For any data, if at least one processing verification result does not meet the preset result, the verification logic that does not meet the preset result can be modified or adjusted through the test terminal, or new verification logic can be added to further verify the processing quality of the data.
[0082] The above technical solution first obtains business requirements, writes them into an interactive computing environment, then connects a target database from multiple databases to the interactive computing environment, and based on the business requirements in the interactive computing environment, determines at least one processing verification result for the data in the target database. Finally, at least one processing verification result is sent to a test terminal so that the test terminal can determine whether the data meets the data quality requirements. This application directly verifies large amounts of data without sampling and without errors, effectively improving test quality and efficiency while reducing labor costs.
[0083] It should be understood that although the steps in the flowchart above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0084] Figure 2 This schematically illustrates a structural block diagram of an apparatus for verifying the quality of data processing according to an embodiment of this application. Figure 2 As shown in the figure, this application provides an apparatus for verifying data processing quality, which may include:
[0085] The memory is configured to store instructions; and
[0086] The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the method for verifying data processing quality as described above.
[0087] Specifically, in one embodiment of this application, the processor 220 can be configured to:
[0088] Obtain business requirements;
[0089] Write business requirements into the interactive computing environment;
[0090] Connects a target database across multiple databases with an interactive computing environment;
[0091] Based on the business requirements in the interactive computing environment, determine at least one processing verification result of the data in the target database;
[0092] At least one processing verification result is sent to the test terminal so that the test terminal can determine whether the data meets the data quality requirements.
[0093] Furthermore, the processor 220 can also be configured to:
[0094] Multiple verification function points are determined based on business requirements;
[0095] Convert multiple verification function points into multiple data processing logics;
[0096] Perform at least one data processing logic on the data in the target database to determine at least one processing verification result of the data in the target database.
[0097] Furthermore, the processor 220 can also be configured to:
[0098] Acquire data query methods and multiple data processing logics;
[0099] Execute at least one data processing logic based on the data query method;
[0100] The query methods include row-based query, column-based query, whole-table query, direct query, and encapsulated function query.
[0101] In this embodiment of the application, the data processing logic includes at least one of the following:
[0102] Regular expression checks, quantity checks, enumeration checks, and business logic checks.
[0103] Furthermore, the processor 220 can also be configured to:
[0104] For any data, if at least one processing verification result does not meet the preset result, new data processing logic is added through the test terminal.
[0105] Execute new data processing logic on arbitrary data to obtain new processing verification results;
[0106] The new processing verification results are sent to the test terminal so that the test terminal can determine whether any data meets the data quality requirements.
[0107] Furthermore, the processor 220 can also be configured to:
[0108] Set up the Python environment and install the interactive computing environment dependencies;
[0109] Execute the target command in the Python environment to open the interactive computing environment;
[0110] Import the target database into the interactive computing environment to connect to the target database.
[0111] The above technical solution first obtains business requirements, writes them into an interactive computing environment, then connects a target database from multiple databases to the interactive computing environment, and based on the business requirements in the interactive computing environment, determines at least one processing verification result for the data in the target database. Finally, at least one processing verification result is sent to a test terminal so that the test terminal can determine whether the data meets the data quality requirements. This application directly verifies large amounts of data without sampling and without errors, effectively improving test quality and efficiency while reducing labor costs.
[0112] This application provides a system for verifying data processing quality, the system may include:
[0113] A device for verifying the quality of data processing;
[0114] Multiple databases, communicating with a device for verifying data processing quality, are configured to store processed data; and
[0115] The test terminal communicates with the device for verifying the quality of data processing and is configured to receive the verification results and determine whether the data meets the data quality requirements.
[0116] In this embodiment, primarily based on Python, a device for verifying data processing quality establishes a connection between an interactive computing environment and a target database across multiple databases. Business requirements are then written into the interactive computing environment, which directly determines whether the data in the target database meets these requirements. The determination result is then marked and printed out, achieving customized output, and returned to a test terminal. The test terminal uses the returned result in conjunction with actual requirements to determine whether the data meets processing quality requirements. For any data, if at least one processing verification result does not meet a preset result, the test terminal can modify or adjust the verification logic that does not meet the preset result, or add new verification logic to further verify the data's processing quality.
[0117] This application also provides a storage medium storing a program thereon, which, when executed by a processor, implements the above-described method for verifying data processing quality.
[0118] Figure 3 A schematic diagram illustrating the internal structure of a computer device according to an embodiment of this application is provided. Figure 3 As shown, in one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows. Figure 3As shown, the computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05, and a memory (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A06. The network interface A02 is used for communication with external terminals via a network connection. When the computer program is executed by the processor A01, it implements a method for verifying data processing quality. The display screen A04 can be a liquid crystal display (LCD) or an e-ink display. The input device A05 can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.
[0119] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0120] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: obtaining business requirements; writing the business requirements into an interactive computing environment; connecting a target database in multiple databases to the interactive computing environment; determining at least one processing verification result of data in the target database based on the business requirements in the interactive computing environment; and sending at least one processing verification result to a test terminal so that the test terminal can determine whether the data meets the data quality requirements.
[0121] In one embodiment, multiple verification function points are determined according to business requirements; the multiple verification function points are converted into multiple data processing logics; at least one data processing logic is executed on the data in the target database to determine at least one processing verification result of the data in the target database.
[0122] In one embodiment, a data query method and multiple data processing logics are obtained; at least one data processing logic is executed according to the data query method; wherein the query method includes row query, column query, whole table query, direct query, and encapsulated function query.
[0123] In one embodiment of this application, the data processing logic includes at least one of the following: regular expression checking, quantity checking, enumeration checking, and business logic checking.
[0124] In one embodiment, for any data, if at least one processing verification result does not meet the preset result, new data processing logic is added through the test terminal; the new data processing logic is executed on the arbitrary data to obtain a new processing verification result; the new processing verification result is sent to the test terminal so that the test terminal can determine whether the arbitrary data meets the data quality requirements.
[0125] In one embodiment, a Python environment is set up and interactive computing environment dependencies are installed; the target command is executed in the Python environment to open the interactive computing environment; and the target database is imported into the interactive computing environment to connect to the target database.
[0126] The above technical solution first obtains business requirements, writes them into an interactive computing environment, then connects a target database from multiple databases to the interactive computing environment, and based on the business requirements in the interactive computing environment, determines at least one processing verification result for the data in the target database. Finally, at least one processing verification result is sent to a test terminal so that the test terminal can determine whether the data meets the data quality requirements. This application directly verifies large amounts of data without sampling and without errors, effectively improving test quality and efficiency while reducing labor costs.
[0127] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0128] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0131] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0132] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0133] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0134] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0135] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A method for verifying data processing quality, characterized in that, A system for verifying the quality of data processing, the system comprising a device for verifying the quality of data processing, multiple databases, and a test terminal, wherein the device for verifying the quality of data processing communicates with the multiple databases and the test terminal respectively, and the method comprising: Obtain business requirements; Write the aforementioned business requirements into the interactive computing environment; Connect the target database and the interactive computing environment within the multiple databases; Based on the business requirements in the interactive computing environment, determine at least one processing verification result of the data in the target database; The at least one processing verification result is sent to the test terminal so that the test terminal can determine whether the data meets the data quality requirements; Based on the business requirements in the interactive computing environment, determining at least one processing verification result of the data in the target database includes: Based on the aforementioned business requirements, multiple verification function points are determined; The multiple verification function points are converted into multiple data processing logics; Perform at least one data processing logic on the data in the target database to determine at least one processing verification result of the data in the target database; The step of performing at least one data processing logic on the data in the target database to determine at least one processing verification result of the data in the target database includes: Obtain the data query method and the multiple data processing logics; At least one data processing logic is executed according to the data query method described above; The query methods include row-based query, column-based query, whole-table query, direct query, and encapsulated function query.
2. The method according to claim 1, characterized in that, The data processing logic includes at least one of the following: Regular expression checks, quantity checks, enumeration checks, and business logic checks.
3. The method according to claim 1, characterized in that, The method further includes: For any data, if at least one processing verification result does not meet the preset result, new data processing logic is added through the test terminal. The new data processing logic is executed on the arbitrary data to obtain a new processing verification result; The new processing verification result is sent to the test terminal so that the test terminal can determine whether the arbitrary data meets the data quality requirements.
4. The method according to claim 1, characterized in that, The target database and interactive computing environment connecting the multiple databases include: Set up the Python environment and install the dependencies for the interactive computing environment; Execute the target command in the Python environment to open the interactive computing environment; Import the target database into the interactive computing environment to connect to the target database.
5. An apparatus for verifying the quality of data processing, characterized in that, The device includes: The memory is configured to store instructions; and A processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the method for verifying the quality of data processing according to any one of claims 1 to 4.
6. A system for verifying the quality of data processing, characterized in that, include: The apparatus for verifying data processing quality according to claim 5; Multiple databases, communicating with the device for verifying the quality of data processing, are configured to store the processed data; as well as The test terminal communicates with the device for verifying the quality of data processing and is configured to receive the processing verification results and determine whether the data meets the data quality requirements.
7. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform a method for verifying the quality of data processing according to any one of claims 1 to 4.
8. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the method for verifying the quality of data processing according to any one of claims 1 to 4.
Citation Information
Patent Citations
Database interface conformance test method and device
CN110764997A
Non-Intrusive Measurement of Content Quality Using Dry Runs with Roll-back
US20110321012A1