Data quality testing method and apparatus, computer device, and storage medium

Abstract

The present disclosure relates to the technical field of data processing, and discloses a data quality testing method and apparatus, a computer device, and a storage medium. The method comprises: obtaining data under test for a target service; in response to a setting operation for testing parameters of the data under test, determining testing information of the data under test, the testing parameters being used for representing a data standard of the target service; in response to a mapping setting operation for the data under test, determining mapping information corresponding to the data under test, the mapping information associating the data under test with the category of the data under test; and, on the basis of the testing parameters and the mapping information, constructing a testing task for the data under test, running the testing task, and generating a data quality testing result corresponding to the testing task.

Description

Data quality detection method, device, computer equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese invention patent application with application number 202311029186.9, entitled “Data quality detection method, device, computer equipment and storage medium” and filing date August 15, 2023, and incorporates the entirety of that application into this document by reference. Technical Field

[0003] The present disclosure relates to the field of data processing technology, and in particular to a data quality detection method, apparatus, computer equipment, and storage medium. Background Art

[0004] With the development of Internet technology and computer technology, the generation of data is increasing rapidly. How to better implement data management to ensure the accuracy and availability of various business data has become a technical problem that needs to be solved urgently. At present, the common method is to develop different management platforms for different businesses to manage data through different management platforms. However, each data management platform can only display and store the entered data, and it is difficult to detect the accuracy of the entered data, that is, it is difficult to ensure data quality. For data that needs to be verified, it is usually necessary to export the data table for manual verification or set simple verification rules for automatic data verification. However, its accuracy is difficult to guarantee. Manual verification is time-consuming and labor-intensive, and its accuracy and timeliness are difficult to guarantee.

[0005] Summary of the Invention

[0006] In view of this, the embodiments of the present disclosure provide a data quality detection method, apparatus, computer equipment and storage medium to solve the problem of difficulty in ensuring the timeliness and quality of data detection.

[0007] In a first aspect, an embodiment of the present disclosure provides a data quality detection method, comprising: obtaining data to be detected of a target business; determining detection information of the data to be detected in response to a setting operation of detection parameters for the data to be detected, wherein the detection parameters are used to characterize the data standard of the target business; determining mapping information corresponding to the data to be detected in response to a mapping setting operation for the data to be detected, wherein the mapping information associates the data to be detected with a category of the data to be detected; constructing a detection task for the data to be detected based on the detection parameters and the mapping information, and running the detection task to generate a data quality detection result corresponding to the detection task.

[0008] The data quality detection method provided in this embodiment automatically constructs a detection task for the data to be detected by setting detection parameters and mapping parameters for the data to be detected, reducing the time required to construct the detection task and improving the timeliness of data detection. Furthermore, the combination of detection parameters and mapping parameters enables standard detection of the data to be detected, improving the quality of data detection. Maintenance personnel only need to specify the relevant detection parameters for the data to be detected, reducing their workload and facilitating simple and effective data management.

[0009] In the second aspect, an embodiment of the present disclosure provides a data quality detection device, including: a data acquisition module for acquiring the data to be detected of the target business; a detection parameter setting module for determining the detection information of the data to be detected in response to the setting operation of the detection parameters for the data to be detected, the detection parameters are used to characterize the data standards of the target business; a mapping parameter setting module for determining the mapping information corresponding to the data to be detected in response to the mapping setting operation for the data to be detected, the mapping information associates the data to be detected with the category of the data to be detected; a data detection module for constructing a detection task for the data to be detected based on the detection parameters and the mapping information, and running the detection task to generate a data quality detection result corresponding to the detection task.

[0010] In a third aspect, an embodiment of the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the data quality detection method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0011] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data quality detection method of the first aspect or any corresponding embodiment thereof.

[0012] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, comprising instructions, which, when executed by a processor, implement the data quality detection method of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] FIG1 is a flow chart of a data quality detection method according to some embodiments of the present disclosure;

[0015] FIG2 is a schematic diagram of setting basic attribute parameters according to some embodiments of the present disclosure;

[0016] FIG3 is a flow chart of another data quality detection method according to some embodiments of the present disclosure;

[0017] FIG4 is a schematic diagram of basic attribute information of an information display interface according to some embodiments of the present disclosure;

[0018] FIG5 is a schematic diagram of a reference code modification pop-up window according to some embodiments of the present disclosure;

[0019] FIG6 is a schematic diagram of setting mapping parameters according to some embodiments of the present disclosure;

[0020] FIG7 is a schematic diagram of mapping information of an information display interface according to some embodiments of the present disclosure;

[0021] FIG8 is a flow chart of another data quality detection method according to some embodiments of the present disclosure;

[0022] FIG9 is a schematic diagram showing data quality detection results according to some embodiments of the present disclosure;

[0023] FIG10 is a schematic diagram showing a data statistics interface according to some embodiments of the present disclosure;

[0024] FIG11 is a structural block diagram of a data quality detection device according to an embodiment of the present disclosure;

[0025] FIG12 is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present disclosure.

[0027] Different business data can be managed through different management platforms. However, each data management platform can only display and store input data, making it difficult to verify the accuracy of the input data, that is, to ensure data quality. For example, e-commerce product input is usually manually verified after data entry is completed, which is not only time-consuming and labor-intensive, but also difficult to ensure accuracy and timeliness.

[0028] Based on this, the technical solution disclosed in the present invention combines detection parameters and mapping parameters to realize the automatic construction of detection tasks, thereby realizing standard detection for the data to be detected, improving the quality and timeliness of data detection, and facilitating simple and effective data management.

[0029] According to an embodiment of the present disclosure, an embodiment of a data quality detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] In this embodiment, a data quality detection method is provided that can be used on a computer device, such as a computer or server, wherein a data platform is deployed on the computer device to manage the business data of each business line. FIG1 is a flow chart of the data quality detection method according to an embodiment of the present disclosure. As shown in FIG1 , the flow chart includes the following steps:

[0031] Step S101: Acquire the data to be detected of the target service.

[0032] The target business is the business line currently using the data platform for data analysis. This target business can be e-commerce, video, audio, etc. The data to be tested is the business data corresponding to the target business. Specifically, for e-commerce, the data to be tested can be product entry data; for video, the data to be tested can be video views; for audio, the data to be tested can be audio playback volume, etc.

[0033] In some optional implementations, the data to be detected may be entered by the user through the data entry interface of the data platform. Accordingly, the data platform may respond to the user's data entry operation and obtain the data to be detected corresponding to the entry operation.

[0034] In some optional implementations, the data to be tested may also be transmitted to the data platform for the business line corresponding to the target business. Correspondingly, the data platform may receive the data to be tested transmitted by the business line.

[0035] Of course, the data to be detected may also be obtained in other ways. The method for obtaining the data to be detected is not specifically limited here, and those skilled in the art may set it according to actual needs.

[0036] Step S102 : In response to the setting operation of the detection parameters of the data to be detected, the detection information of the data to be detected is determined.

[0037] The detection parameters are used to characterize the data standards of the target business. Different target businesses have corresponding data standards, and the data standards are used to verify whether their business data meets the standards.

[0038] Specifically, as shown in Figure 2, the data platform has a parameter setting interface, which includes an operation control display area and a parameter setting area. A detection parameter setting control is set in the operation control display area. The technician triggers the detection parameter setting control, and the parameter setting area displays the detection parameter setting interface, so that the technician can set the detection information related to the data to be detected in the detection parameter setting interface.

[0039] Accordingly, the data platform can respond to the technician's triggering operation on the detection parameter setting control and the setting operation of the detection parameters of the data to be detected, and generate corresponding detection information on the detection parameter setting interface.

[0040] Step S103 : In response to the mapping setting operation for the data to be detected, determining mapping information corresponding to the data to be detected.

[0041] The mapping information associates the data to be detected with the category of the data to be detected, and the category of the data to be detected indicates the classification of the data to be detected.

[0042] Specifically, as shown in Figure 2, the data platform's parameter setting interface includes a mapping setting control within the control display area. When a technician triggers the mapping setting control, the parameter setting area displays the corresponding mapping setting interface. Within this interface, the technician sets the mapping information for the database, table, partition, and other mappings corresponding to the data to be tested. This mapping information allows the technician to map the data to the corresponding database, table, partition, and queue for data testing during the execution of a testing task.

[0043] Accordingly, the data platform can respond to the technician's triggering operation on the mapping setting control and the mapping setting operation, and generate corresponding mapping information on the mapping setting interface.

[0044] Step S104: constructing a detection task for the data to be detected based on the detection parameters and the mapping information, and running the detection task to generate a data quality detection result corresponding to the detection task.

[0045] The data standard for the data to be tested is determined by combining the test parameters, and the corresponding library, table, and partition are determined based on the mapping information. This allows the corresponding category information of the data to be tested to be determined. Based on the test parameters and mapping information, a test task for the data to be tested is automatically constructed to determine the queue for executing the test task. Furthermore, a timestamp is added when the test task is constructed, allowing for subsequent verification of the build time of the test task.

[0046] After completing the construction of the detection task, the detection task is automatically executed to perform data detection on the data to be detected according to the data standard and mapping information to determine whether the data to be detected meets the standards, and generate corresponding data quality detection results and display them in the data platform for technical personnel to view.

[0047] The data quality detection method provided in this embodiment automatically constructs a detection task for the data to be detected by setting detection parameters and mapping parameters for the data to be detected, reducing the time required to construct the detection task and improving the timeliness of data detection. Furthermore, the combination of detection parameters and mapping parameters enables standard detection of the data to be detected, improving the quality of data detection. Maintenance personnel only need to specify the relevant detection parameters for the data to be detected, reducing their workload and facilitating simple and effective data management.

[0048] In this embodiment, a data quality detection method is provided, which can be used in computer devices, such as computers and servers. FIG3 is a flow chart of the data quality detection method according to an embodiment of the present disclosure. As shown in FIG3 , the flow chart includes the following steps:

[0049] Step S201: Obtain the target service's to-be-detected data. Detailed descriptions refer to the corresponding descriptions of the above embodiments, which will not be repeated here.

[0050] Step S202 : In response to the setting operation of detection parameters for the data to be detected, detection information of the data to be detected is determined, where the detection parameters are used to characterize the data standard of the target service.

[0051] Specifically, the detection information includes basic attribute information and quality attribute information, wherein the basic attribute information represents the basic information of the data to be detected, and the quality attribute information represents the standard information of the data to be detected.

[0052] Accordingly, the above step S202 may include:

[0053] Step S2021: In response to the editing operation on the basic configuration parameters, basic attribute information of the data to be detected is determined.

[0054] Basic configuration parameters represent the fundamental parameters of the data to be tested, including its classification, standard number, Chinese name, English name, data type, and business definition. Based on the characteristics of the target business, technicians set the basic configuration parameters for the data to be tested on the test parameter setup interface. The data platform responds to these edits by generating and displaying the corresponding basic attribute information on the test parameter setup interface.

[0055] Step S2022: In response to the editing operation on the quality configuration parameters, determine the quality attribute information of the data to be detected.

[0056] Quality configuration parameters are used to characterize whether the data to be tested meets the data standards of the target business, specifically including data length and data range.

[0057] Among them, the data range includes multiple fields such as numerical type, reference code and valid value. The numerical type is mainly for the string type, and its setting has the corresponding judgment operator and range value setting, as shown in Figure 2. The data length has a preset length range (for example, 1 to 1000). If the technician has not edited the data length, it means that there is no limit on the length of the character. The reference code is a reference code maintained by the standard domain of the target business. The reference code is used to represent the standard code identifier corresponding to the data to be tested. The valid value is a valid enumeration value allowed for the data to be tested. The valid value can be set within a preset number (for example, 50) and separated by a separator (for example, a Western comma), as shown in Figure 2.

[0058] The technicians set the quality configuration parameters of the data to be tested in the detection parameter setting interface according to the characteristics of the target business. Accordingly, the data platform can respond to the technicians' editing operations on the quality configuration parameters and generate and display the corresponding quality attribute information in the detection parameter setting interface.

[0059] After completing the setting of basic configuration parameters and quality configuration parameters, the technician can trigger the confirmation control of the detection parameter setting interface, switch to the information display interface of the data to be detected, and display the basic attribute information and quality attribute information on the information display interface, as shown in Figure 4.

[0060] In some optional implementations, after determining the quality attribute information of the data to be tested in response to the editing operation on the quality configuration parameters, the method may further include:

[0061] Step a1: In response to a modification operation on a quality configuration parameter of the data to be detected, detecting whether the data to be detected is bound with mapping information based on the modification operation.

[0062] Step a2: When the data to be detected is bound with mapping information, the modified target quality configuration parameters are obtained.

[0063] Step a3: Update the detection task of the data to be detected based on the target quality configuration parameters and mapping information.

[0064] The modification operation is triggered when there is a configuration error in the quality attribute information or when the quality attribute information needs to be updated according to the target business update. As shown in Figure 4, an edit control is set on the information display interface, and the modification operation for the quality configuration parameters is triggered through the edit control.

[0065] Before configuring the mapping information, that is, after completing the setting of basic configuration parameters and quality configuration parameters, when you determine that there is a configuration error by viewing the quality attribute information displayed on the information display interface, after triggering the modification operation, directly jump to the detection parameter setting interface to modify the quality configuration parameters.

[0066] After the mapping information is not configured, when it is determined that there is a configuration error in the quality attribute information or the quality attribute information needs to be updated according to the update of the target business, at this time, when the triggering modification operation is detected, the data platform will detect whether the data to be detected is bound with the corresponding mapping information. When the data to be detected is bound with the corresponding mapping information, the modified target quality configuration parameters are determined based on the modification operation, and the detection task for the data to be detected is rebuilt according to the target quality configuration parameters and the mapping information, so as to avoid detection errors caused by the failure to update the detection task, thereby improving the detection accuracy of the data.

[0067] In some optional embodiments, the quality configuration parameter includes a reference code, and the reference code is used to determine the identification information corresponding to the data to be tested. After determining the quality attribute information of the data to be tested in response to the editing operation on the quality configuration parameter, the above method may further include:

[0068] Step b1: In response to a modification operation on the reference code, generate prompt information based on the modification operation.

[0069] Step b2: in response to a confirmation operation performed on the prompt information, updating the reference code based on the confirmation operation.

[0070] Prompt information is used to inform technical personnel of the impact of modifying the reference code. When a modification to a reference code in a quality attribute parameter is detected, the data platform will display a corresponding prompt based on the modification operation. This prompt can be displayed in the form of a pop-up window or as a text reminder in the edit box of the reference code. The form of the prompt is not specifically limited here.

[0071] For example, when the data platform detects a modification to a reference code within a quality attribute parameter, it will display the pop-up window shown in Figure 5. This pop-up window includes a confirmation control. When the confirmation control is triggered, the data platform responds to the confirmation and, based on the confirmation, allows the technician to modify the reference code, thus updating the reference code. This prevents subsequent data testing from being affected by erroneous modifications to the reference code.

[0072] Step S203 : In response to the mapping setting operation for the data to be detected, determining mapping information corresponding to the data to be detected, wherein the mapping information associates the data to be detected with the category of the data to be detected.

[0073] Specifically, the above step S203 may include:

[0074] Step S2031: Detect whether there is any detection requirement for the data to be detected.

[0075] The detection requirement indicates whether the data to be tested requires data detection. After configuring the detection parameters for the data to be tested, further testing is performed to determine whether the data to be tested currently requires data detection. If data detection is required, the mapping parameters are set. If not, the data to be tested is displayed on the data platform.

[0076] In some optional implementations, the above step S2031 may include:

[0077] Step c1: Detect whether the detection control for the data to be detected is in an open state.

[0078] Step c2: when the detection control for the data to be detected is in an on state, it is determined that there is a detection requirement for the data to be detected.

[0079] The mapping settings interface includes a detection control, which has an on and off state. When the detection control is on, it indicates that the data to be detected has a detection requirement; when the detection control is off, it indicates that the data to be detected does not have a detection requirement. This allows for flexible adjustment of detection requirements, facilitating targeted setting of detection requirements for the data to be detected and reducing the detection of irrelevant data.

[0080] Step S2032: When there is a detection requirement for the data to be detected, a mapping parameter configuration interface is displayed or activated. The mapping parameter configuration interface includes a plurality of mapping parameters to be configured.

[0081] When there is a test requirement for the data to be tested, a mapping parameter configuration interface for the data to be tested will pop up on the mapping settings interface. Alternatively, the mapping parameter configuration interface is displayed on the mapping settings interface, but it can only be activated to configure mapping parameters when there is a test requirement for the data to be tested. That is, when there is no test requirement, the mapping parameters on the mapping parameter configuration interface are not editable.

[0082] The mapping parameter configuration interface includes a plurality of mapping parameters to be configured and their corresponding parameter editing boxes, as shown in FIG6 .

[0083] Specifically, mapping parameters can include library and table partition settings, mapping methods, data development projects, execution frequency, and queue settings. Library and table partition settings specify the library, table, and partition to which the data to be tested belongs. A library is equivalent to a primary classification, a table is equivalent to a secondary classification, and a partition is equivalent to a tertiary classification. The mapping method determines the mapping field. The data development project represents the project corresponding to the target business to which the data to be tested belongs. The execution frequency indicates the time interval between executing the testing task. The queue setting indicates the queue that matches the testing task.

[0084] Step S2033: In response to the configuration operation on the plurality of mapping parameters, mapping information corresponding to the data to be detected is generated based on the configuration operation.

[0085] The mapping parameters corresponding to the data to be tested are configured through the mapping parameter configuration interface. Accordingly, the data platform can generate corresponding mapping information in response to the technician's configuration operation on the mapping parameters, and can be viewed in the information display interface, as shown in Figure 7.

[0086] In an optional embodiment, the mapping parameter configuration interface is provided with an update control, and the technician can update the mapping parameters by triggering the update control.

[0087] In an optional embodiment, if it is necessary to unbind the data to be detected from its mapping information, the mapping display control can be triggered in the information display interface, and by triggering the mapping display control, the user can switch to the mapping display interface. As shown in Figure 7, the mapping display interface displays all the mapping information currently possessed by the data to be detected. Each mapping information has a corresponding unbinding control. By triggering the unbinding control, the binding of the data to be detected and the mapping information can be completed. Accordingly, the detection task corresponding to the unbound mapping information will also be deleted, that is, the data to be detected will no longer execute the detection task corresponding to the unbound mapping information.

[0088] Step S204: Based on the detection parameters and mapping information, a detection task for the data to be detected is constructed, and the detection task is executed to generate a data quality detection result corresponding to the detection task. Detailed descriptions refer to the corresponding descriptions of the above embodiments, which will not be repeated here.

[0089] The data quality detection method provided in this embodiment supports the editing of basic configuration parameters and quality configuration parameters, facilitating flexible adjustment based on actual usage scenarios, thereby supporting data quality detection in a variety of business scenarios. Mapping parameters are configured based on whether the current data to be detected requires data detection, avoiding full detection of the data to be detected. Data detection can be skipped for data to be detected that does not require detection, thus achieving targeted detection of the data to be detected, thereby reducing data detection time and improving data detection efficiency.

[0090] In this embodiment, a data quality detection method is provided, which can be used in computer devices, such as computers and servers. FIG8 is a flow chart of the data quality detection method according to an embodiment of the present disclosure. As shown in FIG8 , the flow chart includes the following steps:

[0091] Step S301: Obtain the target service's to-be-detected data. Detailed descriptions refer to the corresponding descriptions of the above embodiments, which will not be repeated here.

[0092] Step S302: In response to the setting operation of the detection parameters for the data to be detected, the detection information of the data to be detected is determined, and the detection parameters are used to characterize the data standard of the target service. Detailed descriptions refer to the corresponding descriptions of the above embodiments, which will not be repeated here.

[0093] Step S303: In response to the mapping setting operation for the data to be detected, the mapping information corresponding to the data to be detected is determined, and the mapping information associates the data to be detected with the category of the data to be detected. Detailed descriptions refer to the corresponding descriptions of the above embodiments, which will not be repeated here.

[0094] Step S304: constructing a detection task for the data to be detected based on the detection parameters and the mapping information, and running the detection task to generate a data quality detection result corresponding to the detection task.

[0095] Specifically, the above step S304 may include:

[0096] Step S3041: construct a detection task for the data to be detected based on the detection parameters and mapping information.

[0097] The detection parameters are used to determine the data standard corresponding to the data to be detected, and the mapping information is used to determine the detection library, table, partition and corresponding fields of the data to be detected. Therefore, according to the detection parameters and mapping information, the detection task for the data to be detected can be automatically constructed.

[0098] When generating a test task, determine the project to which the test task belongs, bind the responsible object corresponding to the test task, and name the test task in the format of standard domain-data standard-mapping information-timestamp, so that technicians can determine the test task corresponding to the data to be tested based on the test task name.

[0099] Step S3042: Obtain the scheduling information corresponding to the detection task.

[0100] Scheduling information indicates the execution queue corresponding to the detection task and the execution frequency associated with the execution queue. The execution queue is the queue bound in the mapping information and defaults to the public resource scheduling group. The execution frequency is the execution interval of the task bound in the mapping information. The execution frequency can be set by day, hour, week, month, etc., and is not specifically limited here.

[0101] The detection task carries corresponding scheduling information. After the detection task is generated, the corresponding scheduling information can be extracted from the detection task.

[0102] Step S3043: Schedule the detection task for execution according to the scheduling information to generate data quality detection results.

[0103] The detection task is scheduled to the corresponding execution queue according to the determined scheduling information, and the task is scheduled according to the execution frequency in the scheduling information. After the execution of the detection task is completed, the corresponding data quality detection result is generated.

[0104] Step S3044: When there is an abnormality in the data quality detection result, an alarm message is generated and sent to the target object.

[0105] The alarm message is a pre-set data anomaly warning. The alarm message can be sent to the target person via email, and the target person is the technician who set the inspection task. Of course, the alarm message can be sent via sound or text message, which is not specifically limited here and can be determined according to actual needs.

[0106] The data platform analyzes the data quality test results output by the detection task to determine whether there are any anomalies in the data quality test results. If there are anomalies in the data quality test results, an alarm message can be generated and sent to the corresponding target object.

[0107] In some optional embodiments, the above method may further include:

[0108] Step S305 , in response to a viewing operation on the data quality detection result, jump to a corresponding data detection interface based on the viewing operation.

[0109] Data quality test results can include both basic attribute information and quality attribute compliance results. The results display the execution results of each test task, as shown in Figure 9. When a technician views the execution results of a test task, the data platform responds by redirecting them to the corresponding data test interface. This facilitates intuitive understanding of the various test results for the data being tested.

[0110] In some optional implementations, when a technician clicks on a test task on the data quality test results display interface, the data platform can respond to the technician's triggering of the test task by jumping to the task link corresponding to the test task. This facilitates verification of the execution status of the test task and avoids abnormal data quality test results caused by the closure of the execution task.

[0111] In some optional embodiments, the method may further include: in response to a retest operation on the data to be tested, retesting the data to be tested and generating corresponding data quality test results, thereby further determining the credibility and accuracy of the data quality test results.

[0112] The display interface for data quality test results is equipped with a test control. If the test data needs to be retested or re-evaluated, the technician can trigger the test control. In response, the data platform can retest the data to generate the corresponding data quality test results. Of course, the data platform can also trigger the manual test process when it determines that the test control has been triggered, so that the technician can further verify the data to generate the corresponding data quality test results.

[0113] In some optional embodiments, the above method may further include:

[0114] Step d1, based on the data quality detection result, determine the first detection pass rate corresponding to the basic attribute information and the second detection pass rate corresponding to the quality attribute information.

[0115] In step d2, the first detection pass rate and the second detection pass rate are displayed in a data statistics interface corresponding to the data quality detection result.

[0116] The first pass rate is used to characterize the compliance rate of basic attribute information, and its value is the ratio of the current number of basic attribute information items that meet the standards to the total number of basic attribute information items; the second pass rate is used to characterize the compliance rate of quality attribute information, and its value is the ratio of the current number of quality attribute information items that meet the standards to the total number of quality attribute information items.

[0117] The data platform has a corresponding data statistics interface, as shown in Figure 10. After the test of the data to be tested is completed, the data standard, statistical information, first test pass rate and second test pass of the data to be tested are displayed in the data statistics interface for technical personnel to view.

[0118] The data quality detection method provided in this embodiment implements orderly execution of detection tasks by executing detection tasks according to scheduling information. When data quality detection results are abnormal, an alarm message is generated to alert relevant technical personnel to conduct data verification. By displaying data quality detection results in the data statistics interface, it is easy to intuitively understand the detection content and corresponding detection results of the data to be detected, facilitate targeted review of abnormal data, and facilitate timely data maintenance.

[0119] As one or more specific application embodiments of the embodiments of the present disclosure, the above-mentioned data quality detection method is described here in conjunction with the e-commerce usage scenario. E-commerce businesses have multiple categories of goods that require data entry in order to count the product information. The basic attribute information and quality attribute information of each category of goods are set in the data platform, wherein the basic attribute information may include the Chinese and English names of the goods, the category to which the goods belong (such as maternal and child categories, clothing categories, food categories, etc.), product codes, etc.; quality attribute information may include the price range of the goods, the age value applicable to the goods, the standard code corresponding to the goods, etc.

[0120] In order to be able to detect the product information entered subsequently, the mapping information of the product data is further configured to determine the category to which each product belongs, such as maternal and child category - sanitary products - tissue paper, where the maternal and child category corresponds to the library in the mapping information (i.e., the first-level classification), sanitary products correspond to the table in the mapping information (i.e., the second-level classification), and tissue paper corresponds to the partition in the mapping information (i.e., the third-level classification). Since the product information may be entered at any time, in order to ensure the accuracy of the product information entry, the detection frequency for the product information can be set, that is, the execution frequency in the mapping information. For the product information, the product data item corresponding to the product information can be determined to construct the detection task in the product data item, and the detection task of the product information can be scheduled to the corresponding project queue for execution.

[0121] After completing the entry of product information, the data platform performs a detection task, performing data detection on the entered product information to determine whether the entered product information meets the data standards. For product information that does not meet the data standards, an alarm reminder will be issued and displayed on the data statistics interface, so that technical personnel can further verify the non-compliant product information to determine whether the product information is incorrect or the detection task needs to be updated. This improves the accuracy of product information entry, ensures the quality of product data entry, and enhances the level of product information rectification.

[0122] This embodiment also provides a data quality detection device, which is used to implement the above-mentioned embodiments and preferred embodiments. Details that have already been described will not be repeated. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0123] This embodiment provides a data quality detection device, as shown in FIG11 , including:

[0124] The data acquisition module 401 is used to acquire the data to be detected of the target service.

[0125] The detection parameter setting module 402 is configured to determine detection information of the data to be detected in response to a setting operation of detection parameters for the data to be detected, where the detection parameters are used to characterize data standards of a target service.

[0126] The mapping parameter setting module 403 is configured to determine mapping information corresponding to the data to be detected in response to a mapping setting operation for the data to be detected, where the mapping information associates the data to be detected with a category of the data to be detected.

[0127] The data detection module 404 is used to construct a detection task for the data to be detected based on the detection parameters and mapping information, and run the detection task to generate a data quality detection result corresponding to the detection task.

[0128] In some optional embodiments, the detection information includes basic attribute information and quality attribute information. The detection parameter setting module 402 may include:

[0129] The basic parameter editing unit is used to determine the basic attribute information of the data to be detected in response to the editing operation of the basic configuration parameters.

[0130] The quality parameter unit is used to determine the quality attribute information of the data to be detected in response to the editing operation of the quality configuration parameters.

[0131] In some optional embodiments, the detection parameter setting module 402 may further include:

[0132] The first modifying unit is configured to, in response to a modification operation on a quality configuration parameter of the data to be detected, detect whether the data to be detected is bound with mapping information based on the modification operation.

[0133] The parameter acquisition unit is used to acquire the modified target quality configuration parameters when the data to be detected is bound with mapping information.

[0134] The first updating unit is configured to update the detection task of the to-be-detected data based on the target quality configuration parameter and the mapping information.

[0135] In some optional implementations, the quality configuration parameters include reference codes, and the detection parameter setting module 402 may further include:

[0136] The second modifying unit is configured to generate prompt information based on the modification operation in response to the modification operation on the reference code.

[0137] The second updating unit is configured to respond to a confirmation operation performed on the prompt information and update the reference code based on the confirmation operation.

[0138] In some optional implementations, the mapping parameter setting module 403 may include:

[0139] The detection unit is used to detect whether the data to be detected meets the detection requirements.

[0140] The display unit is used to display a mapping parameter configuration interface when there is a detection requirement for the data to be detected. The mapping parameter configuration interface includes multiple mapping parameters to be configured.

[0141] The parameter configuration unit is configured to respond to configuration operations on a plurality of mapping parameters and generate mapping information corresponding to the data to be detected based on the configuration operations.

[0142] In some optional embodiments, the detection unit is specifically used to: detect whether a detection control for the data to be detected is in an on state; when the detection control for the data to be detected is in an on state, determine that there is a detection requirement for the data to be detected.

[0143] In some optional implementations, the data detection module 404 may include:

[0144] The detection task construction unit is used to construct the detection task of the data to be detected based on the detection parameters and mapping information.

[0145] The scheduling information acquisition unit is used to obtain the scheduling information corresponding to the detection task.

[0146] The task execution unit is used to schedule and execute the detection tasks according to the scheduling information and generate data quality detection results.

[0147] The alarm unit is used to generate an alarm message when there is an abnormality in the data quality detection result, and send the alarm message to the target object.

[0148] In some optional embodiments, the above device may further include:

[0149] The result viewing module is used to respond to a viewing operation on the data quality detection result and jump to the corresponding data detection interface based on the viewing operation.

[0150] In some optional embodiments, the above device may further include:

[0151] The re-detection module is used to re-detect the data to be detected in response to the re-detection operation on the data to be detected, and generate corresponding data quality detection results.

[0152] In some optional embodiments, the above device may further include:

[0153] The pass rate determination module is used to determine a first test pass rate corresponding to the basic attribute information and a second test pass rate corresponding to the quality attribute information based on the data quality test results.

[0154] The display module is used to display the first detection pass rate and the second detection pass rate in the data statistics interface corresponding to the data quality detection result.

[0155] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0156] The data quality detection device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0157] The data quality detection device provided in this embodiment automatically constructs a detection task for the data to be detected by setting detection parameters and mapping parameters for the data to be detected, reducing the time required to construct the detection task and improving the timeliness of data detection. At the same time, the combination of detection parameters and mapping parameters enables standard detection of the data to be detected, improving the quality of data detection. Maintenance personnel only need to specify the relevant detection parameters for the data to be detected, reducing their workload and facilitating simple and effective data management.

[0158] The embodiment of the present disclosure further provides a computer device having the data quality detection device shown in FIG11 .

[0159] Please refer to Figure 12, which is a structural diagram of a computer device provided by an optional embodiment of the present disclosure. As shown in Figure 12, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 12 takes a processor 10 as an example.

[0160] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0161] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0162] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0163] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0164] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means, and FIG12 takes the bus connection as an example.

[0165] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0166] The computer device further includes a communication interface for the computer device to communicate with other devices or a communication network.

[0167] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0168] Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data quality detection method, comprising: Obtain the data to be tested of the target business; In response to a setting operation of a detection parameter for the data to be detected, determining detection information of the data to be detected, wherein the detection parameter is used to characterize a data standard of the target service; In response to a mapping setting operation for the data to be detected, determining mapping information corresponding to the data to be detected, the mapping information associating the data to be detected with a category of the data to be detected; Based on the detection parameters and the mapping information, a detection task for the data to be detected is constructed, and the detection task is run to generate a data quality detection result corresponding to the detection task.

2. The method according to claim 1, wherein the detection information includes basic attribute information and quality attribute information, and the determining the detection information of the data to be detected in response to the setting operation of the detection parameters of the data to be detected comprises: In response to an editing operation on the basic configuration parameters, determining basic attribute information of the data to be detected; In response to the editing operation on the quality configuration parameter, quality attribute information of the data to be detected is determined.

3. The method according to claim 2, wherein after determining the quality attribute information of the to-be-detected data in response to the editing operation on the quality configuration parameter, the method further comprises: In response to a modification operation on a quality configuration parameter of the data to be detected, detecting whether the data to be detected is bound with mapping information based on the modification operation; When the data to be detected is bound with mapping information, obtaining modified target quality configuration parameters; Based on the target quality configuration parameter and the mapping information, the detection task of the data to be detected is updated.

4. The method according to claim 2, wherein the quality configuration parameter includes a reference code, and the reference code is used to determine the identification information corresponding to the data to be detected; after determining the quality attribute information of the data to be detected in response to the editing operation of the quality configuration parameter, the method further includes: In response to a modification operation on the reference code, generating prompt information based on the modification operation; In response to a confirmation operation performed on the prompt information, the reference code is updated based on the confirmation operation.

5. The method according to claim 1, wherein the determining, in response to a mapping setting operation for the data to be detected, mapping information corresponding to the data to be detected comprises: Detecting whether there is a detection requirement for the data to be detected; When the data to be detected has the detection requirement, displaying or activating a mapping parameter configuration interface, the mapping parameter configuration interface including a plurality of mapping parameters to be configured; In response to configuration operations for the plurality of mapping parameters, the mapping information corresponding to the data to be detected is generated based on the configuration operations.

6. The method according to claim 5, wherein the detecting whether the data to be detected has a detection requirement comprises: Detecting whether a detection control for the data to be detected is in an open state; When the detection control for the data to be detected is in an on state, it is determined that there is a detection requirement for the data to be detected.

7. The method according to claim 1, wherein the running of the detection task and generating a data quality detection result corresponding to the detection task comprises: Obtaining scheduling information corresponding to the detection task; Scheduling and executing the detection task according to the scheduling information to generate the data quality detection result; When there is an abnormality in the data quality detection result, an alarm message is generated and sent to the target object.

8. The method according to claim 7, further comprising: In response to a viewing operation on the data quality detection result, jumping to a corresponding data detection interface based on the viewing operation; and / or, In response to the re-detection operation on the data to be detected, the data to be detected is re-detected to generate a corresponding data quality detection result.

9. The method according to claim 7, further comprising: Based on the data quality detection result, determining a first detection pass rate corresponding to the basic attribute information and a second detection pass rate corresponding to the quality attribute information; The first detection pass rate and the second detection pass rate are displayed in a data statistics interface corresponding to the data quality detection result.

10. A data quality detection device, comprising: A data acquisition module is used to acquire the data to be detected of the target business; A detection parameter setting module, configured to determine detection information of the data to be detected in response to a setting operation of a detection parameter for the data to be detected, wherein the detection parameter is used to characterize a data standard of the target service; a mapping parameter setting module, configured to determine, in response to a mapping setting operation for the data to be detected, mapping information corresponding to the data to be detected, wherein the mapping information associates the data to be detected with a category of the data to be detected; The data detection module is used to construct a detection task for the data to be detected based on the detection parameters and the mapping information, and to run the detection task to generate a data quality detection result corresponding to the detection task.

11. A computer device comprising: A memory and a processor, wherein the memory and the processor are connected to each other in communication, the memory stores computer instructions, and the processor executes the computer instructions to execute the rights The data quality detection method according to any one of claims 1 to 9.

12. A computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, wherein the computer instructions are used to enable a computer to execute the data quality detection method according to any one of claims 1 to 9.

13. A computer program product, comprising instructions, which, when executed by a processor, implement the data quality detection method according to any one of claims 1 to 9.

Images