Method and device for automatically generating label quality detection script, equipment and medium

Abstract

The invention relates to the technical field of data processing, and discloses a method and device for automatically generating a label quality detection script, equipment and a medium, and the method comprises the steps: obtaining a to-be-detected label list, and traversing the to-be-detected label list to confirm the data type of each to-be-detected label in the to-be-detected label list; and configuring a corresponding retrieval formula for each to-be-detected label according to the data type of the to-be-detected label so as to generate a label quality detection script. The detection efficiency can be improved, the timeliness requirement can be met, and the cost can be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing technology, and in particular to a method, device, equipment and medium for automatically generating a label quality detection script. Background Art

[0002] During the data development process, the data quality of the label will affect the reliability of downstream applications. Therefore, the label needs to be quality tested before it is officially launched. However, the data types of labels are relatively complex, usually including numbers, text, time, numbers, enumerations, etc., which leads to large differences in the quality testing of labels. For example, for digital data, the maximum, minimum and average values ​​need to be tested, for text data, the maximum and minimum lengths need to be tested, and for time data, the date and format legitimacy need to be tested. At present, the commonly used detection method is manual splicing through EXCEL tables.

[0003] The inventors realized that the above detection method is time-consuming and requires a lot of manpower, resulting in high costs. At the same time, it cannot meet the timeliness requirements. Summary of the invention

[0004] The present invention provides a method, device, computer equipment and medium for automatically generating label quality detection scripts by artificial intelligence to solve the technical problem that the cost of quality detection of labels is high and the timeliness requirements cannot be met.

[0005] In a first aspect, a method for automatically generating a tag quality detection script is provided, comprising:

[0006] Obtain a list of tags to be tested, and traverse the list of tags to be tested to confirm the data type of each tag to be tested in the list of tags to be tested;

[0007] According to the data type of the tag to be tested, a corresponding search formula is configured for each tag to be tested to generate a tag quality detection script.

[0008] In a second aspect, a device for automatically generating a label quality detection script is provided, comprising:

[0009] An acquisition module is used to acquire a list of tags to be tested, and traverse the list of tags to be tested to confirm the data type of each tag to be tested in the list of tags to be tested;

[0010] The generating module is used to configure a corresponding search formula for each tag to be tested according to the data type of the tag to be tested so as to generate a tag quality detection script.

[0011] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for automatically generating a label quality detection script when executing the computer program.

[0012] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for automatically generating a label quality detection script are implemented.

[0013] In the scheme implemented by the above-mentioned method, device, computer equipment and storage medium for automatically generating label quality detection scripts, a list of labels to be tested can be obtained and the list of labels to be tested can be traversed to confirm the labels to be tested and the data types of the labels to be tested, and then a corresponding search formula is configured for each label to be tested according to the data type of the labels to be tested. That is, one label to be tested corresponds to one search formula, and the search formula is used to perform quality search for the corresponding label to be tested. Multiple search formulas constitute a label quality detection script, so that quality detection can be performed for labels to be tested of different data types without manual inspection, which reduces costs, and at the same time improves retrieval efficiency, which can meet timeliness requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0015] Figure 1 It is a flow chart of a method for automatically generating a label quality detection script in one embodiment of the present invention;

[0016] Figure 2 This is a schematic diagram of a first sub-process of a method for automatically generating a label quality detection script in one embodiment of the present invention;

[0017] Figure 3 is a schematic diagram of a second sub-process of a method for automatically generating a label quality detection script in one embodiment of the present invention;

[0018] Figure 4 yes Figure 1 A schematic flow chart of a specific implementation of step S10;

[0019] Figure 5 is a schematic diagram of a third sub-process of a method for automatically generating a label quality detection script in one embodiment of the present invention;

[0020] Figure 6is a schematic diagram of a fourth sub-process of a method for automatically generating a label quality detection script in an embodiment of the present invention;

[0021] Figure 7 is a schematic diagram of a fifth sub-process of a method for automatically generating a label quality detection script in one embodiment of the present invention;

[0022] Figure 8 It is a structural schematic diagram of a device for automatically generating a label quality detection script in one embodiment of the present invention;

[0023] Fig. 9 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0024] Fig.10 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The method for automatically generating a label quality detection script provided by an embodiment of the present invention can be applied to computer devices, including but not limited to various personal computers, laptops, smart phones, and tablet computers. The labels to be tested can be obtained from the list of labels to be tested and the data type of each label to be tested can be confirmed. Then, a corresponding search formula can be configured for each label to be tested according to the data type of each label to be tested. Multiple labels to be tested correspond to multiple search formulas, and multiple search formulas constitute a label quality detection script. Then, the label quality detection script can be used to automatically perform quality detection on the labels to be tested, which can improve detection efficiency and reduce detection costs.

[0027] See also Figure 1 As shown, Figure 1 A flowchart of a method for automatically generating a label quality detection script provided by an embodiment of the present invention includes the following steps:

[0028] S10: Obtain a list of tags to be tested, and traverse the list of tags to be tested to confirm the data type of each tag to be tested in the list of tags to be tested.

[0029] The method for automatically generating a tag quality detection script provided by the present invention can be used for quality detection of tags, that is, can be used for quality detection of tags to be tested. Before performing quality detection, it is necessary to first obtain a list of tags to be tested, and the list of tags to be tested contains tags that need to be quality-checked, that is, tags to be tested. A list of tags to be tested may include multiple tags to be tested, and the data types between the multiple tags to be tested may be the same or different, and the number of tags to be tested contained in the list of tags to be tested and the data type of each tag to be tested may be selected by the user.

[0030] For example, a user can manually input a tag ID or tag name that needs to be quality inspected, or any field that can be used to identify a tag. When quality inspection is required, the tag ID input by the user is obtained, and then a list of tags to be tested is generated based on the obtained tag ID. That is, the list of tags to be tested includes at least the tag ID. In addition to the tag ID, the list of tags to be tested may also include other parameters related to the tag, such as the tag object type and data type.

[0031] If the list of tags to be tested contains data types, the data type of the tag to be tested can be obtained at the same time when the tag to be tested is obtained, thereby completing the confirmation of the data type of the tag to be tested. If the list of tags to be tested does not contain data types, the tag ID of the tag to be tested can be obtained, and the data type corresponding to the tag ID can be confirmed in a database or other table that can query the data type of the tag based on the tag ID, thereby completing the confirmation of the data type of the tag to be tested.

[0032] S20: configuring a corresponding search formula for each tag to be tested according to the data type of the tag to be tested to generate a tag quality detection script.

[0033] After confirming the data type of each tag to be tested, a corresponding search formula can be configured for each tag to be tested according to the data type of each tag to be tested, that is, one tag to be tested corresponds to one search formula, multiple tags to be tested correspond to multiple search formulas, and multiple search formulas constitute a tag quality detection script.

[0034] The search formula is used to perform quality inspection on the corresponding tags to be tested, and its specific form can be a search statement, and the corresponding preset processing template can be called to perform quality inspection on the tags to be tested. For example, the search formula can be an SQL statement, which is used to perform quality inspection and return the corresponding inspection results.

[0035] It should be understood that a preset configuration information table may be preset, and the preset configuration information table contains parameters associated with the tag to facilitate quality inspection and data reading. Figure 2 The method for automatically generating a label quality detection script further includes the following steps:

[0036] S30: Acquire preset configuration information, and extract preset parameters related to the tag from the preset configuration information;

[0037] S40, generating a preset data information table according to the preset parameters.

[0038] The preset configuration information can be extracted from the data source, which can be any data platform. After obtaining the preset configuration information, the preset parameters related to the tag can be extracted from the preset configuration information, and a preset data information table is generated based on the extracted preset parameters, that is, the preset data information table contains multiple preset parameters related to the tag.

[0039] The preset parameters in the preset data information table include but are not limited to tag ID, tag name (in Chinese and English), tag object type, data type, Hive table name, partition field, etc. When any preset parameter is needed, it can be queried from the preset data information table.

[0040] After obtaining the tags to be tested, you can sort them. For example, query the corresponding Hive table name in the preset data information table according to the tag ID of the tag to be tested, and confirm the corresponding Hive table according to the Hive table name. If the confirmed Hive table is a partition table, you can further confirm the partition field name and the partition data to be selected. For example, if you need to perform quality inspection on data within a specific time period, you need to select the partition containing the data in that time period.

[0041] Among them, Figure 3 The method for automatically generating a label quality detection script further includes the following steps:

[0042] S50: Acquire a preset processing template, and traverse the preset data information table according to the preset processing template to confirm preset parameters of the processing type that conforms to the preset processing template to obtain parameters to be configured.

[0043] S60: configuring the preset processing template according to the parameters to be configured to obtain the search formula.

[0044] The preset processing template mainly includes rules for quality inspection of the labels to be tested. There can be multiple preset processing templates for detecting labels of different data types. For example, for digital labels to be tested, the preset processing template is used to detect the maximum value, minimum value and mean value contained in the label to be tested. For text labels to be tested, the preset processing template is used to detect the maximum length and minimum length of the text contained in the label to be tested.

[0045] The preset processing template is mainly used to detect the tag to be tested of a certain data type. When detecting a specific tag to be tested, you can first traverse the preset data information table to confirm the preset parameters associated with the tag to be tested, and then set the confirmed preset parameters as the parameters to be configured, and finally replace the parameters to be configured into the preset processing template to obtain the retrieval formula.

[0046] For example, the preset processing template may include a placeholder for a tag name, and the placeholder containing the tag name may be replaced by parameters related to the tag name in the parameters to be configured, thereby associating the preset processing template with the corresponding tag to be tested.

[0047] In actual use, you can configure a preset processing template, that is, an SQL template, according to the quality inspection standard HiveSQL, so as to confirm the quality inspection method for different data types, and then traverse the preset data information table according to the requirements of the SQL template, and then replace the placeholders in the SQL template with the relevant parameters in the preset data information table to generate a complete SQL statement. The generated SQL statement is the search formula. In addition, if the Hive table where the label to be tested is located is a partition table, you also need to add corresponding partition screening conditions in the SQL statement according to the requirements of the partition field and partition data.

[0048] like Figure 4 As shown, in step S10, that is, traversing the list of tags to be tested to confirm the data type of each tag to be tested in the list of tags to be tested, specifically includes the following steps S11-S12:

[0049] S11: traverse the list of tags to be tested to confirm the tag ID of each tag to be tested.

[0050] S12: Confirming the data type corresponding to the tag ID in the preset data information table according to the tag ID.

[0051] The list of tags to be tested can include the tag ID of each tag to be tested. After confirming the tag to be tested, confirm the tag ID of the corresponding tag to be tested, and then confirm the tag ID of the tag to be tested in the preset data information table to be the same as the tag ID in the list of tags to be tested. That is, the list of tags to be tested includes tag IDs that need to be quality inspected, and the preset data information table contains both tag IDs that need to be quality inspected and tag IDs that do not need to be quality inspected. The tag ID can be used to confirm in the preset data information table whether the tag to be tested exists in the preset data information table. If the corresponding tag ID can be queried in the preset data information table, the data type corresponding to the tag ID can be further confirmed, thereby completing the confirmation of the data type of the tag to be tested.

[0052] Among them, Figure 5 The method for automatically generating a label quality detection script further includes the following steps:

[0053] S70: Obtain a preset enumeration dictionary related table, and reconstruct the data format of the preset enumeration dictionary related table into a format that matches the tag quality detection script.

[0054] The data type of the tag includes an enumeration type, so the preset enumeration dictionary related table can be obtained separately, and the number can be multiple, which is obtained according to actual needs. For example, the preset enumeration dictionary related table can be a week enumeration table, which contains seven data such as Monday to Sunday. The preset enumeration dictionary related table can also be a Boolean enumeration table, which contains two data, TRUE and FALSE. For the enumeration type of the tag to be tested, it is necessary to generate a grouping statistics SQL statement based on the enumeration value.

[0055] After obtaining the preset enumeration dictionary related table, the data format of the preset enumeration dictionary related table can be reconstructed. For example, the data format of the preset enumeration dictionary related table can be reconstructed into JSON format, so that this information can be cached in memory, reducing the number of times the label quality detection script accesses the database and improving query efficiency.

[0056] Among them, Figure 6 The method for automatically generating a label quality detection script further includes the following steps:

[0057] S80: Obtain preset script parameters, and configure the label quality detection script according to the preset script parameters.

[0058] The script parameter can be how many tags are grouped together and output to the same script file, that is, how many search expressions generate a tag quality detection script. The specific value can be determined by environmental resources and time requirements. When computing resources are relatively loose or calculation results are required to be obtained in a short time, the value can be set to a smaller value to facilitate the rapid generation of tag quality detection scripts and improve detection efficiency.

[0059] Among them, Figure 7 The method for automatically generating a label quality detection script further includes the following steps:

[0060] S90: Perform total statistics for each tag to be tested according to the data type of the tag to be tested.

[0061] Total statistics are used to count the total number of times the tag to be tested appears in the data set. When performing total statistics, different statistical methods can be set according to the data type of the tag to be tested. For example, for Boolean and enumeration types of tags to be tested, group statistics can be performed according to the values, and the number of times each enumeration value or Boolean value appears in the data set can be counted.

[0062] It can be seen that in the above scheme, the corresponding search formula can be configured according to the label to be tested, and a label quality detection script can be generated according to multiple search formulas. Finally, the label quality is automatically detected by the label quality detection script, which improves the detection efficiency and reduces the cost.

[0063] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0064] In one embodiment, a device for automatically generating a label quality detection script is provided, and the device for automatically generating a label quality detection script corresponds to the method for automatically generating a label quality detection script in the above embodiment. Figure 8 As shown, the device for automatically generating a label quality detection script includes an acquisition module 101 and a generation module 102. The detailed description of each functional module is as follows:

[0065] The acquisition module 101 is used to acquire a list of tags to be tested, and traverse the list of tags to be tested to confirm the data type of each tag to be tested in the list of tags to be tested;

[0066] The generating module 102 is used to configure a corresponding search formula for each tag to be tested according to the data type of the tag to be tested to generate a tag quality detection script.

[0067] In one embodiment, the device for automatically generating a label quality detection script further includes:

[0068] An extraction module, used to obtain preset configuration information and extract preset parameters related to the tag from the preset configuration information;

[0069] A generating module, used for generating a preset data information table according to the preset parameters;

[0070] A traversal module, used for acquiring a preset processing template, and traversing the preset data information table according to the preset processing template to confirm preset parameters of the processing type that conforms to the preset processing template to obtain parameters to be configured;

[0071] The first configuration module is used to configure the preset processing template according to the parameters to be configured to obtain the search formula.

[0072] In one embodiment, the determination module 101 is specifically configured to:

[0073] Traversing the list of tags to be tested to confirm the tag ID of each tag to be tested;

[0074] The data type corresponding to the tag ID is confirmed in the preset data information table according to the tag ID.

[0075] In one embodiment, the device for automatically generating a label quality detection script further includes:

[0076] A reconstruction module, used for obtaining a preset enumeration dictionary related table, and reconstructing the data format of the preset enumeration dictionary related table into a format matching the label quality detection script;

[0077] A second configuration module, used to obtain preset script parameters and configure the label quality detection script according to the preset script parameters;

[0078] The statistics module is used to perform total statistics for each tag to be tested according to the data type of the tag to be tested.

[0079] The present invention provides a device for automatically generating a label quality detection script, which can configure a corresponding search formula according to the label to be tested, and generate a label quality detection script according to multiple search formulas, and finally automatically detect the label quality through the label quality detection script, thereby improving the detection efficiency and reducing the cost.

[0080] For the specific definition of the device for automatically generating label quality detection scripts, please refer to the definition of the method for automatically generating label quality detection scripts above, which will not be repeated here. Each module in the above-mentioned device for automatically generating label quality detection scripts can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0081] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a method for automatically generating a label quality detection script.

[0082] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the client side of a method for automatically generating a label quality detection script

[0083] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0084] Obtain a list of tags to be tested, and traverse the list of tags to be tested to confirm the data type of each tag to be tested in the list of tags to be tested;

[0085] According to the data type of the tag to be tested, a corresponding search formula is configured for each tag to be tested to generate a tag quality detection script.

[0086] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0087] Obtain a list of tags to be tested, and traverse the list of tags to be tested to confirm the data type of each tag to be tested in the list of tags to be tested;

[0088] According to the data type of the tag to be tested, a corresponding search formula is configured for each tag to be tested to generate a tag quality detection script.

[0089] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0090] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0091] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0092] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for automatically generating a label quality detection script, characterized in that: include: Obtain a list of tags to be tested, and traverse the list of tags to be tested to confirm the data type of each tag to be tested in the list of tags to be tested; According to the data type of the tag to be tested, a corresponding search formula is configured for each tag to be tested to generate a tag quality detection script.

2. The method for automatically generating a label quality detection script according to claim 1, characterized in that: Also includes: Acquire preset configuration information, and extract preset parameters related to the tag from the preset configuration information; A preset data information table is generated according to the preset parameters.

3. The method for automatically generating a label quality detection script according to claim 2, characterized in that: Also includes: Acquire a preset processing template, and traverse the preset data information table according to the preset processing template to confirm preset parameters of the processing type that conforms to the preset processing template to obtain parameters to be configured; The preset processing template is configured according to the parameters to be configured to obtain the search formula.

4. The method for automatically generating a label quality detection script according to claim 2, characterized in that: The traversing the list of tags to be tested to confirm the data type of each tag to be tested in the list of tags to be tested includes: Traversing the list of tags to be tested to confirm the tag ID of each tag to be tested; The data type corresponding to the tag ID is confirmed in the preset data information table according to the tag ID.

5. The method for automatically generating a label quality detection script according to claim 1, characterized in that: Also includes: A preset enumeration dictionary related table is obtained, and a data format of the preset enumeration dictionary related table is reconstructed into a format that matches the label quality detection script.

6. The method for automatically generating a label quality detection script according to claim 1, characterized in that: Also includes: Obtain preset script parameters, and configure the label quality detection script according to the preset script parameters.

7. The method for automatically generating a label quality detection script according to claim 1, characterized in that: Also includes: The total amount statistics are performed for each tag to be tested according to the data type of the tag to be tested.

8. A device for automatically generating a label quality detection script, characterized in that: include: An acquisition module is used to acquire a list of tags to be tested, and traverse the list of tags to be tested to confirm the data type of each tag to be tested in the list of tags to be tested; The generating module is used to configure a corresponding search formula for each tag to be tested according to the data type of the tag to be tested so as to generate a tag quality detection script.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for automatically generating a label quality detection script as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for automatically generating a label quality detection script as claimed in any one of claims 1 to 7 are implemented.

Images