A label generation method, device, computer equipment and storage medium

Abstract

The present invention discloses a label generation method, device, computer equipment and storage medium, the method comprising: obtaining historical data records containing labels of different categories, and constructing a label metadata table based on the labels in the historical data records; summarizing the labels in the historical data records into label configuration information in the label metadata table; extracting each label in the label metadata table based on the label configuration information, and constructing an original label data table according to the extracted labels; performing label cleaning processing on the original label data table through a preset label mapping table, and setting the original label data table after label cleaning as the final label data table. The present invention summarizes the labels in the historical data records into label configuration information, and constructs an original label data table based on this, and then performs cleaning and other processing on the original label data table to obtain the final label data table, so that the label generation efficiency and label management effect can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer software technology, and in particular to a label generation method, device, computer equipment and storage medium. Background Art

[0002] Existing label generation technology is usually based on manual label definition and organization at the database field level, which is then configured in the system background and the logic of each label is generated through code. However, this method consumes a lot of manpower. In addition, labels are extracted and calculated based on traditional relational databases, but when the amount of data is large, there are disadvantages such as insufficient data storage space and slow computing efficiency. At the same time, when managing labels, it is necessary to manually maintain the metadata of each label, which may lead to omissions and errors when there are too many labels. Summary of the invention

[0003] The embodiments of the present invention provide a label generation method, apparatus, computer equipment and storage medium, aiming to improve label generation efficiency and label management effect.

[0004] In a first aspect, an embodiment of the present invention provides a label generation method, including:

[0005] Obtain historical data records containing different category tags, and construct a tag metadata table based on the tags in the historical data records;

[0006] Summarizing the tags in the historical data records into tag configuration information in the tag metadata table;

[0007] Extracting each tag in the tag metadata table based on the tag configuration information, and constructing an original tag data table according to the extracted tags;

[0008] The original label data table is subjected to label cleaning processing through a preset label mapping table, and the original label data table after label cleaning is set as the final label data table.

[0009] In a second aspect, an embodiment of the present invention provides a label generation device, including:

[0010] A data acquisition unit, used to acquire historical data records containing different categories of tags, and construct a tag metadata table based on the tags in the historical data records;

[0011] A summarizing and summarizing unit, used for summarizing the tags in the historical data records into tag configuration information in the tag metadata table;

[0012] A tag extraction unit, configured to extract each tag in the tag metadata table based on the tag configuration information, and construct an original tag data table according to the extracted tags;

[0013] The label cleaning unit is used to perform label cleaning processing on the original label data table through a preset label mapping table, and set the original label data table after label cleaning as the final label data table.

[0014] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the label generation method as described in the first aspect when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the label generation method as described in the first aspect is implemented.

[0016] The embodiment of the present invention provides a label generation method, device, computer equipment and storage medium, the method comprising: obtaining historical data records containing labels of different categories, and constructing a label metadata table based on the labels in the historical data records; summarizing the labels in the historical data records into label configuration information in the label metadata table; extracting each label in the label metadata table based on the label configuration information, and constructing an original label data table according to the extracted labels; performing label cleaning processing on the original label data table through a preset label mapping table, and setting the original label data table after label cleaning as the final label data table. The embodiment of the present invention summarizes the labels in the historical data records into label configuration information, and constructs an original label data table based on this, and then performs cleaning and other processing on the original label data table in combination with the label mapping table, thereby obtaining the final label data table, which can improve the label generation efficiency and label management effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0018] Figure 1 A flowchart of a label generation method provided by an embodiment of the present invention;

[0019] Figure 2A schematic block diagram of a label generating device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0022] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0023] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0024] See below Figure 1 , Figure 1 A flowchart of a label generation method provided by an embodiment of the present invention specifically includes: steps S101 to S104.

[0025] S101, obtaining historical data records containing different categories of tags, and constructing a tag metadata table based on the tags in the historical data records;

[0026] S102, summarizing the tags in the historical data records into tag configuration information in the tag metadata table;

[0027] S103, extracting each tag in the tag metadata table based on the tag configuration information, and constructing an original tag data table according to the extracted tags;

[0028] S104: Perform label cleaning on the original label data table using a preset label mapping table, and set the original label data table after label cleaning as a final label data table.

[0029] In this embodiment, the labels in the historical data records are summarized as label configuration information, and the original label data table is constructed based on this. The original label data table is then cleaned and processed in combination with the label mapping table to obtain the final label data table, which can improve the label generation efficiency and label management effect.

[0030] In addition, this embodiment also uses a custom tag method, that is, summarizes the tags into tag configuration information, so as to cover the generation of complex tags that cannot be defined by tag type. In addition, during the generation process, the generation of complex logical tags can be achieved through SQL statements.

[0031] In one embodiment, the step S102 includes:

[0032] When there are multiple data values ​​corresponding to the tags in the historical data record, the tags are summarized into a column of multiple tags;

[0033] When the data value corresponding to the tag in the historical data record is one, the tags are summarized into a column of single tags;

[0034] When the labels in the historical data records correspond to identification codes, the labels are summarized as one-hot encoded labels;

[0035] When the tag in the historical data record is a table header name, the tag is summarized as a table tag;

[0036] When the data value corresponding to the tag in the historical data record is obtained by calculation, the tag is summarized as a custom tag;

[0037] The one column of multiple labels, one column of single label, one hot encoding label, table label and custom label are set as the label configuration information.

[0038] In this embodiment, according to the data values ​​or identifiers corresponding to the tags in the historical data records, the tags are respectively summarized into a column of multiple tags, a column of single tags, unique hot encoding tags, table tags and custom tags, and further, a column of multiple tags, a column of single tags, unique hot encoding tags, table tags and custom tags are summarized as the tag configuration information.

[0039] For example, if one of the columns in the historical data record is: 'Enterprise Type', where the data value is 'Fortune 500 Enterprises, Listed Companies', it can be seen that the two labels (i.e., Fortune 500 Enterprises, Listed Companies) are separated by the delimiters ',', that is, by configuring the delimiter, the labels can be automatically segmented, so this type of label can be defined as 'one column with multiple labels'; if one of the columns in the historical data record is 'Green Channel Enterprises', enterprises with green channel qualifications will have a 'Green Channel' value stored in the corresponding field, so this type of label can be defined as 'one column with single label'; if According to the record, there is a column named 'Is it one of the top 500 companies in China', and the storage value in the column is 0 or 1, that is, if the corresponding enterprise is one of the top 500 companies in China, it is marked as 1, otherwise it is 0. In this case, the label can be defined as 'unique hot encoding label'; if the table name in the historical data record is 'Green Channel Enterprise List', then the label in this case can be defined as 'the table itself is a label', that is, a table label; and for labels that need to be obtained by calculation for label data, they can be defined as 'custom labels', and specific calculation rules can be defined by configuring the label calculation logic (sql).

[0040] In one embodiment, the step S103 includes:

[0041] Based on the one-column multi-label, one-column single-label, one-hot encoded labels, table labels and custom labels, batch extract the labels in the label metadata table;

[0042] The labels extracted in batches are arranged in sequence to construct the original label data table.

[0043] In this embodiment, after the tags are classified into different types, the tags in the tag metadata table can correspond to their own tag configuration information, that is, the tag type, etc. Then, according to the tag configuration information, the tags in the tag metadata table are extracted in batches according to type, and the tags are set by region or arranged in sequence according to the order of extraction, and the form thus formed is set as the original tag data table. Of course, in the original tag data table, the order between different types of tags can be freely set, for example, a column of tags is located before a column of single tags, or a unique hot encoded tag is located after a custom tag, and so on.

[0044] In one embodiment, the tag generation method further includes:

[0045] An enterprise standard information table is obtained, and based on the enterprise standard information table, basic enterprise information in the original label data table is corrected and completed so that each label in the original label data table is correctly associated with the enterprise.

[0046] In this embodiment, based on the existing enterprise standard information table (i.e., containing true and accurate basic information of the enterprise, such as enterprise name, unified code, registration number, etc.), the basic information of the enterprise (such as name, unified code, registration number, etc.) in the original label data table is corrected, completed, etc. to ensure that the labels in the original label data table can be associated with the correct enterprises and enterprise information.

[0047] In one embodiment, the step S104 includes:

[0048] Based on the label mapping table, labels with different names but the same meaning in the original label data table are mapped into a standard label;

[0049] The tags with the same name in the original tag data table are obtained, and the tags with the same name are deduplicated according to the enterprise standard information table.

[0050] In this embodiment, the label mapping table is configured with information on how to clean labels from different data tables, as well as the standard name of the label. For example, a company is marked as "China's Top 500" and "Top 500" in two different data tables. The two label names are different, but the meanings are essentially the same. Therefore, by mapping the two labels to a standard "China's Top 500" label in the label mapping table, the consistency and accuracy of the final label can be guaranteed. In a specific embodiment, a label mapping system can be constructed based on the labels in the historical data records and the past label configuration experience. Through this label mapping system, it can be determined whether to map the name of the label and which standard name it should be mapped to.

[0051] Furthermore, in one embodiment, the tag generation method further includes:

[0052] Set an expiration date for the tags that have been mapped and deduplicated;

[0053] Disable invalid labels and expired labels in the original label data table.

[0054] In this embodiment, the validity and validity period are set for the tag, so that the tag can be used only within the validity period. Once the validity period is exceeded, the tag loses its validity and becomes an invalid tag or an expired tag. Invalid and expired tags can be disabled in time to avoid being found invalid or expired after use, which can improve the user experience.

[0055] In one embodiment, the tag generation method further includes:

[0056] Load the Hive configuration file based on Spark SQL and obtain the Hive metadata information;

[0057] Storing the tag data table as a Hive table through the metadata information of the Hive;

[0058] Perform corresponding operations on the Hive table based on Spark SQL.

[0059] In this embodiment, by using big data technology, based on Hive (data warehouse tool) and Spark (a computing engine), etc., the label data table is stored and processed, the label calculation process can be processed concurrently, the data storage space can be reduced, and the efficiency of label generation can be improved. It can also achieve linear improvement by horizontally expanding hardware resources. Specifically, first load the Hive configuration file through Spark SQL to obtain the corresponding metadata information, and store the label data table as a Hive table, and then perform corresponding operations on the Hive table through Spark SQL, such as query, update, etc. Further, through distributed computing, millions of data labels can be generated in a very short time. The above steps can effectively ensure the accuracy and timeliness of the generated label data, and only require very little manual intervention in the process. Compared with the existing label generation technology, it has a great improvement in efficiency.

[0060] Figure 2 A schematic block diagram of a label generation device 200 provided in an embodiment of the present invention, the device 200 includes:

[0061] The data acquisition unit 201 is used to acquire historical data records containing different types of tags, and construct a tag metadata table based on the tags in the historical data records;

[0062] A summarizing and summarizing unit 202, configured to summarize the tags in the historical data records into tag configuration information in the tag metadata table;

[0063] The tag extraction unit 203 is used to extract each tag in the tag metadata table based on the tag configuration information, and construct an original tag data table according to the extracted tags;

[0064] The label cleaning unit 204 is used to perform label cleaning processing on the original label data table through a preset label mapping table, and set the original label data table after label cleaning as the final label data table.

[0065] In one embodiment, the summarizing and summarizing unit 202 includes:

[0066] A first summarizing unit, configured to summarize the labels into a column of multiple labels when there are multiple data values ​​corresponding to the labels in the historical data record;

[0067] A second summarizing unit, configured to summarize the labels into a list of single labels when the label corresponding to the data value in the historical data record is one;

[0068] A third summarizing unit, configured to summarize the labels into one-hot encoded labels when the labels in the historical data records correspond to identification codes;

[0069] a fourth summarizing unit, configured to summarize the label as a table label when the label in the historical data record is a table header name;

[0070] a fifth summarizing unit, configured to summarize the label as a custom label when a data value corresponding to the label in the historical data record is obtained by calculation;

[0071] A summary unit is used to set the one column of multiple labels, one column of single label, one hot encoding label, table label and custom label as the label configuration information.

[0072] In one embodiment, the tag extraction unit 203 includes:

[0073] A batch extraction unit, for batch extraction of tags in the tag metadata table based on the one-column multi-tag, one-column single-tag, one-hot encoded tags, table tags, and custom tags;

[0074] The sequence setting unit is used to set the labels extracted from the batches in sequence, so as to construct the original label data table.

[0075] In one embodiment, the label generating device 200 further includes:

[0076] The correction and completion unit is used to obtain the enterprise standard information table, and based on the enterprise standard information table, correct and complete the basic information of the enterprise in the original label data table, so that each label in the original label data table is correctly associated with the enterprise.

[0077] In one embodiment, the label cleaning unit 204 includes:

[0078] A label mapping unit, configured to map labels with different names but the same meaning in the original label data table into a standard label based on the label mapping table;

[0079] The tag deduplication unit is used to obtain tags with the same name in the original tag data table, and perform deduplication processing on the tags with the same name according to the enterprise standard information table.

[0080] In one embodiment, the label generating device 200 further includes:

[0081] A time limit setting unit, used to set a validity period for the label after the mapping and deduplication processing;

[0082] The disabling unit is used to disable invalid labels and expired labels in the original label data table.

[0083] In one embodiment, the label generating device 200 further includes:

[0084] The file loading unit is used to load the Hive configuration file based on Spark SQL and obtain the metadata information of Hive;

[0085] A data storage unit, used to store the tag data table as a Hive table through the metadata information of the Hive;

[0086] The table operation unit is used to perform corresponding operations on the Hive table based on Spark SQL.

[0087] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0088] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0089] The embodiment of the present invention also provides a computer device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the computer device may also include various network interfaces, power supplies and other components.

[0090] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

[0091] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A label generation method, characterized in that: include: Obtain historical data records containing different category tags, and construct a tag metadata table based on the tags in the historical data records; Summarizing the tags in the historical data records into tag configuration information in the tag metadata table; Extracting each tag in the tag metadata table based on the tag configuration information, and constructing an original tag data table according to the extracted tags; Performing label cleaning processing on the original label data table through a preset label mapping table, and setting the original label data table after label cleaning as the final label data table; The step of aggregating the tags in the historical data records into the tag configuration information in the tag metadata table includes: When there are multiple data values ​​corresponding to the tags in the historical data record, the tags are summarized into a column of multiple tags; When the data value corresponding to the tag in the historical data record is one, the tags are summarized into a column of single tags; When the labels in the historical data records correspond to identification codes, the labels are summarized as one-hot encoded labels; When the tag in the historical data record is a table header name, the tag is summarized as a table tag; When the data value corresponding to the tag in the historical data record is obtained by calculation, the tag is summarized as a custom tag; Set the one column of multiple labels, one column of single label, one hot encoding label, table label and custom label as the label configuration information; The step of extracting each tag in the tag metadata table based on the tag configuration information and constructing an original tag data table according to the extracted tags includes: Based on the one-column multi-label, one-column single-label, one-hot encoded labels, table labels and custom labels, batch extract the labels in the label metadata table; The labels extracted in batches are arranged in sequence to construct the original label data table.

2. The label generation method according to claim 1, characterized in that: Also includes: An enterprise standard information table is obtained, and based on the enterprise standard information table, basic enterprise information in the original label data table is corrected and completed so that each label in the original label data table is correctly associated with the enterprise.

3. The label generation method according to claim 2, characterized in that: The process of performing label cleaning on the original label data table by using a preset label mapping table and setting the label-cleaned original label data table as a final label data table includes: Based on the label mapping table, labels with different names but the same meaning in the original label data table are mapped into a standard label; The tags with the same name in the original tag data table are obtained, and the tags with the same name are deduplicated according to the enterprise standard information table.

4. The label generation method according to claim 3, characterized in that: Also includes: Set an expiration date for the tags that have been mapped and deduplicated; Disable invalid labels and expired labels in the original label data table.

5. The label generation method according to claim 1, characterized in that: Also includes: Load the Hive configuration file based on Spark SQL and obtain the Hive metadata information; Storing the tag data table as a Hive table through the metadata information of the Hive; Perform corresponding operations on the Hive table based on Spark SQL.

6. A label generating device, characterized in that: include: A data acquisition unit, used to acquire historical data records containing different categories of tags, and construct a tag metadata table based on the tags in the historical data records; A summarizing and summarizing unit, used for summarizing the tags in the historical data records into tag configuration information in the tag metadata table; A tag extraction unit, configured to extract each tag in the tag metadata table based on the tag configuration information, and construct an original tag data table according to the extracted tags; A label cleaning unit, used for performing label cleaning processing on the original label data table through a preset label mapping table, and setting the original label data table after label cleaning as the final label data table; The summarizing and summarizing unit 202 includes: A first summarizing unit, configured to summarize the labels into a column of multiple labels when there are multiple data values ​​corresponding to the labels in the historical data record; A second summarizing unit, configured to summarize the labels into a list of single labels when the label corresponding to the data value in the historical data record is one; A third summarizing unit, configured to summarize the labels into one-hot encoded labels when the labels in the historical data records correspond to identification codes; a fourth summarizing unit, configured to summarize the label as a table label when the label in the historical data record is a table header name; a fifth summarizing unit, configured to summarize the label as a custom label when a data value corresponding to the label in the historical data record is obtained by calculation; A summarizing unit, used for setting the one-column multi-label, one-column single-label, one-hot encoding label, table label and custom label as the label configuration information; The label extraction unit comprises: A batch extraction unit, for batch extraction of tags in the tag metadata table based on the one-column multi-tag, one-column single-tag, one-hot encoded tags, table tags, and custom tags; The sequence setting unit is used to set the labels extracted in batches in sequence, so as to construct the original label data table.

7. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the label generation method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the label generation method according to any one of claims 1 to 5 is implemented.

Images