Method and device for batch labeling of data items

Abstract

The embodiment of the present invention discloses a method and device for batch bidding of data items, which can be used in the financial field or other technical fields. The method comprises: creating a database table, importing the table structure of the table in each system into the database table, and then filtering the data items in the database table for garbled characters; after filtering the garbled characters, performing deduplication processing on the data items in the database table to obtain a result table; obtaining a data dictionary table according to the data items in the result table according to standard naming; and performing bidding processing on the system tables to be bid according to the data dictionary table. The present invention achieves the beneficial effect of accurately and efficiently batch bidding on the data items in the tables of each system.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for batch labeling of data items. Background Art

[0002] In the development work related to big data, data reading and processing between multiple systems are often involved. Obviously, this process involves a large number of tables. Different systems may be developed by different people, so in different tables, field names with the same meaning may have different names. This leads to confusion in the meaning of data items in the subsequent system upgrade and integration process, and it is necessary to formulate unified data item names. In big data systems, because there are many systems involved, and different developers have different standards for the formulation of data item names. Therefore, in big data development, it often involves the calculation of massive data in the system, and data item names with different standards will make program development difficult and difficult to accurately obtain data. Therefore, manual methods are often used to correct English names through the Chinese meaning of data items. At present, the manual labeling method has the problems of low efficiency and insufficient accuracy. To address these problems, the existing technology urgently needs a more efficient and accurate labeling solution. Summary of the invention

[0003] In order to solve at least one technical problem in the above-mentioned background technology, the present invention proposes a method and device for batch labeling of data items.

[0004] In order to achieve the above object, according to one aspect of the present invention, a method for batch tagging of data items is provided, the method comprising:

[0005] Create a database table, import the table structure of the table in each system into the database table, and then filter the data items in the database table for garbled characters;

[0006] After filtering out garbled characters, deduplication is performed on the data items in the database table to obtain a result table;

[0007] Obtain a data dictionary table according to the data items in the result table in accordance with standard naming;

[0008] The system table to be rejected is processed according to the data dictionary table.

[0009] Optionally, filtering the data items in the database table for garbled characters specifically includes:

[0010] Filter out the data items in the database table whose Chinese names appear garbled.

[0011] Optionally, the performing deduplication processing on the data items in the database table specifically includes:

[0012] Deduplication processing is performed on each data item in the database table according to the Chinese name of the data item.

[0013] Optionally, obtaining a data dictionary table according to standard naming of the data items in the result table specifically includes:

[0014] For each data item in the result table, a standard English name corresponding to the Chinese name of the data item is established.

[0015] In order to achieve the above object, according to another aspect of the present invention, a device for batch labeling of data items is provided, the device comprising:

[0016] The data item Chinese name processing module is used to create a database table, import the table structure of the table in each system into the database table, and then filter the garbled characters of the data items in the database table;

[0017] A full table generation module is used to perform deduplication processing on the data items in the database table after filtering the garbled characters to obtain a result table;

[0018] A data dictionary table generation module, used to obtain a data dictionary table according to the data items in the result table in accordance with standard naming;

[0019] The tag drop module is used to perform tag drop processing on the system table to be dropped according to the data dictionary table.

[0020] Optionally, the data item Chinese name processing module specifically includes:

[0021] The filtering processing submodule is used to filter out the data items in the database table whose Chinese names appear garbled.

[0022] Optionally, the full scale generation module specifically includes:

[0023] The deduplication processing submodule is used to perform deduplication processing on each data item in the database table according to the Chinese name of the data item.

[0024] Optionally, the data dictionary table generating module specifically includes:

[0025] The standard English name setting submodule is used to set the standard English name corresponding to the Chinese name of each data item in the result table.

[0026] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for batch labeling of data items when executing the computer program.

[0027] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the steps of the above-mentioned method for batch labeling of data items are implemented.

[0028] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, a computer program product is also provided, including a computer program / instruction, which implements the steps of the above-mentioned method for batch labeling of data items when executed by a processor.

[0029] The beneficial effects of the present invention are:

[0030] The embodiment of the present invention creates a database table, imports the table structure of the tables in each system into the database table, and then performs garbled code filtering on the data items in the database table. After the garbled code filtering, the data items in the database table are deduplicated to obtain a result table, and then a data dictionary table is obtained according to the data items in the result table according to standard naming. Finally, the system tables to be marked are marked according to the data dictionary table, thereby achieving the beneficial effect of accurately and efficiently marking the data items in the tables of each system in batches. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0032] Figure 1 is a flow chart of a method for batch labeling of data items according to an embodiment of the present invention;

[0033] Figure 2 is a flow chart of processing a database table according to an embodiment of the present invention;

[0034] Figure 3 This is a first structural block diagram of a device for batch labeling of data items according to an embodiment of the present invention;

[0035] Figure 4 This is a second structural block diagram of the device for batch labeling of data items according to an embodiment of the present invention;

[0036] Figure 5 It is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0039] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatus.

[0040] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations.

[0042] Figure 1 Flowchart of a method for batch labeling of data items according to an embodiment of the present invention. Figure 1 As shown, in one embodiment of the present invention, the method for batch labeling of data items of the present invention includes steps S101 to S104.

[0043] Step S101, creating a database table, importing the table structure of the table in each system into the database table, and then filtering the data items in the database table for garbled characters.

[0044] In the present invention, the present invention creates a database table, imports the table structure of all tables in the system into the database table, and the table contains the Chinese names and English names of all data items. The present invention then filters the Chinese names of the data items for garbled characters.

[0045] Step S102, after filtering out garbled characters, deduplication processing is performed on the data items in the database table to obtain a result table.

[0046] The present invention performs deduplication processing on data items in a database table to ensure that the same data item appears only once.

[0047] Step S103, obtaining a data dictionary table according to the data items in the result table according to standard naming.

[0048] The present invention corrects the English name of the data item according to the meaning of the Chinese name, thus obtaining a data dictionary table, and then the present invention can automatically place the label according to the data in the data dictionary table. This step solves the efficiency problem of multiple labeling caused by repeated appearance of data items.

[0049] Step S104, performing tag dropping processing on the system table to be dropped according to the data dictionary table.

[0050] In one embodiment of the present invention, the present invention can import all system tables into the database, which are tables to be marked, and the data items therein are subsets of the data dictionary table. Then the data dictionary table obtained in the previous step is connected with each table according to the Chinese name, and then the English names of the data items in the data dictionary table are used to replace the English names of the data items in the marked table, and the batch marking work is completed.

[0051] As can be seen from the above embodiments, the present invention proposes an efficient batch labeling method, which automatically imports the tables of various systems into the database and filters the garbled Chinese names of data items. The present invention performs deduplication operations on repeated data items, reducing manpower consumption. The method can directly batch label a large number of data items according to the data dictionary table. This process does not require manual intervention, and can ensure the uniformity of data item naming standards while improving efficiency.

[0052] In one embodiment of the present invention, the filtering of garbled characters from the data items in the database table in the above step S101 specifically includes:

[0053] Filter out the data items in the database table whose Chinese names appear garbled.

[0054] In one embodiment of the present invention, the deduplication processing of the data items in the database table in the above step S102 specifically includes:

[0055] Deduplication processing is performed on each data item in the database table according to the Chinese name of the data item.

[0056] In one embodiment of the present invention, the step S103 of obtaining a data dictionary table according to the data items in the result table in accordance with standard naming specifically includes:

[0057] For each data item in the result table, a standard English name corresponding to the Chinese name of the data item is established.

[0058] Figure 2 Flowchart of the processing of the database table by the embodiment of the present invention, such as Figure 2 As shown, in one embodiment of the present invention, the process of processing a database table of the present invention includes step S201 and step S202.

[0059] Step S201, filtering out data items in the database table whose Chinese names appear garbled.

[0060] Step S202: De-duplicate the data items in the database table according to the Chinese names of the data items.

[0061] In the development work related to big data, it often involves reading and processing data between multiple systems. Obviously, this process involves a large number of tables. Different systems may be developed by different people, so in different tables, the field names with the same meaning may have different names. As a result, the subsequent system upgrade and integration process requires the formulation of unified data item names.

[0062] To sum up, the reasons that affect the efficiency and quality of label placement are as follows:

[0063] 1. There are many types of systems;

[0064] 2. The table structure is diversified and prominent;

[0065] 3. Data items appear repeatedly.

[0066] In traditional bidding, it is difficult to perform operations with unified standards. At the same time, the appearance of duplicate data items leads to increased workload, affecting quality and efficiency.

[0067] The advantages of the method proposed by the present invention are as follows:

[0068] 1. Each system table is entered into the database (database table), and subsequent operations reduce labor costs and error rates;

[0069] 2. The present invention combines various system tables into one table. Since there are duplicate data items, deduplication operation is performed to reduce the problem of low efficiency caused by duplicate data items.

[0070] 3. After the present invention generates a data dictionary table according to the standard, the data dictionary table is connected with the system table waiting for the bid to be dropped, so that the bid dropping task can be quickly realized without manual input, which greatly improves the efficiency and quality.

[0071] As can be seen from the above embodiments, the present invention proposes a method for batch labeling of data items. Developers only need to formulate a standard name for each data item, and all data items involved in the system will be automatically updated, thereby realizing automated labeling operations and saving labor costs. Specifically, developers need to create a table in the database and import the table structure of the tables in each system into the table. Then remove the garbled characters in the Chinese names of the data items, and then deduplicate the table. Next, establish standard English names for the data items based on the Chinese keywords of the data items in the deduplicated table to generate a standard data item table. The data item table is connected to the table structure with labels, and the names of the data items in the labeling table are updated. At this point, the labeling of the data items in the system is completed. It is convenient for subsequent big data programs to process data from different systems.

[0072] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0073] Based on the same inventive concept, an embodiment of the present invention further provides a device for batch labeling of data items, which can be used to implement the method for batch labeling of data items described in the above embodiment, as described in the following embodiment. Since the principle of solving the problem by the device for batch labeling of data items is similar to that of the method for batch labeling of data items, the embodiment of the device for batch labeling of data items can refer to the embodiment of the method for batch labeling of data items, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0074] Figure 3 is a first structural block diagram of a device for batch labeling of data items according to an embodiment of the present invention, such as Figure 3 As shown, in one embodiment of the present invention, the data item batch labeling device of the present invention includes:

[0075] The data item Chinese name processing module is used to create a database table, import the table structure of the table in each system into the database table, and then filter the garbled characters of the data items in the database table;

[0076] A full table generation module is used to perform deduplication processing on the data items in the database table after filtering the garbled characters to obtain a result table;

[0077] A data dictionary table generation module, used to obtain a data dictionary table according to the data items in the result table in accordance with standard naming;

[0078] The tag drop module is used to perform tag drop processing on the system table to be dropped according to the data dictionary table.

[0079] In one embodiment of the present invention, the data item Chinese name processing module specifically includes:

[0080] The filtering processing submodule is used to filter out the data items in the database table whose Chinese names appear garbled.

[0081] In one embodiment of the present invention, the full scale generation module specifically includes:

[0082] The deduplication processing submodule is used to perform deduplication processing on each data item in the database table according to the Chinese name of the data item.

[0083] In one embodiment of the present invention, the data dictionary table generation module specifically includes:

[0084] The standard English name setting submodule is used to set the standard English name corresponding to the Chinese name of each data item in the result table.

[0085] In one embodiment of the present invention, the data item Chinese name processing module is specifically used to enter the tables in each system into the database, and then filter the Chinese names to remove garbled characters.

[0086] In one embodiment of the present invention, the full table generation module is specifically used to merge the output tables of the data item Chinese name processing module into one table, and the result table has the Chinese name and English name of the data item.

[0087] In one embodiment of the present invention, the data dictionary table generation module is specifically used to perform a deduplication operation on the result table of the full table generation module, and then name the data items according to the naming rules to obtain the data dictionary table.

[0088] In one embodiment of the present invention, the tagging module is specifically used to connect the data dictionary table with each system table and perform tagging operations on the data items.

[0089] like Figure 4As shown, in one embodiment of the present invention, the input of the data item Chinese name processing module is the table of each system. Because the Chinese name of the data item exists in the Chinese name, data item description, and garbled code, which affects the subsequent operations, this module first filters out the garbled code. The input of the full table generation module is the output table of the data item Chinese name processing module, which performs a weight reduction operation to obtain a result table. The function of this module is to lay the foundation for generating a data dictionary table. The input of the data dictionary table generation module is the output of the full table generation module, and the data dictionary table is output by naming the data items according to the standard. The function of this module is to generate a data dictionary table. The input of the label drop module is the output table of the data item Chinese name processing module, and these tables are matched with the data dictionary table to obtain the table after label drop. This module is the last step to realize the label drop work and improve the work efficiency of label drop.

[0090] In order to achieve the above object, according to another aspect of the present application, a computer device is also provided. Figure 5 As shown, the computer device includes a memory, a processor, a communication interface and a communication bus. A computer program that can be executed on the processor is stored in the memory. When the processor executes the computer program, the steps in the above-mentioned embodiment method are implemented.

[0091] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0092] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and units, such as the corresponding program units in the above method embodiments of the present invention. The processor executes various functional applications of the processor and works data processing by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiments.

[0093] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0094] The one or more units are stored in the memory, and when executed by the processor, the method in the above embodiment is performed.

[0095] The specific details of the above-mentioned computer device can be understood by referring to the corresponding related descriptions and effects in the above-mentioned embodiments, and will not be repeated here.

[0096] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer-readable storage medium is also provided, wherein the computer-readable storage medium stores a computer program, and the computer program implements the steps in the above-mentioned method for batch labeling of data items when executed in a computer processor. Those skilled in the art can understand that the implementation of all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program can include the processes of the embodiments of the above-mentioned methods when executed. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0097] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer program product is also provided, including a computer program / instruction, which implements the steps of the above-mentioned method for batch labeling of data items when executed by a processor.

[0098] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for batch labeling of data items, characterized in that: include: Create a database table, import the table structure of the table in each system into the database table, and then filter the data items in the database table for garbled characters; After filtering out garbled characters, deduplication is performed on the data items in the database table to obtain a result table; Obtain a data dictionary table according to the data items in the result table in accordance with standard naming; Perform bid rejection processing on the system table to be rejected according to the data dictionary table; The step of obtaining a data dictionary table according to the data items in the result table in accordance with standard naming specifically includes: For each data item in the result table, establish a standard English name corresponding to the Chinese name of the data item; The performing bid rejection processing on the system table to be rejected according to the data dictionary table includes: The data dictionary table is connected to each table according to the Chinese name, and then the English names of the data items in the data dictionary table are used to replace the English names of the data items in the table to be bid, so as to complete the batch bidding work.

2. The method for batch labeling of data items according to claim 1, characterized in that: The filtering of garbled characters from the data items in the database table specifically includes: Filter out the data items in the database table whose Chinese names appear garbled.

3. The method for batch labeling of data items according to claim 1, characterized in that: The deduplication processing of the data items in the database table specifically includes: Deduplication processing is performed on each data item in the database table according to the Chinese name of the data item.

4. A device for batch labeling of data items, characterized in that: include: The data item Chinese name processing module is used to create a database table, import the table structure of the table in each system into the database table, and then filter the data items in the database table for garbled characters; A full table generation module is used to perform deduplication processing on the data items in the database table after filtering the garbled characters to obtain a result table; A data dictionary table generation module, used to obtain a data dictionary table according to the data items in the result table in accordance with standard naming; A bid dropping module, used for performing bid dropping processing on the system table to be dropped according to the data dictionary table; The data dictionary table generation module specifically includes: A standard English name setting submodule is used to set, for each data item in the result table, a standard English name corresponding to the Chinese name of the data item; The performing bid rejection processing on the system table to be rejected according to the data dictionary table includes: The data dictionary table is connected to each table according to the Chinese name, and then the English names of the data items in the data dictionary table are used to replace the English names of the data items in the table to be bid, so as to complete the batch bidding work.

5. The device for batch labeling of data items according to claim 4, characterized in that: The data item Chinese name processing module specifically includes: The filtering processing submodule is used to filter out the data items in the database table whose Chinese names appear garbled.

6. The device for batch labeling of data items according to claim 4, characterized in that: The full scale generation module specifically includes: The deduplication processing submodule is used to perform deduplication processing on each data item in the database table according to the Chinese name of the data item.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

8. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images