Data tag tracing method and device, electronic equipment or storage medium

Abstract

The invention relates to a data label tracing method and device, electronic equipment or a storage medium, and belongs to the technical field of data management and processing.The method comprises the steps that a data source table is determined according to a label task; analyzing the data source table, and determining a storage path and a query condition of data; taking information with uniqueness in the query condition as a traceability code, and creating label information containing the traceability code; and determining a data source of the tag according to the tag information, the storage path and the query condition. According to the invention, the purposes of efficiently marking in mass data and ensuring that the source of each label is traceable are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management and processing, and in particular to a data label traceability method and device, electronic equipment or storage medium. BACKGROUND

[0002] With the popularization of information technology, efficient data management and classification is imperative. Data labels are a common solution to data classification. They not only enable efficient classification and management of different things, but also provide convenience for information retrieval and decision-making. By accurately labeling data, more accurate user profiling, market analysis, and risk control can be achieved. However, the reliability of labels is closely related to their source. Once the transparency of the source of the label is lost, the authenticity of the data is no longer reliable, and the value of the data is greatly reduced. Building a perfect data label traceability mechanism can not only ensure the authenticity of the label, but also improve the efficiency and accuracy of data processing.

[0003] The existing label traceability technology mainly uses metadata management tools (such as Apache Atlas, Informatica, Collibra, etc.) to automatically collect and track metadata information of data, forming a data flow path. Users can trace the data label migration history based on metadata. This traceability method requires high dependence on the completeness and accuracy of metadata. If the metadata is missing or incomplete, the traceability effect will be affected. Moreover, the standardization of data labels and metadata is low, and different organizations and systems may use different label definitions and classification standards, resulting in poor data interoperability and difficulty in effective cross-system and cross-database type data tracking. Another label traceability method based on logs in the prior art is to record operation logs, change logs, audit logs, etc. in the data system to track the change history of data labels. However, this method has the problems of large log volume, complex management, and dependence on the completeness and timeliness of logs, and is rarely used.

[0004] Therefore, how to efficiently label massive data and ensure the traceability of the source of each label has become a problem to be solved. SUMMARY

[0005] Therefore, it is necessary to provide a data label traceability method, device, electronic equipment or storage medium to achieve the purpose of efficiently labeling massive data and ensuring the traceability of the source of each label.

[0006] To achieve the above purpose, the present application provides a data label traceability method, comprising: determining a data source table according to a label task; parsing the data source table to determine a storage path and a query condition of the data. Using the unique information in the query condition as the traceability code, creating label information containing the traceability code; The data source of the tag is determined according to the tag information, storage path and query conditions.

[0007] In a possible implementation, determining the data source table according to the label task includes: Determine the number of data source tables based on the labeling task scenario and the data table dimension information; If the data source table is a single data table, the single data table is used as the data source table; If the data source table is multiple data tables, a query statement is constructed according to the labeling task requirements, the multiple data tables are determined through the query statement, and the multiple data tables are used as the data source tables.

[0008] In a possible implementation, if the data source table is multiple data tables, unique information in the query condition is used as the traceability code, and tag information containing the traceability code is created, including: When the data source table consists of multiple data tables, each data table corresponds to a traceability code, and multiple traceability codes are stored in a nested manner.

[0009] In a possible implementation, parsing the data in the data source table to determine the data storage path and query conditions includes: Parse the data in the data source table to determine the schema name, logical structure and field information of the data source table; Determine a storage path according to the schema name and the logical structure of the data table; The query condition is determined according to the field information.

[0010] In a possible implementation, after determining the storage path according to the schema name and the logical structure of the data table, the method further includes: Construct a traceability information table based on the schema name and the logical structure of the data table.

[0011] In a possible implementation, determining the data source of the tag according to the tag information, the storage path, and the query condition includes: Get query request; According to the query request, the data source of the tag is determined based on the traceability code and traceability information table.

[0012] In one possible implementation, constructing a query statement according to labeling task requirements and determining multiple data tables through the query statement includes: Use constructed SQL to determine multiple data tables.

[0013] In a second aspect, the present application further provides a data label traceability device, comprising: A data source table determination module is configured to determine a data source table according to a label task; An analysis module is configured to analyze the data source table to determine a storage path and a query condition of data; A traceability code determination module is configured to take information with uniqueness in the query condition as a traceability code, and create label information containing the traceability code; A traceability module is configured to determine a data source of a label according to the label information, the storage path and the query condition.

[0014] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory; The memory stores a computer readable program which can be executed by the processor; The processor executes the computer readable program to implement the steps in the data label traceability method.

[0015] In a fourth aspect, the present application further provides a computer readable storage medium which stores one or more programs which can be executed by one or more processors to implement the steps in the data label traceability method.

[0016] The present application has the following beneficial effects: first, a data source table is determined according to a label task; then, the data source table is analyzed to determine a storage path and a query condition of data; and information with uniqueness in the query condition is taken as a traceability code to create label information containing the traceability code; the real-time and accuracy of the data label are ensured, and a personalized scheme meeting the needs can be customized according to the actual business scenario, and the query condition based on the unique information is taken as the traceability code, and the label information is created synchronously to realize real-time and dynamic updating of the traceability change of the label; finally, a data source of the label is determined according to the label information, the storage path and the query condition. Therefore, the present application can realize tracing and exploring of a life cycle of a bottom layer data source and a marking process (rule) according to the data label, the traceability code based on unique information ensures the reliability, accuracy and traceability of the data label, and the user can understand the forming process of the label. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A method flowchart of an embodiment of the data label traceability method provided by the present application; Figure 2 A method flowchart of an embodiment of step S102 in the method; Figure 1 Figure 3 A schematic diagram of an embodiment of the data label traceability device provided by the present application;​ Figure 4 is a running environment schematic diagram of an embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION

[0018] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, wherein the accompanying drawings form a part of the present application and are used to explain the principles of the embodiments of the present application, but are not used to limit the scope of the present application.

[0019] Data label: refers to the identification information or category label assigned to each data point in a dataset. These labels are used to describe the characteristics, categories or states of data, so that the data can be effectively analyzed and processed. In machine learning and data mining, data labels are usually a key component of supervised learning, helping algorithms understand and learn the relationship between data. By labeling data, researchers and data scientists can train models to make predictions or classifications on unlabeled data. The accuracy and consistency of data labels are crucial to the performance and reliability of the model.

[0020] Data label traceability: refers to the practice of tracking and recording the source, generation process and change history of data labels. This process aims to ensure the reliability, accuracy and traceability of data labels, so that users can understand how the labels are formed.

[0021] Labeling: refers to the process of assigning labels or categories to data points in a dataset. This process is usually used to prepare data for machine learning, ensuring that the model can identify and learn different data features. Labeling can be done manually or through automated tools, commonly used in image, text and audio data processing. Accurate labeling is crucial to improving the performance and reliability of the model.

[0022] Subject: refers to a collection of individuals with the same attributes, which can be an entity or a portrait of an entity.

[0023] One specific embodiment of the present application discloses a data label traceability method, please refer to Figure 1 , comprising: S101, determining a data source table according to a label task; S102, parsing the data source table to determine the storage path and query condition of the data; S103, taking the information with uniqueness in the query condition as a traceability code, and creating label information containing the traceability code; S104, determining the data source of the label according to the label information, storage path and query condition.

[0024] In the embodiment, first, the data source table is determined according to the label task; then the data source table is parsed to determine the storage path and the query condition of the data; and the information with uniqueness in the query condition is taken as the trace code to create the label information containing the trace code; the real-time and accuracy of the data label are ensured, the personalized scheme meeting the needs can be customized according to the actual business scene, and the query condition based on the unique information is taken as the trace code, and the label information is created synchronously to realize the real-time and dynamic update of the trace change of the label. Finally, the data source of the label is determined according to the label information, the storage path and the query condition. Therefore, the application can realize the trace and exploration of the bottom data source and the labeling process (rule) in a life cycle according to the data label, the trace code based on the unique information ensures the reliability, accuracy and traceability of the data label, and the user can understand the forming process of the label.

[0025] It should be noted that the data source table refers to a database table used to store and manage raw data in the data management and analysis process. These tables usually contain detailed records of data and are the starting point of data processing, analysis and label generation. The data source table can be one or more tables in the database or tables from external data sources.

[0026] In step S103, the trace code can be one or more of the query conditions. It can be understood that in other embodiments, the trace code can also be other data with non-unique properties, but must have certain characteristics to be used as a distinguishing feature for label tracing.

[0027] In step S102, the data source table is parsed to determine the storage path and the query condition, in order to ensure that the data source used in the subsequent labeling work summary is clear, structured and accurate, thereby improving the efficiency and quality of labeling, and improving the accuracy and efficiency of label tracing.

[0028] In step S101, before running the labeling task, a main information carrying label information is needed. Here, the main body can be an entity or an object with specific attributes. Then a label information is added, which is used to describe an object with common characteristics, such as "having a car", "having a house", "criminal record" and the like, and the label information can be added according to the corresponding attributes.

[0029] In some embodiments, the data source table is determined according to the label task, comprising: The number of data source tables is determined according to the scene of the label task and the dimension information of the data table; If the data source table is a single data table, the single data table is taken as the data source table; If the data source table is multiple data tables, a query statement is constructed according to the label task requirements, the multiple data tables are determined through the query statement, and the multiple data tables are taken as the data source table.

[0030] In this embodiment, in some cases, the fields in a single data table may be split or scattered, making it difficult to directly use the data for label generation. Or the data table may contain less dimensions, which cannot meet the complex label generation requirements. Therefore, a user writes a custom complex SQL query for extracting and integrating data. Such a query can span multiple tables, perform complex JOIN operations, subqueries, aggregation operations, etc., to obtain more rich data information.

[0031] In some embodiments, if the data source table is multiple data tables, the information with uniqueness in the query condition is taken as the trace code, and a label information containing the trace code is created, including: When the data source table is multiple data tables, each data table corresponds to a trace code, and multiple trace codes are stored in a nested manner.

[0032] In this embodiment, the current data label mainly stores data around the TAG_INFO table, and the current data writing task is the association relationship of TAG_INFO ->TAG_RULE_DETAIL ->DP_DATA_PROCESSING_RULE.

[0033] In the existing table design, the DP_DATA_PROCESSING_RULE table records the source dataset ID, and for a single table, the source table information can be obtained according to the field. But currently, SQL is a special dataset, and its ID is also directly stored in this field. Finally, the data in Table 1 is generated and stored in the TAG_DATA_LINEAGE table to form trace record information.

[0034] Table 1: Trace record information

[0035] The data label is stored in the ES, and in order to meet the possibility of multiple data sources corresponding to a label task, the label storage at the destination end adopts a nested structure, which is as follows: { "DG_DM_ID": "data primary key ID", "XM": "main display name", "GMSFHM": "main body ID", "tags": [ { "tag_code": "label code", "tag_name": "tag name", "rule": [ "task rule ID 1", "task rule ID 2", "task rule ID 3" ], "data_trace": [ { "rule_id": "task rule ID 1", "trace_id": "trace ID 1" }, { "rule_id": "task rule ID 1", "trace_id": "trace ID 1, trace ID 2" } A data_trace nested field is added under the tag to record the trace information. The data_trace includes a rule_id and a trace_id. The rule_id corresponds to the task rule, and the trace_id corresponds to the trace ID. When there are multiple source tables, the multiple trace IDs recorded in the trace_id are separated by commas.

[0036] After the user completes the trace configuration, the trace task can be started. The trace field and related information are written in the process of marking. This information will not only be carried in the tag data to form a historical trace information of the tag (if the information is tampered with in the process, it can also be found in the historical information), so as to trace back to the basis at each time of marking. At the same time, the trace related information will be stored in the trace information table, which includes data source ID, mode name, trace field and a series of descriptive content for subsequent implementation of the tag trace function. Through the trace information table, unified storage of multiple types of data sources and multiple sources of information can be realized.

[0037] After the marking task is completed, the system will write the trace value (at this time, because the selected trace field is the ID card number, the trace value is the ID card information of the principal) into the tag information.

[0038] In some embodiments, the data in the data source table is parsed to determine the storage path and query condition of the data, please refer to Figure 2 , including: ​​​​S201, parse data in the data source table, determine the mode name of the data source table, the logical structure of the data table and the field information; S202, determine the storage path according to the mode name and the logical structure of the data table; S203, determine the query condition according to the field information.

[0039] In the embodiment, the purpose of parsing the data source table is to clarify the path, structure and source of the data source, so as to better perform the tagging operation and the label traceability operation on the data.

[0040] In step S201, the data in the data source table is parsed to obtain the mode name of the data source table, the name of the data table, the ID of the data table, the logical structure of the data table and the field information, etc., wherein the name of the data table and the ID of the data table are mainly for the case of multi-table source data table, and can quickly locate the target table in the multi-table.

[0041] Specifically, according to the mode name and the logical structure of the data table, the storage path can be determined, so that when performing label traceability, the table where the metadata corresponding to the label is located can be quickly located, and the field information is used as the query condition, and the information with uniqueness in the query condition is used as the traceability code, so that the label traceability can be quickly and accurately realized, and the credibility of the label is improved.

[0042] Further, the label traceability configuration is to collect the positioning information required for tracing the label to the row level, that is, through the parameter item of the label traceability, the unique data in the table / data set can be accurately positioned, and we usually use the business ID that can ensure uniqueness or the unique identity information of the label subject, such as the citizen ID number of the personnel subject.

[0043] Further, by storing the blood relationship between the table and the task, that is, the traceability table, it can be traced back to a certain type of data or a certain table, and the traceability efficiency is improved.

[0044] Therefore, by standardizing the label data object and the data implementation process, recording and tracking the implementation process, and completely landing the implementation result data, the label data can not only be used in the system, but also can be fully utilized, tracked and the process of the data label by other third-party systems according to the database connection or interface form. The problems of label definition and label coding that may exist between different systems are eliminated, so that the systems can realize seamless interconnection. And through the configuration of the label task scheduling system, the real-time and accuracy of the data label are ensured, and the personalized scheme meeting the needs can be customized according to the actual business scene, so that the system flexibility is ensured and the problems caused by data delay are reduced to the greatest extent.

[0045] Further, the technical solution of the present application is based on the data bloodline analysis function carried by the platform existing traceability information, which improves the traceability accuracy while minimizing the additional development workload, without the help of external third-party work and platform implementation, and almost no operation and maintenance threshold and demand. The effect brought by the label traceability technology mainly reflects in the aspects of data quality improvement and decision transparency. By tracking the source and change history of the label in detail, the organization can ensure the accuracy and consistency of the data label. In addition, this technology enhances compliance and ensures that data usage complies with relevant regulatory requirements. Finally, label traceability helps to improve team trust in data, optimize analysis results, and improve the reliability of business decisions.

[0046] Based on the above data label traceability method, the embodiment of the present application also provides a data label traceability device, please refer to Figure 3 , comprising: The data source table determination module 310 is used to determine the data source table according to the label task; The analysis module 320 is used to analyze the data source table to determine the storage path and query condition of the data; The traceability code determination module 330 is used to take the information with uniqueness in the query condition as the traceability code, and create label information containing the traceability code; The traceability module 340 is used to determine the data source of the label according to the label information, storage path and query condition.

[0047] As Figure 4 shown, based on the above data label traceability method, the present application also correspondingly provides an electronic device, which can be a mobile terminal, desktop computer, notebook, palm computer and server, etc. The electronic device includes a processor 410, a memory 420 and a display 430. Figure 4 Only part of the components of the electronic device are shown, but it should be understood that it is not required to implement all the shown components, and more or less components can be alternatively implemented.

[0048] The memory 420 can be an internal storage unit of the electronic device in some embodiments, such as a hard disk or a memory of the electronic device. The memory 420 can also be an external storage device of the electronic device in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 420 can include both an internal storage unit and an external storage device of the electronic device. The memory 420 is used to store application software and various data installed on the electronic device, such as program codes installed on the electronic device. The memory 420 can also be used to temporarily store data that has been output or will be output. In an embodiment, the memory 420 stores a data tag traceability program 440, which can be executed by the processor 410 to implement the data tag traceability method of the embodiments of the present application.

[0049] The processor 410 can be a central processing unit (CPU), a microprocessor, or other data processing chip in some embodiments, used to run program codes or process data stored in the memory 420, such as to execute the data tag traceability method, etc.

[0050] The display 430 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 430 is used to display information of the data tag traceability electronic device and to display a visualized user interface. The components 410-430 of the electronic device communicate with each other through a system bus.

[0051] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by instructing relevant hardware through a computer program, and the program can be stored in a computer readable storage medium. The computer readable storage medium includes a magnetic disk, an optical disk, a read-only memory, a random access memory, etc.

[0052] The above description is only a preferred embodiment of the present application, and the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed by the present application can be easily conceived by those skilled in the art, and should be covered within the protection scope of the present application.

Claims

1. A data tag traceability method, characterized in that: include: Determine the data source table based on the labeling task; Parse the data source table to determine the data storage path and query conditions; Using the unique information in the query condition as the traceability code, creating label information containing the traceability code; The data source of the tag is determined according to the tag information, storage path and query conditions.

2. The data tag traceability method according to claim 1, characterized in that: Determining the data source table according to the label task includes: Determine the number of data source tables based on the labeling task scenario and the data table dimension information; If the data source table is a single data table, the single data table is used as the data source table; If the data source table is multiple data tables, a query statement is constructed according to the labeling task requirements, the multiple data tables are determined through the query statement, and the multiple data tables are used as the data source tables.

3. The data tag traceability method according to claim 2, characterized in that: If the data source table is multiple data tables, unique information in the query condition is used as the traceability code, and tag information containing the traceability code is created, including: When the data source table consists of multiple data tables, each data table corresponds to a traceability code, and multiple traceability codes are stored in a nested manner.

4. The data tag traceability method according to claim 1, characterized in that: The parsing of the data in the data source table to determine the storage path and query conditions of the data includes: Parse the data in the data source table to determine the schema name, logical structure and field information of the data source table; Determine a storage path according to the schema name and the logical structure of the data table; The query condition is determined according to the field information.

5. The data tag traceability method according to claim 4, characterized in that: After determining the storage path according to the schema name and the logical structure of the data table, the method further includes: Construct a traceability information table based on the schema name and the logical structure of the data table.

6. The data tag traceability method according to claim 5, characterized in that: The step of determining the data source of the tag according to the tag information, storage path, and query conditions includes: Get query request; According to the query request, the data source of the tag is determined based on the traceability code and traceability information table.

7. The data tag traceability method according to claim 1, characterized in that: The query statement is constructed according to the labeling task requirements, and multiple data tables are determined through the query statement, including: Use constructed SQL to determine multiple data tables.

8. A data tag tracing device, characterized in that: include: The data source table determination module is used to determine the data source table according to the label task; A parsing module, used to parse the data source table to determine the storage path and query conditions of the data; A traceability code determination module is used to use the unique information in the query condition as the traceability code and create label information containing the traceability code; The traceability module is used to determine the data source of the tag based on the tag information, storage path and query conditions.

9. An electronic device, characterized in that: include: processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the data tag tracing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the data tag tracing method according to any one of claims 1 to 7.

Images