Metadata error correction method and system

Abstract

The invention provides a metadata error correction method and system, and belongs to the technical field of data correction, and the method comprises the following steps: collecting metadata to be cleaned, carrying out preliminary cleaning on the data, and classifying the metadata according to a cleaning result to obtain structured data and unstructured data; the method comprises the following steps: respectively performing semantic understanding analysis on structured data and unstructured data, detecting a first potential error and a second potential error, creating a user interface, displaying the first potential error and the second potential error, marking a third potential error by a user through the user interface, and displaying the third potential error. Modifying and annotating the first potential error, the second potential error and the third potential error; according to the method and the device, correctness judgment is performed on the modified annotation, an automatic error correction mechanism is constructed according to a correctness judgment result, the metadata is corrected, the quality and consistency of the metadata are improved, manual intervention is reduced by the automatic error correction mechanism, mixed use of structured and unstructured data is effectively processed, and efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data correction technology, and in particular to a method and system for correcting metadata errors. Background Technology

[0002] In the era of big data, the rapid growth of data has made data quality issues increasingly prominent. The correctness and consistency of metadata are the foundation of data management and analysis. With the diversification of data sources, the mixed use of structured and unstructured data is becoming more and more common. How to effectively clean and analyze this data has become a research focus. Current error correction systems often rely on rules or simple models and lack a deep understanding of complex data structures and semantics.

[0003] Therefore, the present invention provides a method and system for correcting metadata errors. Summary of the Invention

[0004] This invention provides a metadata error correction method and system. By collecting and initially cleaning metadata, classifying it into structured and unstructured data, performing semantic analysis on both types of data, identifying potential errors, displaying errors through a user interface, allowing users to mark and annotate them, judging the correctness of the corrections based on user feedback, and constructing an automatic error correction mechanism, the quality and consistency of metadata are improved. The automatic error correction mechanism reduces manual intervention, effectively handles the mixed use of structured and unstructured data, and improves efficiency.

[0005] This invention provides a method for correcting metadata errors, comprising:

[0006] Step 1: Collect the metadata to be cleaned, perform preliminary cleaning on the data, and classify the metadata according to the cleaning results to obtain structured data and unstructured data;

[0007] Step 2: Perform semantic understanding analysis on the structured data and unstructured data respectively. Based on the results of the semantic understanding analysis, detect the first potential error in the structured data and the second potential error in the unstructured data.

[0008] Step 3: Create a user interface to display the first and second potential errors. Users can mark the third potential error through the user interface and make modification comments on the first, second, and third potential errors.

[0009] Step 4: Determine the correctness of the modified annotations, and construct an automatic error correction mechanism based on the correctness determination results to correct the metadata.

[0010] This invention provides a metadata error correction method, which involves collecting metadata to be cleaned, performing preliminary data cleaning, and classifying the metadata based on the cleaning results to obtain structured and unstructured data, including:

[0011] Determine the source of the metadata, and based on the source, clarify the data format and storage method of the metadata;

[0012] The metadata is initially cleaned according to the data format and storage method. After the initial cleaning, the cleaning effect is evaluated to determine whether a second cleaning is needed, and then the cleaning result is obtained.

[0013] The data type of each metadata field is determined based on the cleaning results, and the degree of structure of each metadata field is judged based on the data type.

[0014] Metadata is categorized based on its degree of structuring to determine whether it is structured or unstructured data.

[0015] This invention provides a metadata error correction method, which performs semantic understanding analysis on structured and unstructured data respectively, and detects a first potential error in the structured data and a second potential error in the unstructured data based on the semantic understanding analysis results, including:

[0016] Perform a first semantic understanding analysis on the structured data to obtain the first semantic understanding analysis result, and determine the first potential error based on the first semantic understanding analysis result;

[0017] Perform second semantic understanding analysis on unstructured data to obtain the second semantic understanding analysis results, and determine the second potential error based on the second semantic understanding analysis results.

[0018] This invention provides a metadata error correction method, which performs a first semantic understanding analysis on structured data to obtain a first semantic understanding analysis result, and determines a first potential error based on the first semantic understanding analysis result, including:

[0019] For each field of the structured data, write a detailed description, analyze the position of the field in the business process based on the detailed description, and determine the first relationship between the fields based on the position;

[0020] Based on the first relationship, a table formed by fields is determined, the primary key of the table is identified, and according to the identified primary key, the field in the secondary table corresponding to the primary key field of the primary table is found to confirm the foreign key;

[0021] Based on the primary key and foreign key, the functional dependencies of the basic fields are clearly defined, and then the secondary relationship between the fields is determined.

[0022] Logical relationships are identified in the fields based on the first and second relationships. The first semantic understanding analysis result of the structured data is obtained by combining the first and second relationships with the logical relationships. The first potential error is determined based on the first semantic understanding analysis result.

[0023] This invention provides a metadata error correction method, which performs second semantic understanding analysis on unstructured data, obtains the second semantic understanding analysis result, and determines a second potential error based on the second semantic understanding analysis result, including:

[0024] The unstructured data is reclassified to obtain text data, image data, and audio data;

[0025] Extract keywords from text data, identify the corresponding text topics in unstructured data based on the keywords, and use sentiment analysis tools to evaluate the sentiment tendency of the text data based on the text topics to obtain the sentiment tendency evaluation results.

[0026] Semantic consistency checks are performed on the text topic, sentiment consistency is assessed using the sentiment assessment results, similarity detection and processing are performed on the text data, and potential text errors are derived by combining the consistency check results, consistency assessment results, and processing results.

[0027] Identify the subject and scene in image data, extract image features based on the subject and scene, perform image error identification based on the image features, and derive potential image errors;

[0028] The audio data is segmented to obtain audio text data and audio sound data. A first sentiment assessment result is obtained based on the analysis of the text data. A second sentiment assessment result is obtained by analyzing the tone of voice of the audio sound data. Potential audio errors are identified based on the first sentiment assessment result and the second sentiment assessment result.

[0029] The second potential error is derived by combining the potential text errors, potential image errors, and potential audio errors.

[0030] This invention provides a metadata error correction method, which includes creating a user interface to display a first potential error and a second potential error. The user then marks a third potential error through the user interface and adds modification comments to the first, second, and third potential errors.

[0031] After comparing the first potential error with the second potential error, the user identifies the third potential error;

[0032] For the third potential error, a superordinate error label is made for the first potential error and the second potential error, and a first modification comment is made for the superordinate error label;

[0033] At the same time, a second modification annotation is made for the first potential error, the second potential error, and the third potential error.

[0034] This invention provides a metadata error correction method, which involves judging the correctness of modified annotations, constructing an automatic error correction mechanism based on the correctness judgment result, and correcting metadata errors, including:

[0035] The classifier model is trained to evaluate the accuracy of the modified annotations. An automatic error correction mechanism is built based on the modified annotations that exceed the preset accuracy threshold to perform metadata error correction.

[0036] This invention provides a metadata error correction system, comprising:

[0037] Classification module: Collects metadata to be cleaned, performs preliminary cleaning of the data, classifies the metadata based on the cleaning results, and obtains structured data and unstructured data;

[0038] Error detection module: Performs semantic understanding analysis on structured data and unstructured data respectively, and detects a first potential error in the structured data and a second potential error in the unstructured data based on the results of the semantic understanding analysis;

[0039] Annotation module: Creates a user interface to display the first and second potential errors, collects third potential errors marked by users through the user interface, and modifies the annotations of the first, second, and third potential errors;

[0040] Error correction module: The module judges the correctness of the modified annotations, constructs an automatic error correction mechanism based on the correctness judgment results, and corrects errors in the metadata.

[0041] Compared with existing technologies, the beneficial effects of this application are as follows: by collecting and initially cleaning metadata, classifying it into structured and unstructured data, performing semantic analysis on the two types of data, identifying potential errors, displaying errors through a user interface, allowing users to mark and annotate, judging the correctness of modifications based on user feedback, and constructing an automatic error correction mechanism, the quality and consistency of metadata are improved. The automatic error correction mechanism reduces manual intervention, effectively handles the mixed use of structured and unstructured data, and improves efficiency.

[0042] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0043] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0044] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0045] Figure 1 This is a flowchart illustrating a metadata error correction method provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the structure of a metadata error correction system provided in an embodiment of the present invention. Detailed Implementation

[0047] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0048] Example 1:

[0049] This invention provides a method for correcting metadata errors, such as... Figure 1 As shown, it includes:

[0050] Step 1: Collect the metadata to be cleaned, perform preliminary cleaning on the data, and classify the metadata according to the cleaning results to obtain structured data and unstructured data;

[0051] Step 2: Perform semantic understanding analysis on the structured data and unstructured data respectively. Based on the results of the semantic understanding analysis, detect the first potential error in the structured data and the second potential error in the unstructured data.

[0052] Step 3: Create a user interface to display the first and second potential errors. Users can mark the third potential error through the user interface and make modification comments on the first, second, and third potential errors.

[0053] Step 4: Determine the correctness of the modified annotations, and construct an automatic error correction mechanism based on the correctness determination results to correct the metadata.

[0054] In this embodiment, metadata is information describing the content, structure, format, and source of data. It helps users understand and manage data, such as the creation date of the dataset, author, and data type (e.g., CSV, JSON). The source refers to the location or method in which the metadata is generated or obtained, such as a database, sensor, user input, or third-party API.

[0055] In this embodiment, preliminary cleaning refers to the initial organization and standardization of metadata to remove obvious errors and inconsistencies, such as removing duplicate metadata entries and standardizing date formats.

[0056] In this embodiment, by determining the source of metadata, clarifying the data format and storage method, performing preliminary cleaning and evaluating the effect, deciding whether to perform secondary cleaning, determining the data type of each field after cleaning, judging the degree of structure, and finally classifying the metadata into structured data and unstructured data.

[0057] In this embodiment, structured data refers to data with a clear format and organization, which is easy for computers to process and analyze, such as rows and columns in a database table; unstructured data refers to data without a fixed format or structure, which is relatively complex to process, such as emails, social media posts, and video files.

[0058] In this embodiment, semantic understanding analysis includes a first semantic understanding analysis of structured data and a second semantic understanding analysis of unstructured data.

[0059] In this embodiment, the first potential error is determined based on the first semantic understanding analysis result. For example, there may be duplicate customer IDs in the customer table, or the customer IDs may not match the customer names (e.g., the same customer ID corresponds to multiple different names).

[0060] In this embodiment, the second potential error refers to the possible errors or inconsistencies identified after comprehensive analysis of text, image and audio data. These errors may originate from conflicts or inconsistencies between text content, image features or audio characteristics, affecting the accuracy and consistency of the overall data.

[0061] In this embodiment, the third potential error is an error discovered by the user. This includes the third potential error identified by the user by comparing the first and second potential errors, and the identifier for this error is the first modification annotation. It also includes a separate third potential error discovered by the user, and the identifier for this error is the second modification annotation. The third potential error is a new potential error identified by the user by comparing the first and second potential errors. For example, the first potential error is that the sentiment of the text is inconsistent with the topic, with the sentiment being negative, but the topic is "climate change". The second potential error is that the sentiment of the audio text is positive, but the tone analysis shows that it is angry. The third potential error is that the sentiment tendencies between the text and the audio are inconsistent, resulting in confusion in the overall information transmission. For example, the text content conveys a negative view on climate change, while the audio conveys a positive sentiment, creating a contradiction.

[0062] In this embodiment, the modification comment is a comment given to the user for correcting the error that occurred, including a first modification comment and a second modification comment.

[0063] In this embodiment, a third potential error is identified by comparing the first and second potential errors with the user's comparison. For the third potential error, the system marks the errors belonging to the first and second potential errors and adds a first modification comment. A second modification comment is added to all potential errors (first, second, and third).

[0064] In this embodiment, accuracy refers to the consistency between the modified annotation and the actual content that should be modified. High accuracy means that the annotation accurately reflects the errors in the original data. For example, a correct annotation is: if the original text has a spelling error, the modified annotation is "change 'teh' to 'the'". An incorrect annotation is: if the original text means "increase", the modified annotation is "decrease", which will lead to information transmission errors.

[0065] In this embodiment, the automatic error correction mechanism is a system that can automatically correct or improve annotations based on evaluation results to improve data quality. For example, if an annotation is marked as "error," the system can automatically generate a new modification suggestion. For instance, the original annotation is "change 'good' to 'bad'," but the system recognizes that the original sentence means "good," so it automatically generates a new annotation as "keep it as is" or "change 'bad' to 'good'."

[0066] In this embodiment, metadata correction refers to correcting the metadata (such as labels, classification information, etc.) in the dataset to ensure its accuracy and consistency. For example, if a record is incorrectly labeled as "negative" but is found to be "positive" after evaluation by the error correction mechanism, the system will automatically update the label of the record.

[0067] The working principle and beneficial effects of the above technical solution are as follows: by collecting and initially cleaning metadata, classifying it into structured and unstructured data, performing semantic analysis on the two types of data, identifying potential errors, displaying errors through the user interface, allowing users to mark and annotate, judging the correctness of modifications based on user feedback, and building an automatic error correction mechanism, thereby improving the quality and consistency of metadata. The automatic error correction mechanism reduces manual intervention, effectively handles the mixed use of structured and unstructured data, and improves efficiency.

[0068] Example 2:

[0069] This invention provides a metadata error correction method, which involves collecting metadata to be cleaned, performing preliminary data cleaning, and classifying the metadata based on the cleaning results to obtain structured and unstructured data, including:

[0070] Determine the source of the metadata, and based on the source, clarify the data format and storage method of the metadata;

[0071] The metadata is initially cleaned according to the data format and storage method. After the initial cleaning, the cleaning effect is evaluated to determine whether a second cleaning is needed, and then the cleaning result is obtained.

[0072] The data type of each metadata field is determined based on the cleaning results, and the degree of structure of each metadata field is judged based on the data type.

[0073] Metadata is categorized based on its degree of structuring to determine whether it is structured or unstructured data.

[0074] In this embodiment, data format refers to the way data is stored and represented. For example, common data formats include CSV, JSON, XML, Parquet, etc.

[0075] In this embodiment, the storage method refers to the storage location and structure of data and metadata, such as relational databases (e.g., MySQL), NoSQL databases (e.g., MongoDB), file systems (e.g., HDFS), etc.

[0076] In this embodiment, evaluating the cleaning effect involves checking whether the metadata after preliminary cleaning meets the expected standards, such as checking whether there are still duplicates after cleaning, and whether the format is consistent.

[0077] In this embodiment, the cleaning result refers to the state of the metadata after cleaning, such as a metadata dataset that is free of duplicates and has a uniform format after cleaning.

[0078] In this embodiment, the metadata fields are the various attributes or elements that constitute the metadata, such as dataset name, description, creator, creation date, etc.

[0079] In this embodiment, data type refers to the specific type of data in the metadata field, such as string, integer, date, etc. For example, the data type of the field "Creation Date" is date format, and the data type of the field "Dataset Name" is string.

[0080] In this embodiment, the degree of structuring refers to the degree of organization and formatting of data, which is usually divided into structured and unstructured.

[0081] The working principle and beneficial effects of the above technical solution are as follows: by determining the source of metadata, clarifying the data format and storage method, conducting preliminary cleaning and evaluating the effect, deciding whether to conduct secondary cleaning, determining the data type of each field after cleaning, judging the degree of structure, and finally classifying the metadata into structured data and unstructured data, ensuring the accuracy and consistency of metadata, and improving the reliability of data management.

[0082] Example 3:

[0083] This invention provides a metadata error correction method, which performs semantic understanding analysis on structured and unstructured data respectively, and detects a first potential error in the structured data and a second potential error in the unstructured data based on the semantic understanding analysis results, including:

[0084] Perform a first semantic understanding analysis on the structured data to obtain the first semantic understanding analysis result, and determine the first potential error based on the first semantic understanding analysis result;

[0085] Perform second semantic understanding analysis on unstructured data to obtain the second semantic understanding analysis results, and determine the second potential error based on the second semantic understanding analysis results.

[0086] In this embodiment, by writing detailed descriptions for each field of structured data and analyzing its position in the business process, the first relationship between fields is determined. Based on this relationship, primary keys and foreign keys are identified, functionally dependent basic fields are clarified, and then the second relationship between fields is determined. By combining the first relationship, the second relationship, and the logical relationship, the first semantic understanding analysis result of the structured data is obtained, and the first potential error is identified.

[0087] In this embodiment, unstructured data is reclassified to identify text, image, and audio data. Keywords are extracted from text data and themes are identified. Sentiment analysis is applied to assess sentiment tendencies, and consistency checks and similarity detection are performed to identify potential errors. Features are extracted from image data to identify errors, while potential errors in audio data are derived through segmentation and sentiment assessment. By combining the potential errors of various types of data, a second potential error is identified.

[0088] In this embodiment, the first semantic understanding analysis result is to synthesize the first relation, the second relation and the logical relation to obtain the understanding analysis result of the structured data. For example, the customer ID in the customer table is unique, the customer name corresponds one-to-one with the customer ID, and a customer can have multiple orders.

[0089] In this embodiment, the second semantic understanding analysis includes semantic understanding analysis of text data, image data, and audio data, and the results of the second semantic understanding analysis include potential errors in text, images, and audio.

[0090] The working principle and beneficial effects of the above technical solution are as follows: by performing first semantic understanding analysis on structured data to identify potential errors, and performing second semantic understanding analysis on unstructured data to determine its potential errors, by deeply understanding the semantic structure of the data, errors in the data can be accurately identified and marked, thereby providing a basis for subsequent error correction and data quality improvement, effectively dealing with the mixed use of structured and unstructured data, and improving the comprehensiveness of data analysis.

[0091] Example 4:

[0092] This invention provides a metadata error correction method, which involves performing a first semantic understanding analysis on structured data to obtain a first semantic understanding analysis result, and determining a first potential error based on the first semantic understanding analysis result, including:

[0093] For each field of the structured data, write a detailed description, analyze the position of the field in the business process based on the detailed description, and determine the first relationship between the fields based on the position;

[0094] Based on the first relationship, a table formed by fields is determined, the primary key of the table is identified, and according to the identified primary key, the field in the secondary table corresponding to the primary key field of the primary table is found to confirm the foreign key;

[0095] Based on the primary key and foreign key, the functional dependencies of the basic fields are clearly defined, and then the secondary relationship between the fields is determined.

[0096] Logical relationships are identified in the fields based on the first and second relationships. The first semantic understanding analysis result of the structured data is obtained by combining the first and second relationships with the logical relationships. The first potential error is determined based on the first semantic understanding analysis result.

[0097] In this embodiment, the detailed description is a detailed description of each field of the structured data, including the field name, data type, and meaning. For example, field name: Customer ID, data type: integer, meaning: uniquely identifies each customer; field name: Customer Name, data type: string, meaning: the customer's full name.

[0098] In this embodiment, the position of the business process is analyzed by examining the role and position of the fields in the business process to determine their importance in the data flow. For example, the customer ID is used to uniquely identify customers in the customer management system, and the customer name is used to display and contact customers.

[0099] In this embodiment, the first relationship is determined based on the position of the field in the business process. For example, the relationship between customer ID and customer name: customer ID is a unique identifier of customer name.

[0100] In this embodiment, the table is determined based on the first relationship, which is formed by the fields. For example, the customer table has the following fields: Customer ID, Customer Name, Customer Email, and Customer Phone Number.

[0101] In this embodiment, primary key identification is to identify the primary key in the table. The primary key is a field that uniquely identifies each row in the table. For example, the primary key of the customer table is the customer ID.

[0102] In this embodiment, the main table is a table that contains the primary key; for example, the customer table is the main table.

[0103] In this embodiment, a foreign key is determined by finding the field in the sub-table that corresponds to the primary key field in the main table. For example, in the order table, customer ID is used as a foreign key, pointing to customer ID in the customer table.

[0104] In this embodiment, the functional dependency base fields are based on the relationship between primary keys and foreign keys, and the functional dependency base fields are clearly defined. For example, in the order table, the order ID depends on the customer ID, and fields such as customer name and customer email also depend on the customer ID.

[0105] In this embodiment, the second relationship is determined based on functional dependency, such as the dependency relationship between customer ID and customer name: customer ID determines customer name.

[0106] In this embodiment, logical relationship identification is based on the first relationship and the second relationship to identify the logical relationship between fields. For example, there is a one-to-one relationship between customer ID and customer name; there is a one-to-many relationship between customer ID and multiple orders in the order table.

[0107] The working principle and beneficial effects of the above technical solution are as follows: by writing detailed descriptions for each field of structured data and analyzing its position in the business process, the first relationship between fields is determined. Based on this relationship, primary keys and foreign keys are identified, functionally dependent basic fields are clarified, and then the second relationship between fields is determined. By combining the first relationship, the second relationship, and the logical relationship, the first semantic understanding analysis result of the structured data is obtained, and potential errors are identified, optimizing data storage and retrieval efficiency, accurately identifying potential errors, and improving the intelligence level of the error correction system.

[0108] Example 5:

[0109] This invention provides a metadata error correction method, which involves performing second semantic understanding analysis on unstructured data to obtain a second semantic understanding analysis result, and determining a second potential error based on the second semantic understanding analysis result, including:

[0110] The unstructured data is reclassified to obtain text data, image data, and audio data;

[0111] Extract keywords from text data, identify the corresponding text topics in unstructured data based on the keywords, and use sentiment analysis tools to evaluate the sentiment tendency of the text data based on the text topics to obtain the sentiment tendency evaluation results.

[0112] Semantic consistency checks are performed on the text topic, sentiment consistency is assessed using the sentiment assessment results, similarity detection and processing are performed on the text data, and potential text errors are derived by combining the consistency check results, consistency assessment results, and processing results.

[0113] Identify the subject and scene in image data, extract image features based on the subject and scene, perform image error identification based on the image features, and derive potential image errors;

[0114] The audio data is segmented to obtain audio text data and audio sound data. A first sentiment assessment result is obtained based on the analysis of the text data. A second sentiment assessment result is obtained by analyzing the tone of voice of the audio sound data. Potential audio errors are identified based on the first sentiment assessment result and the second sentiment assessment result.

[0115] The second potential error is derived by combining the potential text errors, potential image errors, and potential audio errors.

[0116] In this embodiment, data reclassification is the process of classifying unstructured data into text data, image data, and audio data. For example, text data includes social media posts, email content, and articles; image data includes photos, charts, and paintings; and audio data includes podcasts, music, and telephone recordings.

[0117] In this embodiment, extracting keywords from text data involves extracting keywords from text data to identify the main content of the text. For example, keywords such as "climate change", "greenhouse gases", and "global warming" can be extracted from an article about climate change.

[0118] In this embodiment, the identification of the text topic is based on the extracted keywords. For example, the topic may be "environmental protection" or "the impact of climate change".

[0119] In this embodiment, the sentiment analysis tool is used to assess the sentiment tendency of the text data and obtain the sentiment tendency assessment result. For example, after analysis by the tool, the sentiment tendency of the article is found to be "negative" and the score is -0.6 (ranging from -1 to 1).

[0120] In this embodiment, the sentiment assessment result is the output of the sentiment analysis tool, reflecting the sentiment tendency of the text. For example, the sentiment assessment result is "negative".

[0121] In this embodiment, semantic consistency checking is performed using the text topic to ensure that the text content is consistent with the topic. For example, it checks whether the article content revolves around "climate change" and marks any off-topic content as inconsistent.

[0122] In this embodiment, the sentiment consistency assessment uses the sentiment tendency assessment results to ensure that the sentiment tendency of the text is consistent with the theme. For example, if the theme of the article is "environmental protection" and the sentiment tendency is "negative", then there is an inconsistency.

[0123] In this embodiment, similarity detection and processing involves performing similarity detection on text data to identify duplicate or similar content. For example, if two articles are found to have a content similarity of up to 90%, they can be marked as duplicates.

[0124] In this embodiment, potential text errors are identified by combining the consistency check results, consistency assessment results, and processing results. For example, if an article has issues such as being off-topic or inconsistent in sentiment, it is marked as a potential error.

[0125] In this embodiment, identifying the subject and scene in image data involves analyzing the image data and identifying the main subject and scene in the image. For example, in a street view photo, the subject is "pedestrian" and the scene is "city street".

[0126] In this embodiment, image features are extracted based on the subject and scene, such as color, shape, and texture. For example, the extracted image features include "bright colors" and "diverse shapes".

[0127] In this embodiment, image error identification is performed based on extracted image features to identify potential image errors, such as blurry images or unclear subjects, which are then marked as potential errors.

[0128] In this embodiment, audio data segmentation involves dividing the audio data into audio text data and audio sound data. For example, a podcast segment can be segmented into audio text (transcribed text) and audio sound (original audio).

[0129] In this embodiment, the first sentiment assessment result is obtained by performing a sentiment assessment on the audio text data. For example, the sentiment tendency of the transcribed podcast content is "positive" and the score is 0.7.

[0130] In this embodiment, tone analysis involves performing tone analysis on audio sound data to obtain a second emotion assessment result. For example, tone analysis shows that the speaker's tone is "angry" and the score is -0.8.

[0131] In this embodiment, potential audio errors are derived based on the results of the first and second emotion assessments. For example, the audio text may be "positive" but the tone may be "angry," indicating an emotional inconsistency.

[0132] In this embodiment, the second potential error is derived by combining potential text errors, potential image errors, and potential audio errors. For example, if the text is off-topic, the image is blurry and the subject is unclear, or the audio has inconsistent emotion, these are all collectively marked as the second potential error.

[0133] The working principle and beneficial effects of the above technical solution are as follows: by reclassifying unstructured data, text, image and audio data are identified; keywords are extracted from text data and themes are identified; sentiment analysis is applied to assess sentiment tendencies, and consistency checks and similarity detection are performed to identify potential errors; features are extracted from image data for error identification; potential errors are obtained from audio data through segmentation and sentiment assessment; potential errors are identified by combining the potential errors of various types of data, and a second potential error is identified, thereby improving the recognition rate of potential errors and enhancing the flexibility and adaptability of the system.

[0134] Example 6:

[0135] This invention provides a metadata error correction method, which includes creating a user interface to display a first potential error and a second potential error. The user then marks a third potential error through the user interface and adds modification comments to the first, second, and third potential errors.

[0136] After comparing the first potential error with the second potential error, the user identifies the third potential error;

[0137] For the third potential error, a superordinate error label is made for the first potential error and the second potential error, and a first modification comment is made for the superordinate error label;

[0138] At the same time, a second modification annotation is made for the first potential error, the second potential error, and the third potential error.

[0139] In this embodiment, the above error mark, which is a mark of potential errors caused by the first potential error and the second potential error, is a multi-segment error mark.

[0140] In this embodiment, the first modification annotation is to modify the above-mentioned error mark, explaining the nature of the error and its impact. For example, the first modification annotation is: for the first potential error: "The sentiment of the text is inconsistent with the theme, which may mislead the reader's understanding of the content"; for the second potential error: "The tone in the audio is inconsistent with the sentiment of the text, which may cause confusion in the transmission of information".

[0141] In this embodiment, the second modification annotation is a second modification annotation of the first potential error, the second potential error, and the third potential error, further clarifying the impact of the error and suggestions. For example, the second modification annotation is as follows: First potential error: It is recommended to modify the sentiment of the text to make it consistent with the theme and ensure the accuracy of information transmission; Second potential error: It is recommended to adjust the tone of the audio or re-record to ensure that the tone is consistent with the sentiment of the text and avoid misleading the audience; Third potential error: It is recommended to conduct a comprehensive review of the text and audio to ensure the consistency of the overall information and avoid emotional contradictions.

[0142] The working principle and beneficial effects of the above technical solution are as follows: by comparing the first potential error with the second potential error, the third potential error is identified. For the third potential error, the system marks the errors above the first and second potential errors and adds the first modification annotation. The system adds the second modification annotation to all potential errors (first, second, and third) in order to better record and manage error information, accumulate modification annotations, form a knowledge base for future reference, and improve the level of data management.

[0143] Example 7:

[0144] This invention provides a metadata error correction method, which involves judging the correctness of modified annotations, constructing an automatic error correction mechanism based on the correctness judgment result, and correcting metadata errors, including:

[0145] The classifier model is trained to evaluate the accuracy of the modified annotations. An automatic error correction mechanism is built based on the modified annotations that exceed the preset accuracy threshold to perform metadata error correction.

[0146] In this embodiment, the classifier model is a machine learning model used to classify input data into different categories. In this scenario, the classifier model is used to evaluate the correctness of the modified annotation. For example, the model type could be a Support Vector Machine (SVM), Random Forest, or a deep learning model (such as LSTM or BERT). A model is trained to determine whether the modified annotation is correct; for example, given a modified annotation, the model outputs "correct" or "incorrect".

[0147] In this embodiment, the preset accuracy threshold refers to the minimum accuracy score set during the evaluation process. Only when the accuracy of a modified annotation exceeds this threshold will it be considered valid. For example, the threshold is set to 80%. If the model evaluates the accuracy of a modified annotation to 75%, the annotation is marked as "incorrect"; if the accuracy is 85%, it is marked as "correct".

[0148] The working principle and beneficial effects of the above technical solution are as follows: the correctness of the modified annotation is evaluated by training a classifier model, a preset correctness threshold is set, and only when the correctness of the modified annotation exceeds the threshold will the system include it in the automatic error correction mechanism, thereby correcting the metadata and ensuring that high-quality modified annotations are effectively utilized.

[0149] Example 8:

[0150] This invention provides a metadata error correction system, such as... Figure 2 As shown, it includes:

[0151] Classification module: Collects metadata to be cleaned, performs preliminary cleaning of the data, classifies the metadata based on the cleaning results, and obtains structured data and unstructured data;

[0152] Error detection module: Performs semantic understanding analysis on structured data and unstructured data respectively, and detects a first potential error in the structured data and a second potential error in the unstructured data based on the results of the semantic understanding analysis;

[0153] Annotation module: Creates a user interface to display the first and second potential errors, collects third potential errors marked by users through the user interface, and modifies the first, second, and third potential errors with annotations;

[0154] Error correction module: The module judges the correctness of the modified annotations, constructs an automatic error correction mechanism based on the correctness judgment results, and corrects errors in the metadata.

[0155] The working principle and beneficial effects of the above technical solution are as follows: by collecting and initially cleaning metadata, classifying it into structured and unstructured data, performing semantic analysis on the two types of data, identifying potential errors, displaying errors through the user interface, allowing users to mark and annotate, judging the correctness of modifications based on user feedback, and building an automatic error correction mechanism, thereby improving the quality and consistency of metadata. The automatic error correction mechanism reduces manual intervention, effectively handles the mixed use of structured and unstructured data, and improves efficiency.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for correcting metadata errors, characterized in that, include: Step 1: Collect the metadata to be cleaned, perform preliminary cleaning on the data, and classify the metadata according to the cleaning results to obtain structured data and unstructured data; Step 2: Perform semantic understanding analysis on the structured data and unstructured data respectively. Based on the results of the semantic understanding analysis, detect the first potential error in the structured data and the second potential error in the unstructured data. Step 3: Create a user interface to display the first and second potential errors. Users can mark the third potential error through the user interface and make modification comments on the first, second, and third potential errors. Step 4: Determine the correctness of the modified annotations, and construct an automatic error correction mechanism based on the correctness determination results to correct the metadata.

2. The metadata error correction method according to claim 1, characterized in that, Collect metadata to be cleaned, perform preliminary data cleaning, and classify the metadata based on the cleaning results to obtain structured and unstructured data, including: Determine the source of the metadata, and based on the source, clarify the data format and storage method of the metadata; The metadata is initially cleaned according to the data format and storage method. After the initial cleaning, the cleaning effect is evaluated to determine whether a second cleaning is needed, and then the cleaning result is obtained. The data type of each metadata field is determined based on the cleaning results, and the degree of structure of each metadata field is judged based on the data type. Metadata is categorized based on its degree of structuring to determine whether it is structured or unstructured data.

3. The metadata error correction method according to claim 1, characterized in that, Semantic understanding analysis is performed on both structured and unstructured data. Based on the results of the semantic understanding analysis, a first potential error in the structured data and a second potential error in the unstructured data are detected, including: Perform a first semantic understanding analysis on the structured data to obtain the first semantic understanding analysis result, and determine the first potential error based on the first semantic understanding analysis result; Perform second semantic understanding analysis on unstructured data to obtain the second semantic understanding analysis results, and determine the second potential error based on the second semantic understanding analysis results.

4. The metadata error correction method according to claim 1, characterized in that, Perform first semantic understanding analysis on structured data to obtain first semantic understanding analysis results, and determine first potential errors based on the first semantic understanding analysis results, including: For each field of the structured data, write a detailed description, analyze the position of the field in the business process based on the detailed description, and determine the first relationship between the fields based on the position; Based on the first relationship, a table formed by fields is determined, the primary key of the table is identified, and according to the identified primary key, the field in the secondary table corresponding to the primary key field of the primary table is found to confirm the foreign key; Based on the primary key and foreign key, the functional dependencies of the basic fields are clearly defined, and then the secondary relationship between the fields is determined. Logical relationships are identified in the fields based on the first and second relationships. The first semantic understanding analysis result of the structured data is obtained by combining the first and second relationships with the logical relationships. The first potential error is determined based on the first semantic understanding analysis result.

5. A metadata error correction method according to claim 1, characterized in that, Perform second semantic understanding analysis on unstructured data to obtain the second semantic understanding analysis results, and determine the second potential errors based on the second semantic understanding analysis results, including: The unstructured data is reclassified to obtain text data, image data, and audio data; Extract keywords from text data, identify the corresponding text topics in unstructured data based on the keywords, and use sentiment analysis tools to evaluate the sentiment tendency of the text data based on the text topics to obtain the sentiment tendency evaluation results. Semantic consistency checks are performed on the text topic, sentiment consistency is assessed using the sentiment assessment results, similarity detection and processing are performed on the text data, and potential text errors are derived by combining the consistency check results, consistency assessment results, and processing results. Identify the subject and scene in image data, extract image features based on the subject and scene, perform image error identification based on the image features, and derive potential image errors; The audio data is segmented to obtain audio text data and audio sound data. A first sentiment assessment result is obtained based on the analysis of the text data. A second sentiment assessment result is obtained by analyzing the tone of voice of the audio sound data. Potential audio errors are identified based on the first sentiment assessment result and the second sentiment assessment result. The second potential error is derived by combining the potential text errors, potential image errors, and potential audio errors.

6. The metadata error correction method according to claim 1, characterized in that, Create a user interface that displays the first and second potential errors. The user can use the user interface to mark a third potential error and add comments to the first, second, and third potential errors, including: After comparing the first potential error with the second potential error, the user identifies the third potential error; For the third potential error, a superordinate error label is made for the first potential error and the second potential error, and a first modification comment is made for the superordinate error label; At the same time, a second modification annotation is made for the first potential error, the second potential error, and the third potential error.

7. A metadata error correction method according to claim 6, characterized in that, The correctness of the modified annotations is assessed, and an automatic error correction mechanism is constructed based on the accuracy assessment results to correct errors in the metadata, including: The classifier model is trained to evaluate the accuracy of the modified annotations. An automatic error correction mechanism is built based on the modified annotations that exceed the preset accuracy threshold to perform metadata error correction.

8. A metadata error correction system, characterized in that, include: Classification module: Collects metadata to be cleaned, performs preliminary cleaning of the data, classifies the metadata based on the cleaning results, and obtains structured data and unstructured data; Error detection module: Performs semantic understanding analysis on structured and unstructured data respectively, and detects a first potential error in the structured data and a second potential error in the unstructured data based on the results of the semantic understanding analysis; Annotation module: Creates a user interface to display the first and second potential errors, collects third potential errors marked by users through the user interface, and modifies the annotations of the first, second, and third potential errors; Error correction module: The module judges the correctness of the modified annotations, constructs an automatic error correction mechanism based on the correctness judgment results, and corrects errors in the metadata.

Images