Data cleaning system and method for inconsistent remote sensing image attribute information

Abstract

The invention discloses a data cleaning system and method for inconsistent remote sensing image attribute information in the technical field of satellite surveying and mapping. The method comprises the following steps: acquiring multi-source remote sensing image data; carrying out attribute information analysis based on the multi-source and multi-source remote sensing image data and the product content thereof; extracting attribute information of the analyzed multi-source remote sensing image data; performing inconsistency detection on the extracted attribute information of the multi-source remote sensing image data, and identifying and positioning inconsistent data; and repairing inconsistent data by mining a frequent item set and an association rule in the data. According to the method, inconsistent remote sensing image metadata, database field tables and the like are effectively detected and repaired through technical means such as an inconsistent data cleaning method based on the maximum dependency set and attribute association, the inconsistent processing efficiency of attribute information such as the metadata, the database field tables and the like is further improved, and remote sensing image storage, application and management are supported.

Description

Technical Field

[0001] This invention relates to a data cleaning system and method for addressing inconsistencies in the attribute information of remote sensing images, belonging to the field of satellite mapping technology. Background Technology

[0002] Remote sensing satellite data is image data generated by acquiring electromagnetic wave reflection and emission characteristics of objects on the Earth's surface through remote sensing equipment onboard satellites. It includes latitude and longitude information, and the raw data is unlabeled. Satellite image resolution is typically between 1 kilometer and 1 meter, and orbital altitudes cover a range from 4,000 kilometers to 600 kilometers. According to application fields, they are divided into three categories: land resource satellites, meteorological satellites, and ocean satellites.

[0003] Existing methods for cleaning inconsistencies in remote sensing image metadata, database field tables, and other attribute information may face performance bottlenecks when processing large-scale data, and the repair costs are high. Furthermore, data inconsistency issues are particularly prominent in multi-source data fusion and heterogeneous data processing, leading to increased complexity in data processing and analysis. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a data cleaning system and method for inconsistent remote sensing image attribute information, so as to realize the unified management and efficient utilization of multi-source remote sensing data and ensure the consistency and integrity of the data.

[0005] To achieve the above objectives, the present invention is implemented using the following technical solution: In a first aspect, the present invention provides a data cleaning method for addressing inconsistencies in remote sensing image attribute information, including: Acquire multi-source remote sensing image data; Attribute information analysis based on multi-source remote sensing image data and its product content; Extract attribute information from the multi-source remote sensing image data obtained after parsing; Inconsistency detection is performed on the extracted attribute information of multi-source remote sensing image data to identify and locate inconsistent data. Inconsistent data can be repaired by mining frequent itemsets and association rules in the data.

[0006] Furthermore, attribute information analysis is performed based on multi-source remote sensing image data and its product content, including: collecting attribute information according to the remote sensing data and its product content, and obtaining complete information of relevant metadata and database tables. The attribute information includes remote sensing image metadata and database field tables. The remote sensing image metadata includes sensor type, band, resolution, time, and spatial information.

[0007] Furthermore, the attribute information of the multi-source remote sensing image data obtained after parsing is extracted, including: metadata XML file information extraction and database field table information extraction. The metadata XML file is processed by a fixed program to obtain the element attribute information in the metadata XML file according to the metadata standard structure; the database field table information extraction is performed by automatically collecting data information through an automatic algorithm.

[0008] Furthermore, the extraction of metadata XML file information includes: Get the specified XML file path, load the XML file, parse the file into XML root elements, read the XML file content from the storage medium, and verify the file format and integrity. Use an XML parser to process text content and build a document object model tree structure, i.e., an XML file tree structure. Determine the node selection method and perform node traversal, including: applying XPath or selector expressions to locate the target set of nodes; and traversing the XML file tree structure to access parent nodes, child nodes, and sibling nodes. Extract node data text / attributes / tags to obtain node content, which includes sensor type, band, resolution, time, and spatial information; Extract structured data, obtain attribute data, and output lists, data frames, and other formats, while releasing resources; The extraction of database field table information includes: obtaining database field table information using SQL queries / database management tools / programming languages' database-included automatic algorithms, performing queries through the database connection library of the programming language Python, and obtaining table and column information by querying the database's metadata.

[0009] Furthermore, inconsistency detection is performed on the extracted multi-source remote sensing image data attribute information to identify and locate inconsistent data. This includes: defining a conditional dependency function, introducing constants and semantic binding, and performing data consistency detection. The defined conditional dependency function is as follows: In the formula: Φ is the definition of the conditional dependency function; R represents the relation schema; It is a standard functional dependency, where X and Y For attribute set; It is a pattern tuple; The introduction of constants and semantic binding includes: based on the attribute information of remote sensing image metadata and database field tables, defining consistency rules by introducing constants and semantically related data binding, mapping semantically related attributes from different data sources, and constraining their values ​​through conditional dependency function rules. The semantic mapping function f is: In the formula: A collection of metadata attributes for remote sensing images; A collection of database field attributes; The conditional dependency function rules are as follows: In the formula: Φ2 represents the conditional dependency function rule, DataSource represents the data source, and a d Represents a database field attribute; it is a collection of database field attributes. A single attribute in It is a pattern tuple; The data consistency detection includes: detecting the consistency of attribute information, including remote sensing image metadata and database field tables; and using SQL technology or static analysis tools to detect tuples in the data that violate the conditional dependency function rules, thereby identifying inconsistencies in the data.

[0010] Furthermore, by mining frequent itemsets and association rules in the data, inconsistent data is repaired, including: Data preprocessing and cleaning are performed on attribute information, including remote sensing image metadata and database field tables. The Apriori algorithm is used to mine frequent itemsets in data and identify frequently occurring attribute combinations. Generate association rules from frequent itemsets to discover potential relationships in the data; Inconsistent data is repaired based on the discovered association rules and frequent itemsets.

[0011] Furthermore, the Apriori algorithm is used to mine frequent itemsets in the data and identify frequently occurring attribute combinations. This includes defining a support quantifying the frequency of an itemset in the dataset, and setting a minimum support threshold. The support calculation formula is as follows: In the formula: support(x) is the attribute value. Support level; This indicates that the values ​​in the dataset are... The number of records; N is the total number of records in the dataset; Generate association rules from frequent itemsets to discover potential relationships in the data, including: measuring the strength of association rules using confidence scores, whereby the confidence score is calculated using the following formula: In the formula: confidence(A→B) is the confidence level of A→B. A Under what conditions does it occur? B The confidence level of occurrence; It is the frequency of itemsets A and B appearing simultaneously; It is the frequency of itemset A appearing alone; Based on the discovered association rules and frequent itemsets, inconsistent data is repaired, including: for missing values, inference is made based on strong association rules; for outliers, correction is made based on strong association rules.

[0012] Secondly, the present invention provides a data cleaning system for addressing inconsistencies in remote sensing image attribute information, comprising: Data acquisition module: used to acquire multi-source remote sensing image data; Data parsing module: used to parse attribute information based on multi-source remote sensing image data and its product content; Data extraction module: used to extract attribute information from the parsed multi-source remote sensing image data; Data detection module: used to detect inconsistencies in the attribute information of the extracted multi-source remote sensing image data, and to identify and locate inconsistent data; Data Repair Module: Used to repair inconsistent data by mining frequent itemsets and association rules in the data.

[0013] Thirdly, the present invention provides a data cleaning apparatus for addressing inconsistencies in remote sensing image attribute information, including a processor and a storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any of the foregoing.

[0014] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0015] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: This solution analyzes several key steps in cleaning inconsistencies in remote sensing image metadata and database field tables, including data acquisition, extraction, verification, and repair, forming a complete workflow for cleaning inconsistencies in remote sensing image attribute information. This method is not only applicable to cleaning remote sensing image attribute information but can also be extended to data cleaning tasks in other fields, demonstrating high practicality and promotional value. Furthermore, it improves data processing efficiency and accuracy through automated algorithms and standardized procedures; ensures data consistency and comparability through metadata standard structures and rule matching; and enhances the intelligence level of data cleaning through association and repair rules. This systematic solution provides reliable data support for subsequent data processing, analysis, and application. Attached Figure Description

[0016] The accompanying drawings, which form part of this specification, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart illustrating the data cleaning method for inconsistent attribute information in remote sensing images provided in Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of metadata XML file extraction for the data cleaning method for inconsistent attribute information in remote sensing images provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram illustrating the repair of attribute information such as remote sensing image metadata, database field tables, etc., in the data cleaning method for inconsistent remote sensing image attribute information provided in Embodiment 1 of the present invention. Detailed Implementation

[0017] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0018] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this invention is for describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0019] To address the need for cleaning inconsistencies in attribute information from remote sensing image metadata and database fields / tables, this solution designs a data cleaning method for inconsistencies in remote sensing image attribute information. This method enables the checking, verification, and repair of inconsistencies in attribute information from remote sensing image metadata and database fields / tables, providing assurance for subsequent data cleaning. The data cleaning process for inconsistencies in remote sensing image attribute information includes: attribute information acquisition, attribute structure verification, attribute information extraction, attribute information verification, attribute information matching, and checking and repair. The overall technical approach is as follows: (1) First, attribute information parsing. Based on the remote sensing data and its product content, the attribute information such as metadata, database fields and tables is parsed.

[0020] (2) Then, attribute information extraction. Remote sensing image attribute information extraction includes metadata XML file information extraction and database table information extraction. Among them, the metadata XML file needs to be obtained by a fixed program according to the metadata standard structure to obtain the element attribute information in the metadata XML file; the database field table information extraction is carried out by automatically collecting data information through automatic algorithms.

[0021] (3) Secondly, inconsistency detection of attribute information content. Inconsistency detection is performed on the content of the extracted metadata, database fields and tables and other attribute information. By introducing constants on the basis of functional dependency through conditional dependency functions, semantically related data are bound together to detect the consistency of remote sensing image metadata, database fields and tables and other attribute information.

[0022] (4) Finally, attribute information is supplemented and repaired. For cases where inconsistencies are found in the attribute information of remote sensing image metadata, database fields, and tables, the Apriori algorithm is used to mine frequent itemsets and association rules in the data to discover potential relationships and repair the inconsistent data. This solution will be described in further detail below.

[0023] Example 1: During the acquisition, processing, storage, and application of remote sensing image data, inconsistencies in metadata, database fields, and table attributes are prone to occur due to diverse data sources, complex processing procedures, and insufficient data quality control. These issues not only affect data usability but may also lead to a decrease in the accuracy of subsequent analysis and applications. This paper proposes a data cleaning method to address inconsistencies in remote sensing image attribute information. This method utilizes conditional dependency functions to apply integrity constraints, deriving inconsistency detection rules for data inconsistency detection. Furthermore, it employs association rules and other methods to formulate corresponding rules for repairing inconsistent data.

[0024] Based on the above analysis, this invention designs a data cleaning method for inconsistencies in remote sensing image attribute information, such as... Figure 1 As shown, the specific steps are as follows: S1: Attribute Information Parsing. Based on the remote sensing data and its product content, attribute information such as metadata and database fields / tables is collected to obtain complete information about the relevant metadata and database tables. Metadata for remote sensing data is typically contained in data files (such as header files) or metadata files. For example, the metadata of remote sensing imagery includes sensor type, band, resolution, time, and spatial information, specifically related to the satellite itself, the satellite constellation (single satellite or satellite formation), and the satellite product level.

[0025] S2: Attribute Information Extraction. Remote sensing image attribute information extraction includes metadata XML file information extraction and database field / table information extraction. The metadata XML file requires a standardization process to extract element attribute information according to the metadata standard structure; database field / table information extraction uses automated algorithms to automatically collect data.

[0026] S2-1: Metadata XML File Extraction. This uses open-source XML parsing libraries, such as those provided by languages ​​like R, C, and Python, to automatically collect metadata information. For example... Figure 2 As shown, the steps for extracting metadata from XML files include: S2-1-1: Load XML file. Obtain the specified XML file path, load the XML file, parse the file into XML root elements, read the XML file content from the storage medium, and verify the file format and integrity. S2-1-2: Parsing XML text to build an in-memory tree structure. Using an XML parser (DOM / SAX) to process text content and build a Document Object Model (DOM) tree structure; S2-1-3: Select the node positioning method. Determine the node selection method, such as XPath expressions or CSS selectors, and perform node traversal; S2-1-4: Path matching. Using XPath or selector expressions to locate the target set of nodes; S2-1-5: Path traversal (traversing parent-child and sibling nodes). Traversing the XML file tree structure, visiting parent, child, and sibling nodes; S2-1-6: Extract node data text / attributes / tags. Obtain node content, including sensor type, band, resolution, time, spatial information, etc. S2-1-7: Attribute Data Organization and Output. Extract structured data, obtain attribute data, and output lists, data frames, and other formats, freeing up resources.

[0027] S2-2: Extracting Database Field Table Information. This involves using automated algorithms included in SQL queries, database management tools, and programming languages ​​to retrieve database field table information. This includes querying using Python's database connection library and retrieving table and column information from database metadata (such as the `information_schema` table).

[0028] S3: Inconsistency Detection of Attribute Information. This function detects inconsistencies in the extracted metadata, database fields, and other attribute information. The aim is to identify and locate attributes from different data sources (such as remote sensing image metadata XML files and database tables) that should be semantically consistent but actually have inconsistent values. For example... Figure 3As shown, the sensor attribute value in the remote sensing image metadata XML is "MSS", while the corresponding SENSORID field value in the database table is "CCD", indicating inconsistency in attribute information. This solution introduces Conditional Functional Dependency (CFD), which, based on traditional functional dependencies, introduces constants and semantic binding to detect the consistency of attribute information in remote sensing image metadata, database fields, and tables.

[0029] S3-1: Define Conditional Dependency Functions (CFDs). Conditional Dependency Functions (CFDs) enhance the ability of traditional functional dependencies to constrain data consistency by introducing pattern tuples to express semantic relationships. The definition form Φ of a CFD is: In the formula: R represents the relation schema; It is a standard functional dependency, where X and Y For attribute set; It is a schema tuple used to specify the value constraints of a data tuple when certain conditions are met. The values ​​can be constants or unspecified variables. Defines the relationship between the value of a data tuple on X and... During matching, its value in Y must be equal to .

[0030] The following CFD rule Φ1 can be defined: In the formula: MetadataTable represents the metadata table (the table that stores metadata information), Source represents the data source, and SensorType represents the sensor type. A specific value indicating the source of the data (such as "XML format"). This indicates a specific value representing the sensor type (e.g., a specific sensor model). The rule states that when the data source is... At that time, the sensor type (SensorType) must be... .

[0031] S3-2: Introducing Constants and Semantic Bindings. Based on the attribute information of remote sensing image metadata and database field tables, consistency rules are defined by introducing constants and semantically related data bindings. Specifically, semantically related attributes from different data sources are mapped, and their values ​​are constrained by CFD rules.

[0032] Let the set of metadata attributes for remote sensing images be... The database field attribute set is The semantic mapping function f is: For each pair of mapping attributes ,in ,and The CFD rule Φ2 can be defined as follows: In the formula: DataSource represents the data source, that is, the source that provides remote sensing image metadata or database field data (such as a specific remote sensing satellite, database system, etc.), a d Represents a database field attribute; it is a collection of database field attributes. A single attribute (such as the definition of a specific field) in the data. It is a pattern tuple; where, (Constant), or its value should be specified to be equal to the corresponding attribute value in the metadata through another relation, i.e. In the formula: This represents a data source identifier, indicating a specific data source. `c` is a constant, and `Da` is the data source identifier. d Represents database fields Metadata association attribute identifier (used to associate attributes with remote sensing image metadata, such as the "resolution field" corresponding to the "resolution" attribute of the metadata), a m This represents the metadata attributes of remote sensing images. Elements in the data (i.e., a certain attribute in the remote sensing image data, such as "image resolution" or "capture time").

[0033] S3-3: Data Consistency Detection. This involves detecting the consistency of attribute information in remote sensing image metadata, database fields, and tables. Using SQL techniques or static analysis tools, it identifies tuples in the data that violate CFD rules, thus recognizing inconsistencies. Inconsistencies are mainly categorized into two forms: null or non-existent attribute values, and outliers. Specifically, it addresses situations where an attribute value does not conform to the constraints defined by CFD rules. If the matching pattern conditions are determined but the value constraints are not met, the query is executed, and the returned result is a tuple that violates the CFD rules, that is, there is inconsistent data.

[0034] It should be noted that, in order to accurately identify deep inconsistencies in multi-source data and avoid the missed detections caused by simple rule matching, this solution further introduces a weighted scoring formula for inconsistency detection to detect inconsistencies. The weighted scoring formula for inconsistency detection is as follows: In the formula: InconsistencyScore(a i ,b j ) represents the weighted score for inconsistency detection, a ib is the set of the i-th attribute in the metadata; j For the j-th corresponding attribute set in the database table; sim(a i ,b j ) represents attribute similarity, using cosine similarity or edit distance; θ k w is the similarity threshold for attribute k; k The weight of attribute k is set according to the importance of the business. This is an indicator function; its value is 1 when x < θ, and 0 otherwise. A weighted inconsistency score is calculated for all attribute pairs. If the total score exceeds a preset threshold, it is marked as inconsistent. For example, if the sensor type weight w1 = 0.6 and the resolution weight w2 = 0.4, and the sensor similarity is 0 (complete mismatch), the inconsistency score is 0.6 × 1 + 0.4 × 0 = 0.6, exceeding the threshold of 0.5, triggering the repair process.

[0035] S4: Attribute Information Repair. For any inconsistencies discovered, the Apriori algorithm is used to mine frequent itemsets and association rules in the data to uncover potential relationships. Based on this, inconsistencies are repaired, such as... Figure 3 As shown, attribute information repair specifically includes: S4-1: Data Preprocessing and Cleaning. Preprocessing of remote sensing image metadata, database fields, tables, and other attribute information, including: S4-1-1: Remove missing values. Delete or temporarily exclude records containing missing values; S4-1-2: Handling outliers. Temporarily store or correct obvious outliers based on business logic or statistical methods (such as the standard deviation principle); ensure data quality, and use the cleaned data for subsequent frequent itemset mining.

[0036] S4-2: Mining Frequent Itemsets. Using the Apriori algorithm, we mine frequent itemsets in the data, identifying frequently occurring attribute combinations, including: S4-2-1: Define support. Support is used to quantify how frequently an itemset appears in a dataset. For attribute values... The formula for calculating its support is: In the formula: support(x) is the attribute value. Support level; This indicates that the values ​​in the dataset are... The number of records; N is the total number of records in the dataset.

[0037] S4-2-2: Set the minimum support threshold. This is used to filter frequent itemsets. If the support of itemset X satisfies... If X is a frequent itemset, then X is called a frequent itemset. The Apriori algorithm generates all frequent itemsets iteratively. (in (item set size).

[0038] S4-3: Generate Association Rules. Generate association rules from frequent itemsets to discover potential relationships in the data. The form of the association rules is as follows: A → B ,in A and B These are disjoint itemsets. The strength of a rule is measured by its confidence score, which indicates that... A Under what conditions does it occur? B The probability of occurrence. The formula for calculating the confidence level is: In the formula: confidence(A→B) is the confidence level of A→B. A Under what conditions does it occur? B The confidence level of occurrence; It is the frequency of itemsets A and B appearing simultaneously; This represents the frequency of itemset A occurring alone. Only items with a confidence level higher than a preset threshold are retained. Strong association rules.

[0039] S4-4: Data Repair. Based on the discovered association rules and frequent itemsets, repair inconsistent data to improve data consistency.

[0040] S4-4-1: Missing value imputation. For missing values, apply the strong association rule. A → B Inference is made that if the value of a record in attribute set A matches the antecedent of the rule, but attribute B is missing, then the value of B can be filled in to match the value of the consequent of the rule according to the rule.

[0041] S4-4-2: Outlier Correction. Outliers are corrected according to strong association rules. If a record's value in attribute set A matches the antecedent of a rule, but the value of attribute B does not match the consequent of a rule, then the value of B can be corrected to the value corresponding to the consequent of the rule.

[0042] It should be noted that the formula for evaluating repair priority is as follows: In the formula: Priority(r) is the repair priority; r is the association rule; confidence(r) is the confidence level; Support(r) is the support level; Cost(r) is the repair cost (such as the complexity of field modification); ϵ is a constant (to avoid division by zero errors). After the repair priority is evaluated, rules with higher priority are executed first. For example, correcting the sensor type has a higher priority than correcting the resolution (because the latter has a lower repair cost).

[0043] In summary, the above steps enable consistency detection and repair of attribute information such as metadata and database fields in remote sensing images, significantly improving data quality and reliability.

[0044] Example 2: The data cleaning system for inconsistent remote sensing image attribute information can implement the data cleaning method for inconsistent remote sensing image attribute information described in Embodiment 1, including: Data acquisition module: used to acquire multi-source remote sensing image data; Data parsing module: used to parse attribute information based on multi-source remote sensing image data and its product content; Data extraction module: used to extract attribute information from the parsed multi-source remote sensing image data; Data detection module: used to detect inconsistencies in the attribute information of the extracted multi-source remote sensing image data, and to identify and locate inconsistent data; Data Repair Module: Used to repair inconsistent data by mining frequent itemsets and association rules in the data.

[0045] Example 3: This invention also provides a data cleaning apparatus for inconsistent remote sensing image attribute information, which can implement the data cleaning method for inconsistent remote sensing image attribute information described in Embodiment 1, including a processor and a storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the following method: Acquire multi-source remote sensing image data; Attribute information analysis based on multi-source remote sensing image data and its product content; Extract attribute information from the multi-source remote sensing image data obtained after parsing; Inconsistency detection is performed on the extracted attribute information of multi-source remote sensing image data to identify and locate inconsistent data. Inconsistent data can be repaired by mining frequent itemsets and association rules in the data.

[0046] Example 4: This invention also provides a computer-readable storage medium that can implement the data cleaning method for inconsistent attribute information of remote sensing images as described in Embodiment 1. The medium stores a computer program that, when executed by a processor, performs the following steps: Acquire multi-source remote sensing image data; Attribute information analysis based on multi-source remote sensing image data and its product content; Extract attribute information from the multi-source remote sensing image data obtained after parsing; Inconsistency detection is performed on the extracted attribute information of multi-source remote sensing image data to identify and locate inconsistent data. Inconsistent data can be repaired by mining frequent itemsets and association rules in the data.

[0047] As is known from common technical knowledge, this invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative and not exhaustive. All modifications within the scope of this invention or its equivalents are included in this invention.

[0048] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0049] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0050] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0051] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A data cleaning method for addressing inconsistencies in remote sensing image attribute information, characterized by: include: Acquire multi-source remote sensing image data; Attribute information analysis based on multi-source remote sensing image data and its product content; Extract attribute information from the multi-source remote sensing image data obtained after parsing; Inconsistency detection is performed on the extracted attribute information of multi-source remote sensing image data to identify and locate inconsistent data. Inconsistent data can be repaired by mining frequent itemsets and association rules in the data.

2. The data cleaning method for inconsistent attribute information in remote sensing images according to claim 1, characterized in that, Attribute information analysis based on multi-source remote sensing image data and its product content includes: collecting attribute information according to the remote sensing data and its product content, and obtaining complete information of relevant metadata and database tables. The attribute information includes remote sensing image metadata and database field tables. The remote sensing image metadata includes sensor type, band, resolution, time, and spatial information.

3. The data cleaning method for inconsistent attribute information in remote sensing images according to claim 1, characterized in that, The attribute information of the multi-source remote sensing image data obtained after parsing is extracted, including: metadata XML file information extraction and database field table information extraction. The metadata XML file is obtained by a fixed program according to the metadata standard structure to obtain the element attribute information in the metadata XML file; the database field table information extraction is carried out by automatically collecting data information through an automatic algorithm.

4. The data cleaning method for inconsistent attribute information in remote sensing images according to claim 3, characterized in that, The extraction of metadata (XML) file information includes: Get the specified XML file path, load the XML file, parse the file into XML root elements, read the XML file content from the storage medium, and verify the file format and integrity. Use an XML parser to process text content and build a document object model tree structure, i.e., an XML file tree structure. Determine the node selection method and perform node traversal, including: applying XPath or selector expressions to locate the target set of nodes; and traversing the XML file tree structure to access parent nodes, child nodes, and sibling nodes. Extract node data text / attributes / tags to obtain node content, which includes sensor type, band, resolution, time, and spatial information; Extract structured data, obtain attribute data, and output lists, data frames, and other formats, while releasing resources; The extraction of database field table information includes: obtaining database field table information using SQL queries / database management tools / programming languages' database-included automatic algorithms, performing queries through the database connection library of the programming language Python, and obtaining table and column information by querying the database's metadata.

5. The data cleaning method for inconsistent attribute information in remote sensing images according to claim 1, characterized in that, Inconsistency detection is performed on the extracted attribute information of multi-source remote sensing image data to identify and locate inconsistent data. This includes: defining a conditional dependency function, introducing constants and semantic binding, and data consistency detection. The defined conditional dependency function is as follows: In the formula: Φ is the definition of the conditional dependency function; R represents the relation schema; It is a standard functional dependency, where X and Y For attribute set; It is a pattern tuple; The introduction of constants and semantic binding includes: based on the attribute information of remote sensing image metadata and database field tables, defining consistency rules by introducing constants and semantically related data binding, mapping semantically related attributes from different data sources, and constraining their values ​​through conditional dependency function rules. The semantic mapping function f is: In the formula: A collection of metadata attributes for remote sensing images; A collection of database field attributes; The conditional dependency function rules are as follows: In the formula: Φ2 represents the conditional dependency function rule, DataSource represents the data source, and a d Represents a database field attribute; it is a collection of database field attributes. A single attribute in It is a pattern tuple; The data consistency detection includes: detecting the consistency of attribute information, including remote sensing image metadata and database field tables; and using SQL technology or static analysis tools to detect tuples in the data that violate the conditional dependency function rules, thereby identifying inconsistencies in the data.

6. The data cleaning method for inconsistent attribute information in remote sensing images according to claim 1, characterized in that, Inconsistent data can be repaired by mining frequent itemsets and association rules in the data, including: Data preprocessing and cleaning are performed on attribute information, including remote sensing image metadata and database field tables. The Apriori algorithm is used to mine frequent itemsets in data and identify frequently occurring attribute combinations. Generate association rules from frequent itemsets to discover potential relationships in the data; Inconsistent data is repaired based on the discovered association rules and frequent itemsets.

7. The data cleaning method for inconsistent attribute information in remote sensing images according to claim 6, characterized in that, The Apriori algorithm is used to mine frequent itemsets in data and identify frequently occurring attribute combinations. This includes defining a support quantifying the frequency of an itemset in the dataset, and setting a minimum support threshold. The support calculation formula is as follows: In the formula: support(x) is the attribute value. Support level; This indicates that the values ​​in the dataset are... The number of records; N is the total number of records in the dataset; Generate association rules from frequent itemsets to discover potential relationships in the data, including: measuring the strength of association rules using confidence scores, whereby the confidence score is calculated using the following formula: In the formula: confidence(A→B) is the confidence level of A→B. A Under what conditions does it occur? B The confidence level of occurrence; It is the frequency of itemsets A and B appearing simultaneously; It is the frequency of itemset A appearing alone; Based on the discovered association rules and frequent itemsets, inconsistent data is repaired, including: for missing values, inference is made based on strong association rules; for outliers, correction is made based on strong association rules.

8. A data cleaning system for remote sensing image attribute information with inconsistencies, characterized in that: include: Data acquisition module: used to acquire multi-source remote sensing image data; Data parsing module: used to parse attribute information based on multi-source remote sensing image data and its product content; Data extraction module: used to extract attribute information from the parsed multi-source remote sensing image data; Data detection module: used to detect inconsistencies in the attribute information of the extracted multi-source remote sensing image data, and to identify and locate inconsistent data; Data Repair Module: Used to repair inconsistent data by mining frequent itemsets and association rules in the data.

9. A data cleaning device for addressing inconsistencies in remote sensing image attribute information, characterized in that: Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 7.

Images