A material coding unified mapping and repeated correction method supporting multi-source heterogeneous ERP

Abstract

The present application relates to material coding technical field, specifically, it relates to a kind of material coding unified mapping and repeated correction method supporting multi-source heterogeneous ERP.The present application includes the following steps: multi-source heterogeneous ERP correlation data acquisition;Multi-source data standardization preprocessing;Material coding intelligent unified mapping construction;Mapping code repeated detection and conflict correction;Full-link data persistence and incremental update.The present application distinguishes the structured and unstructured features of material coding by double-branch decoupling, avoids noise superposition, accurately captures material subtle differences to improve mapping accuracy;While reusing historical mapping experience to build incremental memory bank, avoid full-amount standard library traversal comparison, significantly reduce redundant calculation, improve mapping efficiency under mass data.In addition, based on the dynamic evolution characteristics extracted from the whole life cycle log of coding, the stability factor is used for differential determination, which effectively distinguishes stable and temporary coding, reduces the risk of mis-mapping and subsequent business data misuse.

Description

Technical Field

[0001] This invention relates to the field of material coding technology, and more specifically, to a method for unified mapping and duplicate correction of material codes that supports multi-source heterogeneous ERP systems. Background Technology

[0002] As enterprises advance their digital transformation, multiple heterogeneous ERP systems have long coexisted, resulting in significant differences in the composition rules, attribute fields, and description methods of material codes across these systems. To build a unified enterprise-level material master data view, it is necessary to map material data and code data from various heterogeneous sources to a standard material code library through adaptation, feature matching, and other means.

[0003] However, existing mapping schemes rely solely on the static features of material code strings and descriptive text for similarity calculation, resulting in two significant drawbacks: First, they fail to effectively distinguish and utilize the multi-dimensional heterogeneous information contained in material codes from different sources and with different structures. The undifferentiated mixing of structured code fragments with strict logical constraints and flexible text descriptions leads to mutual confusion and noise superposition during similarity calculations, making it difficult to capture subtle differences between materials and limiting matching accuracy. Simultaneously, each mapping operation requires independent comparison across the entire standard library, failing to accumulate and reuse the association knowledge and judgment experience formed during historical mapping processes. This results in a large amount of redundant calculations when dealing with massive amounts of material data, and mapping efficiency continuously declines as the data scale increases. Second, they completely ignore the dynamic evolution of material codes during business processes—creation, modification, deactivation, and cancellation—using a single fixed matching threshold. This fails to distinguish between long-term stable core material codes and frequently changing temporary, low-quality codes, easily leading to incorrect mapping of high-frequency change codes. This not only reduces the overall accuracy of code mapping but also causes misuse and coordination chaos in subsequent inventory, procurement, and production business data. In view of this, we propose a method for unified mapping and duplicate correction of material codes that supports multi-source heterogeneous ERP systems. Summary of the Invention

[0004] The purpose of this invention is to provide a method for unified mapping and repetitive correction of material codes in multi-source heterogeneous ERP systems, in order to solve the problems mentioned in the background art, such as the mixed processing of multi-dimensional heterogeneous information, the inability to reuse historical mapping experience, and the neglect of the dynamic evolution behavior of material codes, which leads to insufficient mapping accuracy, efficiency and reliability.

[0005] To address the aforementioned technical problems, this invention provides a method for unified mapping and duplicate correction of material codes in multi-source heterogeneous ERP systems, comprising the following steps: S1. Multi-source heterogeneous ERP related data collection: Using multi-protocol data adaptation and access technology, it synchronously collects material basic data, multi-source heterogeneous material coding data and coding full life cycle traceable operation log data from various heterogeneous ERP systems. S2. Multi-source data standardization preprocessing: Using structured and unstructured unified standardization technology, the collected material basic data, multi-source heterogeneous material coding data and operation log data are respectively processed for format unification, outlier removal and time dimension normalization. S3. Intelligent unified mapping construction of material codes: The dual-branch heterogeneous feature decoupling-incremental memory fusion mapping algorithm is adopted to establish the mapping relationship between multi-source heterogeneous material codes and standard material codes. Among them, multi-source heterogeneous material codes that reach the final judgment threshold in terms of comprehensive similarity score and have no conflict are unidirectionally and uniquely mapped to a standard material code. The coding evolution stability factor is introduced to dynamically adjust the comprehensive similarity judgment threshold. S4. Duplicate Detection and Conflict Correction after Mapping: Based on the uniqueness constraint of the standard material coding library, feature fingerprint comparison technology is used to detect duplicate and conflicting items in the mapped code, and hierarchical conflict resolution technology is used to complete automatic correction and manual review and diversion. S5. End-to-end data persistence and incremental update: Adopting distributed data persistence technology, the unified mapping relationship, the corrected standard coding data, coding evolution stability data and incremental feature memory are synchronously stored in the standard material coding master database, and real-time incremental updates are supported.

[0006] As a further improvement to this technical solution, the multi-protocol data adaptation access technology in S1 specifically includes: Based on RESTful API interface adaptation, it calls the material data service interface opened by various heterogeneous ERP systems to pull material basic data and multi-source heterogeneous material code data in real time. Based on JDBC database direct connection adaptation, by configuring ERP backend database connection permissions, we can selectively read the original data of multi-source heterogeneous material codes and full life cycle operation logs. Based on Web Service interface adaptation, a data interaction channel is established with the ERP system through SOAP protocol to obtain the externally released multi-source heterogeneous material codes and change records; Based on ETL batch data extraction and adaptation, material data files exported from the ERP system are captured at preset time intervals to complete offline batch data collection of multi-source heterogeneous material codes.

[0007] As a further improvement to this technical solution, the multi-source data standardization preprocessing in step S2 includes the following steps: S21. Perform field mapping alignment, character set unification, and coding rule standardization on material basic data and multi-source heterogeneous material coding data; S22. Identify and remove invalid data that is missing key attributes, has contradictory data logic, or is redundant and repetitive in the coding of multi-source heterogeneous materials; S23. Convert the operation log time records of various heterogeneous ERP systems into a standard timestamp format to achieve time base unification.

[0008] As a further improvement to this technical solution, in step S3, the execution process of the dual-branch heterogeneous feature decoupling-incremental memory fusion mapping algorithm includes the following steps: S31. For the preprocessed multi-source heterogeneous material coding data, perform decoupling extraction and quantization coding of structured features and unstructured text features; S32. Perform fast candidate retrieval of encoded structured features based on incremental feature memory; S33. Weighted fusion of the structured features and unstructured text features of the code to be matched and the candidate standard code to calculate the comprehensive similarity score; S34. Combine the coding evolution stability factor to complete the dynamic similarity threshold determination and output the final mapping matching result.

[0009] As a further improvement to this technical solution, in step S31, the process of decoupling and extracting structured features from unstructured text features and performing quantization encoding includes the following steps: S31.1 Extracting structured feature vectors of category, specification, material, and unit dimensions from multi-source heterogeneous material coding data. Extract core semantic unstructured text feature vectors from material description texts; S31.2, Use the Locality Sensitive Hashing algorithm to process structured feature vectors Encode to generate a unique structured hash identifier. ; S31.3. Perform L2 norm normalization on the unstructured text feature vectors to obtain standardized unstructured text feature vectors.

[0010] As a further improvement to this technical solution, in step S32, the process of quickly retrieving candidates from the encoded structured features based on the incremental feature memory includes the following steps: S32.1, a structured hash identifier for matching multi-source heterogeneous material codes. Use the key as the retrieval key to perform a similarity search in the incremental feature memory; S32.2 Filter out the preset number of candidate standard codes with the highest similarity and proceed to the subsequent fusion matching stage; S32.3, The structured hash identifier of the newly added and successfully matched multi-source heterogeneous material codes. Write the corresponding standard encoding identifier into the incremental feature memory; S32.4. Perform an overwrite update on entries with the same hash identifier that already exist in the memory.

[0011] As a further improvement to this technical solution, in step S33, the process of weightedly fusing the structured features and unstructured text features of the code to be matched and the candidate standard code to calculate the comprehensive similarity score includes the following steps: S33.1 Calculate the structured feature similarity and unstructured text feature similarity between the code of the multi-source heterogeneous material to be matched and each candidate standard code; S33.2 Calculate the dynamic fusion weights of structured features and unstructured text features through a cross-attention mechanism; S33.3. The similarity scores of structured features and unstructured text features are weighted and summed according to the dynamic fusion weights to obtain the comprehensive similarity score.

[0012] As a further improvement to this technical solution, in step S34, the process of determining the dynamic similarity threshold by combining the coding evolution stability factor and outputting the final mapping matching result includes the following steps: S34.1 Extract all change records of the multi-source heterogeneous material codes to be matched within the preset historical period from the standardized operation log data; S34.2 Calculate the three core statistics of the code of the multi-source heterogeneous material to be matched in the historical period: number of changes, average change interval, and cumulative usage time. S34.3 Calculate the coding evolution stability factor using the weighted normalization method. Its value range is strictly limited to [0,1]; S34.4, Based on coding evolution stability factor The final judgment threshold is calculated by comparing the base judgment threshold of the material category to which the code to be matched belongs. Furthermore, it satisfies a monotonically decreasing relationship where higher coding stability leads to a lower final judgment threshold. S34.5, Final Judgment Threshold Apply global upper and lower bound constraints to avoid misjudgments caused by extreme values; S34.6. Combine the comprehensive similarity score with the final judgment threshold. Perform comparisons and determine the matching results; S34.7, If the coding evolution stability factor If the value is below the preset lower limit, the code of the multi-source heterogeneous material to be matched will be marked as pending review.

[0013] As a further improvement to this technical solution, in step S4, the process of detecting duplicates and correcting conflicts after mapping includes the following steps: S4.1 Generate structured feature fingerprints and unstructured text feature fingerprints corresponding to the multi-source heterogeneous material codes after mapping; S4.2 Based on the uniqueness constraint of the standard material coding library, the generated feature fingerprint is compared with the existing standard coding feature fingerprints in the library to identify duplicates with completely identical feature fingerprints and conflicting items with partially matched feature fingerprints. S4.3 For duplicate entries with completely identical feature fingerprints, the newly generated mapping relationship is automatically retained and marked as confirmed, while other redundant mapping entries are deleted. S4.4 For conflict terms with a feature fingerprint similarity higher than a preset conflict threshold, extract the standardized operation log data encoded by both conflicting parties, and calculate their encoding evolution stability factors respectively. ; S4.5, Coding evolution stability factor when the two conflicting parties' codes When the difference is greater than the preset correction threshold, the code with higher stability is automatically selected as the final standard mapping result, while the code with lower stability is marked as discarded. S4.6, Coding evolution stability factor when the two conflicting parties' codes When the difference is less than or equal to the preset correction threshold, the conflict item and the corresponding operation log data will be pushed to the manual review interface. S4.7. The final mapping relationship, which has been automatically corrected or manually reviewed and confirmed, will be synchronously updated to the standard material code library and the incremental feature memory.

[0014] As a further improvement to this technical solution, the process of end-to-end data persistence and incremental update in S5 includes the following steps: S5.1 Classify, package, and format-verify the unified mapping relationship, corrected standard coding data, coding evolution stability data, and incremental feature memory change data generated by this mapping and correction process; S5.2. Distributed multi-replica synchronous write technology is adopted to synchronously store various types of data that have passed verification into the corresponding data partition of the standard material code master database. S5.3 Real-time monitoring of data change events in the standard material code master database to trigger incremental update processes; S5.4 Extract the timestamp and version number of the changed data, and push the incremental changed data to the local cache of all related business systems in a synchronized manner; S5.5 Perform a data consistency check between the standard material code master database and the cache of each related business system. If data inconsistency is found, trigger a full synchronization correction.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention effectively distinguishes and utilizes heterogeneous information from different dimensions in material coding, avoiding confusion and noise superposition between structured coding fragments and text descriptions during similarity calculation. This enables the matching process to capture subtle differences between materials, thereby improving mapping accuracy. At the same time, by accumulating and reusing the associated knowledge and judgment experience formed in the historical mapping process, it avoids traversing the entire standard library for independent comparison in each mapping, reducing redundant calculations under massive material data and improving mapping efficiency. 2. This invention collects and utilizes the full lifecycle operation log data of material codes to extract the dynamic evolution characteristics of the codes. Based on this, it applies differentiated mapping judgment strategies to material codes with different levels of stability, so as to effectively distinguish between core codes that have been used stably for a long time and temporary codes that are frequently changed. This avoids high-frequency change codes being incorrectly mapped, thereby improving mapping reliability and reducing the risk of misuse of subsequent business data such as inventory, procurement, and production caused by incorrect mapping. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall method steps for the unified mapping and duplicate correction method for material codes that supports multi-source heterogeneous ERP in this invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, this embodiment provides a method for unified mapping and duplicate correction of material codes in multi-source heterogeneous ERP systems, including the following steps: S1. Multi-source heterogeneous ERP related data collection: Using multi-protocol data adaptation and access technology, it synchronously collects material basic data, multi-source heterogeneous material coding data and coding full life cycle traceable operation log data from various heterogeneous ERP systems. In this step, S1, the multi-protocol data adaptation access technology specifically includes: Based on RESTful API interface adaptation, this method calls the material data service interfaces exposed by various heterogeneous ERP systems to retrieve basic material data and multi-source heterogeneous material code data in real time. Specifically, for ERP systems that provide RESTful data services, after configuring the corresponding authentication credentials, their basic material data service interface and material code data service interface are called to retrieve key fields such as material code, material description, basic unit of measurement, and material group in real time. This method has minimal invasiveness to the source system, high data real-time performance, and is suitable for near real-time data synchronization scenarios.

[0019] Based on JDBC direct database connection adaptation, this method allows for targeted reading of raw data from multi-source heterogeneous material codes and their full lifecycle operation logs by configuring connection permissions to the ERP backend database. Specifically, the full lifecycle operation logs for material codes are typically stored in the audit log table of the ERP backend database and are not exposed externally through business interfaces. This approach, with read-only database access, uses configured JDBC connection parameters to target and read operation records from the material master table, version revision table, and audit log table. The data collection task is scheduled to execute during off-peak business hours, and connection pool management and data compression transmission are enabled to keep the impact of the collection process on the source database performance within preset thresholds.

[0020] Based on Web Service interface adaptation, a data interaction channel is established with the ERP system via SOAP protocol to obtain externally published multi-source heterogeneous material codes and change records. Specifically, for ERP systems that publish data services via SOAP Web Service, a data interaction channel is established by parsing their WSDL files, and material code query services and material change record query services are called periodically. During the call, an incremental timestamp is used as the request parameter to retrieve only the material codes that have changed since the last synchronization window and a snapshot of the content before and after the change, avoiding the redundancy overhead of transmitting the entire data.

[0021] Based on ETL batch data extraction adaptation, material data files exported from the ERP system are periodically captured at preset time intervals to complete offline batch data collection of multi-source heterogeneous material codes. Specifically, for older system modules that lack real-time interface conditions or are located in network isolation environments, ETL batch data extraction is used as a supplementary solution. The ETL scheduling engine connects to the designated file servers of each system at preset time intervals to capture CSV or XML format material data files exported by the system periodically, completing offline batch collection. This link is configured with breakpoint resume and file integrity verification mechanisms to ensure the reliable and complete transmission of batch data.

[0022] In this step, the operation log data specifically includes a full lifecycle traceable record of the material code's creation time, modification records, code content before and after changes, activation status, deactivation time, and deletion marker. Specifically, the operation log data has specific structured requirements. A single log record contains the following traceable fields: material code creation time, modification records (including change time, operator, and change type), code content before and after changes, activation status indicator, deactivation time, and logical deletion marker. This full lifecycle traceable record is the foundational data source for calculating the code evolution stability factor in the subsequent S3 step. Specifically, in S3, the system extracts three core statistics from the operation log: the number of changes to the code to be matched within a preset historical period, the average change interval, and the cumulative usage time. These are then weighted and normalized to obtain the code evolution stability factor, which directly affects the dynamic adjustment of the mapping judgment threshold. Therefore, the completeness and accuracy of the operation log collection in this step directly determine the reliability of the subsequent mapping judgment.

[0023] S2. Multi-source data standardization preprocessing: Using structured and unstructured unified standardization technology, the collected material basic data, multi-source heterogeneous material coding data and operation log data are respectively processed for format unification, outlier removal and time dimension normalization. In this step, the multi-source data standardization preprocessing in S2 includes the following steps: S21. Perform field mapping alignment, character set unification, and encoding rule standardization for basic material data and multi-source heterogeneous material coding data. Specifically, the character set will uniformly adopt UTF-8 encoding, and heterogeneous character encodings such as GBK, ISO-8859-1, and EBCDIC in the collected data will be batch converted. During the conversion process, for characters that cannot be mapped, an exception log will be recorded and replaced with standard placeholders according to a preset strategy.

[0024] Furthermore, since material codes themselves have different formats due to varying coding rules across different systems, this step performs rule standardization processing on the coding field. Specifically, this includes: removing leading and trailing whitespace and invisible control characters; unifying letter case; completing or trimming placeholders for fixed-length codes according to preset rules; and converting full-width characters to half-width characters. The standardized code serves as the standard input identifier for subsequent processing.

[0025] S22. Identify and remove invalid data that is missing key attributes, has contradictory data logic, or is redundant and repetitive in the coding of multi-source heterogeneous materials; Specifically, records lacking key attribute data are checked by iterating through the set of required fields defined in the standard data model. Fields with empty values, containing only blank characters, or containing invalid system default values ​​are all considered missing. Such records do not provide sufficient matching features and are marked as invalid data and moved to the isolated data area.

[0026] Records with logical inconsistencies are detected using pre-defined cross-field validation rules. For example, if there is a pre-defined correspondence between material categories and primary units of measurement, a logical inconsistency is triggered when a record's material category is "plate" while its primary unit of measurement is "kilogram." The rule engine performs validation on all records one by one, marking records that match the inconsistency rules as invalid data and moving them to the isolated data area.

[0027] Redundant and duplicate records are identified through key comparison. The uniqueness is determined by the combination of the source system identifier, material code, and data acquisition timestamp. For multiple identical records, only the most recently acquired one is retained, and the rest are discarded.

[0028] The data marked as invalid is not deleted directly, but moved to the isolated data area for unified storage, for use in data quality analysis and subsequent manual verification. At the same time, the system generates a data quality report, recording the source system distribution, the number of invalid data removed, and the reasons for removal.

[0029] S23. Convert the operation log time records of various heterogeneous ERP systems into a standard timestamp format to achieve time base unification.

[0030] Specifically, the time string formats of various heterogeneous systems are parsed and uniformly converted to UTC standard timestamps with millisecond precision. For time fields that do not carry time zone information, compensation conversion is performed based on the default time zone parameters configured in the source system. Time fields that fail to convert are marked as abnormal, and the timestamp of the data acquisition task execution time is used as the replacement value, while an error log is recorded.

[0031] The normalized operation log time field is stored in a unified UTC timestamp format to eliminate time zone and format differences, providing a reliable data source with a consistent time base for calculating the coding evolution stability factor in the subsequent S3 step.

[0032] Through this step, the problems of format differences, low-quality data, and inconsistent time bases in multi-source heterogeneous data are systematically addressed, producing a standardized dataset with aligned fields, unified character sets, controlled data quality, and consistent time bases, providing reliable data input for the construction of intelligent unified mapping of material codes in step S3.

[0033] S3. Intelligent Unified Mapping Construction of Material Codes: A dual-branch heterogeneous feature decoupling-incremental memory fusion mapping algorithm is adopted to establish a mapping relationship between multi-source heterogeneous material codes and standard material codes. Multi-source heterogeneous material codes that achieve a comprehensive similarity score reaching the final judgment threshold and are conflict-free are unidirectionally and uniquely mapped to a standard material code. A code evolution stability factor is introduced to dynamically adjust the comprehensive similarity judgment threshold. In S3, the execution process of the dual-branch heterogeneous feature decoupling-incremental memory fusion mapping algorithm includes the following steps: In this step, S31, the preprocessed multi-source heterogeneous material coding data undergoes decoupling extraction and quantization encoding of structured features and unstructured text features. The process of decoupling extraction and quantization encoding of structured features and unstructured text features in S31 includes the following steps: S31.1 Extracting structured feature vectors of category, specification, material, and unit dimensions from multi-source heterogeneous material coding data. Extract core semantic unstructured text feature vectors from material description texts; Specifically, in the structured feature branch, structured feature vectors are extracted from the multi-source heterogeneous material coding data. . The dimensions are determined by a preset standard attribute dimension table, in this embodiment. It consists of four core dimensions: material category, specifications, material grade, and basic unit of measurement, and is represented as follows: ; in, Standard material category codes, This is a specification parameter string. For standard material codes, Standard units of measurement are used for coding. After the field mapping alignment and coding rule standardization processing in step S2, the values ​​of each dimension have been unified under the standard data model: material categories are coded according to the standard classification system; numerical specifications in the specification parameter string have been converted to floating-point values ​​under the standard units of measurement, and character specifications have undergone space removal, case unification, and full / half-width conversion; material grades are mapped to unified codes through a lookup table; and units of measurement have been unified to standard unit codes.

[0034] In the unstructured text feature branch, core semantic feature vectors are extracted from the material description text. The input text is the material description field content after S2 step standardization. Feature extraction is performed using a pre-trained language model BERT fine-tuned with enterprise material domain corpus. The fine-tuning corpus consists of material description texts from the enterprise's historical material master data. The fine-tuning task is the joint training of a masked language model and a next-sentence prediction task. The material description text is input into the fine-tuned BERT model, and the hidden state vector corresponding to the [CLS] label output by the last Transformer layer of the model is taken as the aggregated semantic representation of the text to obtain the unstructured text feature vector. . The dimension of is determined by the hidden layer dimension of the BERT model, denoted as . Taking the BERT-base model as an example, If the length of the descriptive text exceeds the maximum input length of the model, a truncation strategy is adopted, retaining the beginning and end parts, prioritizing the integrity of key specification information at the beginning and supplementary information at the end.

[0035] S31.2, Use the Locality Sensitive Hashing algorithm to process structured feature vectors Encode to generate a unique structured hash identifier. ; Specifically, firstly, Values ​​of each dimension The strings are concatenated in a fixed order to form a characteristic string, and then this string is processed. Split and generate shingle sets ,in For the shingle length, this embodiment takes... , The number of shingles obtained from the segmentation.

[0036] Secondly, select There are families of hash functions, each family containing ... A total of 1 independent random hash function, There are hash functions, denoted as . Each hash function maps a string to an integer value. For a set... Each element in Calculate its hash value, take the minimum value under each hash function, and obtain the minimum hash signature vector. : ; Then, the minimum hash signature vector Divided into order Each band contains [number] bands. The signature value. For the first... Within each band Signature value Calculate the hash value to obtain the band hash value. : ; in This is the preset hash function. The hash values ​​of each band together constitute the structured hash identifier of this material code. : ; parameter and The configuration affects the recall and precision of the search: Increasing the size of the index improves the recall rate but increases the retrieval time. Increasing the size of the hash code library makes the hash collision condition more stringent, improves precision, but decreases recall. This embodiment, based on the size of the standard material coding library and retrieval efficiency requirements, selects... , That is, bucketing is performed based on the minimum hash signature space composed of 100 hash functions.

[0037] S31.3. Perform L2 norm normalization on the unstructured text feature vectors to obtain standardized unstructured text feature vectors.

[0038] Specifically, the text feature vector output by the BERT model The values ​​for each dimension have inconsistent distribution ranges, and directly using them for similarity calculation will lead to some dimensions dominating the results. This sub-step addresses... L2 norm normalization is performed, mapping it to the unit hypersphere. The specific implementation process is as follows: set up In the The components of dimension are , ,but L2 norm The calculation is as follows: ; Normalized unstructured text feature vectors for: ; After normalization The vector direction remains unchanged. In subsequent similarity calculations, the vector dot product can be directly used as the cosine similarity, reducing computational overhead.

[0039] After processing steps S31.1 to S31.3, each multi-source heterogeneous material code data is converted into a binary tuple. ,in For structured hash identifiers, This is a standardized unstructured text feature vector. This pair serves as the input for subsequent candidate retrieval in S32 and fusion matching in S33.

[0040] In this step, S32, fast candidate retrieval is performed on the encoded structured features based on the incremental feature memory; Understandably, this sub-step utilizes structured hash identifiers. Rapid candidate retrieval is performed in the incremental feature memory, and a controllable set of candidate standard codes is selected from the standard material code library. This avoids the one-to-one comparison of all codes in the subsequent S33 step, thereby compressing the computational scale of similarity calculation from the millions to a preset constant level.

[0041] The incremental feature memory is an index structure that maintains the mapping between structured hash identifiers and standard encoded identifiers. Each record in the memory consists of a key-value pair: the key is the structured hash identifier. A band hash value The value is a list storing all standard material code identifiers with that band hash value. The memory is built in batches from historically confirmed standard material codes during the initial system deployment and is incrementally updated according to the rules in S32.3 and S32.4 after each mapping process is completed.

[0042] In step S32, the process of quickly retrieving candidate features based on the incremental feature memory includes the following steps: S32.1, a structured hash identifier for matching multi-source heterogeneous material codes. Use the key as the retrieval key to perform a similarity search in the incremental feature memory; Specifically, Include Each band hash value, i.e. The search process is as follows: Will In Each band hash value is used as a query key to search for a matching record in the incremental feature memory. For any band hash value... band hash value Retrieve the list of standard encoding identifiers corresponding to the band from the memory. All The lists returned by the queries are merged to obtain a multiset of candidate standard encoding identifiers.

[0043] For each standard coded identifier in the multiset, its statistics are calculated. The frequency of occurrence in the query. A higher frequency indicates that the structured features of the standard code and the code to be matched have hash collisions across more bands, meaning the greater the likelihood of similar structured features. The formula for calculating the frequency is: ; in, Indicates the first One candidate standard coding identifier, For the first Candidate list corresponding to each band This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. express.

[0044] S32.2 Filter out the preset number of candidate standard codes with the highest similarity and proceed to the subsequent fusion matching stage; Specifically, based on the frequency statistics obtained in S32.1, the candidate standard codes are sorted and filtered. All candidate standard codes with a collision frequency ≥ 1 are selected. If the number of candidates exceeds a preset upper limit... Sort them in descending order of frequency, and select the first one with the highest frequency. A candidate standard code, denoted as the candidate set. . In this embodiment, the maximum number of candidates is preset. .

[0045] If, after searching, no candidate standard codes are found (i.e., all...), (All empty) indicates that there are no historical mapping records in the incremental feature memory that are similar to the current coding structure to be matched, so the candidate set will be... If the set is set to empty, the code to be matched bypasses the S33 fusion matching step and directly enters the S34 step, where it is judged or marked as pending review based on the default highest threshold.

[0046] Candidate set generated through the above retrieval and filtering As input to the subsequent S33 fusion matching stage, it works in conjunction with the standardized unstructured text feature vector produced in step S31. Complete the calculation of the overall similarity score.

[0047] S32.3, The structured hash identifier of the newly added and successfully matched multi-source heterogeneous material codes. Write the corresponding standard encoding identifier into the incremental feature memory; Specifically, after the mapping result is determined in step S34 and the conflict detection and correction in step S4 are confirmed, the structured hash identifier of the newly added and successfully matched multi-source heterogeneous material code in this process is... Corresponding standard coding identifier Write to the incremental feature memory to achieve incremental updates of the memory, thereby gradually improving the retrieval efficiency of subsequent identical or similar structural feature codes.

[0048] The writing method is: to of band hash value Each one is used as a key to identify the standard encoding. Add to the list corresponding to each key.

[0049] S32.4. Perform an overwrite update on entries with the same hash identifier that already exist in the memory.

[0050] Specifically, for entries with the same hash identifier that already exist in the memory, that is, the list corresponding to a certain band hash value already contains the standard encoded identifier to be written. Instead of performing duplicate appends, an overwrite update is performed: the latest timestamp and status information of the entry are preserved to avoid the retrieval performance degraded due to the infinite growth of the list.

[0051] The specific operation of overwrite update is: locate the matching elements that already exist in the list. The system updates the last hit timestamp of each entry to the current system time and resets its aging count. This timestamp information is used for the periodic maintenance of the memory: the system periodically scans all entries in the memory and deletes entries whose last hit timestamp is earlier than the current time minus the preset aging period (in this embodiment, the aging period is set to 180 days), thus controlling the storage space and retrieval latency of the memory within a reasonable range.

[0052] Through processes S32.1 to S32.4, the incremental feature memory completes the candidate retrieval, candidate filtering, and its own incremental update loop for the codes to be matched. The candidate set output in step S32 is... The standardized unstructured text feature vector output by step S31 Together, they serve as input for step S33, entering the weighted fusion and matching stage of structured features and unstructured text features.

[0053] In this step, S33, the structured features and unstructured text features of the code to be matched and the candidate standard codes are weighted and fused to calculate the comprehensive similarity score; specifically, the candidate standard code set output by step S32... The standardized unstructured text feature vector to be matched and encoded by the output of step S31 In this step, the overall similarity score is calculated together. For For each candidate standard code, the system calculates its structured feature similarity and unstructured text feature similarity with the code to be matched, and dynamically determines the fusion weight of the two types of similarity through a cross-attention mechanism. Finally, a weighted sum is obtained to obtain the comprehensive similarity score. All codes in the standard material coding library have completed the feature extraction and quantization coding processing described in S31 simultaneously upon being added to the library. Their structured feature vectors, minimum hash signature vectors, and standardized unstructured text feature vectors have been persistently stored and are directly called in this step. The specific implementation includes the following steps: S33.1 Calculate the structured feature similarity and unstructured text feature similarity between the code of the multi-source heterogeneous material to be matched and each candidate standard code; Specifically, for the multi-source heterogeneous material codes to be matched With candidate standard coding set any candidate code in Calculate the structural feature similarity between the two respectively. Similarity with unstructured text features The details are as follows: Structured feature similarity The calculation is based on the Jaccard similarity coefficient. Let the code to be matched be... The structured feature vector is Candidate coding The structured feature vector is .Will The values ​​of each dimension are converted into tokens according to the following rules: Material Category Code As a whole, it is treated as a token; specification parameter string By character level Segmentation ( =3 (consistent with S31.2) Generate a token set; Material encoding As a whole, it is treated as a token; unit of measurement is encoded. As a whole, each token is treated as a single token. The union of all tokens constitutes the structured feature set of this encoding. Let... and Let each represent a set of structured feature tokens for the two encodings. The calculation formula is: ; The value range is [0,1]. The larger the value, the higher the overlap between the two material codes in terms of structured attributes such as category, specification, material, and unit.

[0054] Unstructured text feature similarity The calculation is based on cosine similarity. The encoding to be matched... The standardized unstructured text feature vector is Candidate coding The corresponding vector is Both are and satisfy Cosine similarity can be calculated directly using the vector dot product: ; in , They are respectively The Dimensional components. The range of values ​​is To make its value range similar to the structured similarity. Unify to the same dimension range, for Perform a linear mapping to obtain the text similarity after mapping. : ; .

[0055] S33.2 Calculate the dynamic fusion weights of structured features and unstructured text features through a cross-attention mechanism; Specifically, fixed weights cannot accommodate the differences in the reliability of structured and textual information among different material codes. This sub-step introduces a cross-attention mechanism to address the differences in the reliability of the codes to be matched. With candidate coding The structured feature embedding vector and text feature vector are used as inputs to dynamically calculate the fusion weight of the two types of similarity. . The fusion weight for structured similarity has a value range of (0,1). The fusion weights are used for text similarity. The specific implementation process is as follows: First, encode the code to be matched. With candidate coding The structured features are transformed into dense vector representations. The minimum hash signature vector generated for each code in step S31.2 is used as the numerical representation of the structured features. Let the code... The minimum hash signature vector is ,coding The minimum hash signature vector is ,in The minimum hash signature vector is represented in the th order. The value of dimension, The total dimension of the signature vector. The number of bands in the LSH bucketing parameters. The number of hash functions included in each band, in this embodiment , , 100. Through a trainable linear mapping layer... Mapped to the same as the text feature vector In 3D space, we obtain structured embedding vectors. : ; ; in, Representation encoding Structured embedding vectors, Representation encoding Structured embedding vectors, For the mapping weight matrix, For bias vectors, Represents the real number field. and They share the same mapping layer parameters.

[0056] The text feature vectors are directly obtained using the normalized vectors obtained in step S31.3. and .

[0057] Based on this, a cross-attention weight calculation module is constructed. A learnable parameter matrix is ​​introduced. and These are used to generate the query vector and key vector, respectively. The two cross-attention scores are calculated: First cross score : encoding to be matched Structured embedding for queries, with candidate encoding The text features are keys: ; Second Cross Score : encoding to be matched The text features are used as the query, with candidate encoding Structured embedding as keys: ; In the above two formulas, Representation encoding Structured attributes and encoding Cross-modal consistency scores between text descriptions Representation encoding Text description and encoding Cross-modal compatibility scores between structured attributes. Scaling factor The vector dimension is used to prevent the dot product value from becoming too large, which could cause the gradient of the subsequent sigmoid function to saturate. and This comprehensively reflects the overall consistency of cross-modal features between the code to be matched and the candidate code.

[0058] Will and The weights are added together and mapped to the (0,1) interval using the sigmoid function to obtain the dynamic fusion weights. : ; in, This represents the sigmoid function. It is a natural constant.

[0059] S33.3. The similarity scores of structured features and unstructured text features are weighted and summed according to the dynamic fusion weights to obtain the comprehensive similarity score.

[0060] Specifically, based on the dynamic fusion weights obtained in S33.2 The structured feature similarity calculated for S33.1 Similarity with mapped text Perform a weighted summation to obtain the code to be matched. With candidate coding Overall similarity score : ; in, Representation encoding With encoding The comprehensive similarity score, obtained through dynamic weight fusion, ranges from [0,1]. When the cross-modal feature consistency between the code to be matched and the candidate code is strong, [further information is needed]. Larger values ​​indicate a greater emphasis on accurate matching of structured features in the overall score; however, when cross-modal consistency is weak or there are signs of inconsistency between description and attributes, The values ​​are relatively small, and the overall score depends more on the similarity of the text semantics.

[0061] For candidate set Each candidate code in ( , (The upper limit of the number of candidates preset in step S32.2) is used to calculate its match with the code to be matched according to the above process. Overall similarity score All the scores obtained, along with the corresponding candidate code identifiers, are used as inputs for the dynamic threshold determination and mapping results output in step S34.

[0062] In this step, S34, the dynamic similarity threshold is determined by combining the coding evolution stability factor, and the final mapping matching result is output. The core mechanism of this step lies in introducing the coding evolution stability factor to quantitatively evaluate the historical change behavior of the coding of the multi-source heterogeneous materials to be matched, and dynamically adjust the similarity determination threshold based on this: the more stable the coding history, the lower the determination threshold, and the confidence window of automatic mapping is appropriately widened; the more unstable the coding history, the higher the determination threshold, the stricter the conditions for automatic mapping, and manual review is required when necessary. The specific implementation steps are as follows: S34.1 Extract all change records of the multi-source heterogeneous material codes to be matched within the preset historical period from the standardized operation log data; Specifically, the codes of the multi-source heterogeneous materials to be matched are extracted from the standardized preprocessed operation log data in step S2. In the preset historical period All change records within. This indicates the length of the time window, in days. In this embodiment... (i.e., two years). The extracted change records include the timestamp, change type, and snapshots of the coded content before and after the change for each modification record within that period.

[0063] S34.2 Calculate the three core statistics of the code of the multi-source heterogeneous material to be matched in the historical period: number of changes, average change interval, and cumulative usage time. Specifically, statistics are performed on all change records extracted in S34.1, and the following three core statistics are calculated: Number of changes : Code to be matched In historical cycles The total number of code attribute changes that occurred within the record. Code attribute changes include material description modifications, specification parameter adjustments, material changes, unit changes, and status changes (enable / disable). If multiple attributes change simultaneously in the same record, it is counted as one change.

[0064] Average change interval In historical cycles Within this range, the average time interval between two consecutive changes, expressed in days. Assume change records are sorted in ascending order by time, with the timestamp of the change occurring as follows: Then the adjacent change interval ,in and The first Next and first The timestamp of the last change. For the first The average change interval is calculated as follows: when , ; when (No attribute changes) or When the code has not changed within the historical period or has only been initially created once, and there is no adjacent change interval to calculate, then... Values (i.e., the length of the historical cycle).

[0065] Total usage time : Code to be matched The cumulative number of days the code has been in an "enabled" state since its initial creation up to the current system time. Extract the creation time and all enabled / disabled records for this code from the operation log, add the enabled state duration to the total time period, subtract the disabled state period, and calculate in days. The calculation formula is: ; in, Indicates the first The start timestamp of the segment's enabled state. Indicates the first The end timestamp of the segment's enabled state. Iterate through each consecutive enabled state interval of the code.

[0066] S34.3 Calculate the coding evolution stability factor using the weighted normalization method. Its value range is strictly limited to [0,1]; Specifically, the three core statistics obtained in S34.2 are combined into a single scalar index, namely the coding evolution stability factor, through a weighted normalization method. , The range of its value is strictly limited to [0,1]. A larger value indicates a more stable historical state of the encoding. The specific implementation process is as follows: First, normalize the three statistics separately, mapping the statistics with inconsistent dimensions to the [0,1] interval: Number of changes Normalization: The more changes there are, the less stable the encoding becomes. Let... The preset upper limit for the number of changes, when Exceed The negative impacts are no longer distinguished. Normalized change frequency index. The calculation formula is: ; in, , As a preset constant, in this embodiment .

[0067] Average change interval Normalization: The longer the average change interval, the more stable the coding. Let... The preset reference interval number of days, when Exceed This is considered a completely stable dimension. Normalized average interval of change index. The calculation formula is: ; in, , As a preset constant, in this embodiment sky.

[0068] Total usage time Normalization: The longer the cumulative usage time, the higher the degree of practical verification of the code, and the higher its stability. Let... The preset reference usage duration in days, when Exceed This is considered a completely stable dimension. Normalization is used with duration metrics. The calculation formula is: ; in, , As a preset constant, in this embodiment Three years (i.e., three days).

[0069] Then, the three normalized indices are weighted and summed to obtain the coding evolution stability factor. : ; in, Normalized index for the number of changes The weight, Normalized index for average change interval The weight, Normalized index for cumulative usage time The weights. The weights satisfy the constraints. and The weight values ​​are determined based on statistical analysis of the company's historical coding data. In this embodiment, a typical setting is... .

[0070] The range of its value is [0, 1]. When When the code has remained unchanged or changed very little during the historical period, its use has been continuous and stable; when This indicates that the code changes frequently, has been used for a very short period of time, and its historical state is extremely unstable.

[0071] S34.4, Based on coding evolution stability factor The final judgment threshold is calculated by comparing the base judgment threshold of the material category to which the code to be matched belongs. Furthermore, it satisfies a monotonically decreasing relationship where higher coding stability leads to a lower final judgment threshold. Specifically, the coding evolution stability factor obtained based on S34.3 The basic threshold for determining the material category to which the code to be matched belongs. Calculate the final judgment threshold . The similarity threshold is a preset threshold for each category in the standard material classification system. Different thresholds can be set for different material categories due to differences in their attribute characteristics. Value, in this embodiment The typical value range is [0.75, 0.90].

[0072] Final determination threshold The coding stability must satisfy a monotonically decreasing relationship where the higher the coding stability, the lower the threshold. That is, stable coding can be automatically mapped with a relatively lenient threshold, while unstable coding requires a higher similarity score to pass the automatic judgment. The calculation formula is: ; in, The adjustment coefficient controls the magnitude of the stability factor's adjustment on the threshold. ; This is the stability benchmark value, representing when... hour That is, the stability factor does not have a regulatory effect.

[0073] In this embodiment , .when hour, The encoding enjoys a lower judgment threshold due to its stability; when hour, The encoding is subject to higher judgment standards due to its instability.

[0074] S34.5, Final Judgment Threshold Apply global upper and lower bound constraints to avoid misjudgments caused by extreme values; Specifically, the calculation obtained from S34.4 Apply global upper and lower bound constraints to prevent [the problem] due to [the factors that cause this]. Extreme values ​​cause the judgment threshold to deviate from a reasonable range. Let... This is the global lower threshold. The global upper limit threshold, after constraint for: ; In this embodiment 0.70 is acceptable. 0.98 is acceptable.

[0075] S34.6. Combine the comprehensive similarity score with the final judgment threshold. Perform comparisons and determine the matching results; Specifically, for the candidate set Each candidate standard code in The comprehensive similarity score calculated in step S33 is used to calculate the overall similarity score. With the final judgment threshold The comparison and judgment rules are as follows: Select the code with the highest overall similarity score from the candidate set. ,like If the match is successful, the code for the heterogeneous multi-source material to be matched will be entered. Uniquely mapped to standard material codes in a one-way manner. .

[0076] like If no matching item meets the condition in the current candidate set, the code AA to be matched will not establish any mapping relationship in this round of mapping process and will be marked as "unmatched".

[0077] like If the set is empty (i.e., no candidate codes were retrieved in step S32.2), it is directly judged as "not matched".

[0078] S34.7, If the coding evolution stability factor If the value is below the preset lower limit, the code of the multi-source heterogeneous material to be matched will be marked as pending review.

[0079] Specifically, after determining the matching result in S34.6, an additional check is performed on the coding evolution stability factor. If the code to be matched... Coding evolution stability factor Below the preset stability lower limit threshold Regardless of whether the code matches successfully in S34.6, it is additionally marked as "pending review" and pushed to the manual review interface for verification. In this embodiment... .

[0080] This measure establishes a safety net mechanism for coded data that undergoes frequent historical changes and whose data quality is questionable, requiring manual verification to confirm the correctness of its mapping relationships.

[0081] After processing steps S34.1 to S34.7, the multi-source heterogeneous material codes to be matched are obtained. The code is judged to be one of the following three results: a successful match and a one-way unique mapping with the standard code is established; a successful match but marked as pending review; or no match and marked as pending review. Codes that are successfully mapped but not marked as pending review proceed to step S4 for duplicate detection and conflict correction, while codes marked as pending review, along with their candidate score information, are pushed to the manual review queue.

[0082] S4. Duplicate Detection and Conflict Correction after Mapping: Based on the uniqueness constraint of the standard material coding library, feature fingerprint comparison technology is used to detect duplicate and conflicting items in the mapped code, and hierarchical conflict resolution technology is used to complete automatic correction and manual review and diversion. In this step, the process of post-mapping encoding duplicate detection and conflict correction in S4 includes the following steps: S4.1 Generate structured feature fingerprints and unstructured text feature fingerprints corresponding to the multi-source heterogeneous material codes after mapping; Specifically, for each multi-source heterogeneous material code successfully mapped as confirmed in step S3, its corresponding structured feature fingerprint and unstructured text feature fingerprint are generated. The specific implementation process is as follows: Structured fingerprint The generation method is as follows: take the structured feature vector constructed in step S31.1. The values ​​from each dimension are concatenated into a string in a fixed order, and a preset hash function is applied to that string. (In this embodiment) Using the SHA-256 algorithm, the hash value is used as the structured feature fingerprint. : ; in, This represents a string concatenation operation. This is a pre-defined cryptographic hash function.

[0083] Unstructured text feature fingerprint The generation method is as follows: take the standardized unstructured text feature vector from step S31.3. The components of each dimension are arranged in a fixed order and then quantized and encoded. This embodiment uses a binarization method based on a random hyperplane: introducing... a preset random unit vector each and .calculate With each random vector The dot product is used to generate a binary code based on its sign: ; in, Representing unstructured text feature fingerprints The Bits In this embodiment, the fingerprint bit length is... . for A binary bit string of 12 bits.

[0084] Each successfully mapped multi-source heterogeneous material code generates a pair of feature fingerprints. This serves as the basis for subsequent duplicate detection and conflict identification.

[0085] S4.2 Based on the uniqueness constraint of the standard material coding library, the generated feature fingerprint is compared with the existing standard coding feature fingerprints in the library to identify duplicates with completely identical feature fingerprints and conflicting items with partially matched feature fingerprints. Specifically, the feature fingerprint generated by S4.1 is compared with all existing standard coding feature fingerprints in the standard material coding library. Based on the comparison results, the mapping entries are divided into three categories: duplicate entries, conflict entries, and normal entries.

[0086] Let the currently successfully mapped multi-source heterogeneous material code be... The characteristic fingerprint is The feature fingerprint corresponding to a certain mapping record that already exists in the standard material coding library is .

[0087] The criteria for determining duplicate items are: and This means that the structured feature fingerprint and the unstructured text feature fingerprint are completely identical. This indicates that the encoding... With mapped encoding If the structured attributes and textual semantics are completely identical, they belong to the repeated mapping of the same standard encoding and should be deduplicated.

[0088] The criteria for determining conflict items are: but ,or However, the similarity between unstructured text feature fingerprints exceeds a preset conflict threshold. For unstructured text feature fingerprints... fingerprint similarity Hamming distance is used to calculate the number of bits that differ in corresponding bit values ​​between two bit strings. ; in, and Representing fingerprints and The bit value The length of the fingerprint bits. .

[0089] The comprehensive conditions for conflict determination are: ,in This is a preset conflict similarity threshold. When the similarity between two fingerprints exceeds this threshold but they are not completely identical, they are judged as conflicting entries, indicating that the encoding... With encoding While semantically highly similar, subtle differences exist, potentially leading to multiple codes for the same item or attribute ambiguity, requiring further resolution. In this embodiment... .

[0090] Mapping entries that do not meet the above criteria for determining duplicate or conflicting entries are considered normal entries, and their mapping relationships are directly confirmed and persisted.

[0091] S4.3 For duplicate entries with completely identical feature fingerprints, the newly generated mapping relationship is automatically retained and marked as confirmed, while other redundant mapping entries are deleted. Specifically, for duplicate entries with completely identical feature fingerprints identified by S4.2, automatic deduplication is performed. The most recently generated mapping relationship is retained, that is, the mapping entry generated in the most recent S3 mapping process is taken as the valid mapping and marked as "confirmed". At the same time, the remaining redundant mapping entries under the multi-source heterogeneous material code in the standard material code library are deleted to ensure that one multi-source heterogeneous material code corresponds to only one standard material code mapping relationship.

[0092] S4.4 For conflict terms with a feature fingerprint similarity higher than a preset conflict threshold, extract the standardized operation log data encoded by both conflicting parties, and calculate their encoding evolution stability factors respectively. ; Specifically, for the conflicting items identified by S4.2, the codes of both conflicting parties (i.e., the codes to be mapped) are extracted. Corresponding codes to existing mapping records in the standard material code library Standardized operation log data. Based on the methods described in S34.1 to S34.3, calculate the encoding respectively. Coding evolution stability factor With encoding Coding evolution stability factor Both have a range of [0,1].

[0093] S4.5, Coding evolution stability factor when the two conflicting parties' codes When the difference is greater than the preset correction threshold, the code with higher stability is automatically selected as the final standard mapping result, while the code with lower stability is marked as discarded. Specifically, when the difference in the coding evolution stability factor between the two conflicting codes is greater than a preset correction threshold, the system automatically selects the code with higher stability as the final standard mapping result, while marking the code with lower stability as "discarded".

[0094] Let the preset correction threshold be The absolute value of the difference in coding evolution stability factor is The newly added preset comprehensive similarity difference threshold is: The absolute value of the difference in the overall similarity score is ,in , They are respectively the codes ,coding The overall similarity score is calculated in step S33 with the corresponding standard code.

[0095] When the condition is met and The automatic correction rules are as follows: like and Then retain the encoding. The mapping relationship is used as the final mapping result, and the mapping relationship corresponding to the code B is marked as "discarded"; like and Then the encoding The mapping relationship remains valid, and the encoding will be maintained. The mapping relationship is marked as "obsolete".

[0096] When satisfied but If this happens, automatic correction will not be performed; instead, conflict items and corresponding operation log data will be pushed to the manual review interface.

[0097] In this embodiment , .

[0098] The logic behind automatic correction is that the coding evolution stability factor comprehensively reflects the frequency of historical changes, usage continuity, and data reliability of the coding. When there is a significant difference in stability between the conflicting parties, the data from the party with higher stability is more reliable, and the system makes an automatic decision based on this, reducing the cost of manual intervention.

[0099] S4.6, Coding evolution stability factor when the two conflicting parties' codes When the difference is less than or equal to the preset correction threshold, the conflict item and the corresponding operation log data will be pushed to the manual review interface. Specifically, when the difference in coding evolution stability factors between the two conflicting parties is less than or equal to a preset correction threshold, i.e. This indicates that the historical stability of both conflicting parties is comparable, and the system cannot make a reliable automatic decision based on stability differences. At this point, all information regarding the conflict item is pushed to the manual review interface, including: the coded content, feature fingerprints, comprehensive similarity scores, respective coding evolution stability factors, and original operation log summaries of both parties. Manual reviewers then make a final determination based on their business knowledge, confirming which party's mapping relationship to retain or whether to merge the attribute information of both parties.

[0100] S4.7. The final mapping relationship, which has been automatically corrected or manually reviewed and confirmed, will be synchronously updated to the standard material code library and the incremental feature memory.

[0101] Specifically, after automatic deduplication in S4.3, automatic correction in S4.5, or manual review and confirmation in S4.6, the final mapping relationship is synchronously updated to the standard material code library and the incremental feature memory.

[0102] The update operations for the standard material code library include: writing confirmed valid mapping relationships, marking obsolete mapping entries, and updating the status flags of the corresponding standard codes.

[0103] The update operation of the incremental feature memory includes: writing the structured hash identifier and standard code identifier of the newly confirmed mapping multi-source heterogeneous material code into the memory (executing the writing rule of S32.3); removing or marking the feature entries corresponding to the mapping relationship marked as "obsolete" from the incremental feature memory to avoid invalid mappings affecting the accuracy of subsequent candidate retrieval in S32.

[0104] Through this step, duplicate mappings and potential conflicts in the mapping results output by step S3 are systematically detected and resolved hierarchically. The uniqueness constraint of the standard material coding library is maintained, and a set of non-duplicate and conflict-free mapping relationships is finally produced, providing verified and accurate data for the end-to-end data persistence and incremental update of step S5.

[0105] S5. End-to-end data persistence and incremental update: Adopting distributed data persistence technology, the unified mapping relationship, the corrected standard coding data, coding evolution stability data and incremental feature memory are synchronously stored in the standard material coding master database, and real-time incremental updates are supported.

[0106] In this step, the process of end-to-end data persistence and incremental update in S5 includes the following steps: S5.1 Classify, package, and format-verify the unified mapping relationship, corrected standard coding data, coding evolution stability data, and incremental feature memory change data generated by this mapping and correction process; Specifically, the various output data generated in this round of mapping and correction process are classified and packaged, and format verification is performed on each category to ensure that the data written to the main database conforms to the preset constraints in terms of structure and content.

[0107] Data that needs to be persisted falls into the following four categories: The first category is unified mapping relationship data. This includes each confirmed mapping record between multi-source heterogeneous material codes and standard material codes. The fields must include at least: source system identifier, source material code, standard material code identifier, mapping establishment timestamp, mapping source identifier (automatic mapping / manual review), and mapping status (confirmed / pending review / discarded).

[0108] The second category is the revised standard coding data. This includes standard coding records added or updated in the standard material coding library during this round of processing. The fields cover standard material code identifier, material description text, values ​​of each dimension of structured feature vector, standardized unstructured text feature vector, structured hash identifier, and feature fingerprint.

[0109] The third category is coding evolution stability data. This includes the coding evolution stability factor for each multi-source heterogeneous material code involved in this process. Its sub-indicators, with fields including at least: source system identifier, source material code, Value, number of changes Average change interval Cumulative usage time And stability data update timestamps.

[0110] The fourth category is incremental feature memory change data. This includes operation records of newly added, updated, and deleted entries in the incremental feature memory during this round of processing. Each record includes the operation type (add / update / delete), band hash value, standard encoding identifier, and operation timestamp.

[0111] Format validation is performed on the four types of data mentioned above. The validation rules include: non-empty checks for required fields, field value type matching checks, compliance checks for enumerated field value domains, and timestamp format consistency checks. Data records that fail validation are temporarily stored in an abnormal data queue and do not enter the subsequent writing process. At the same time, a validation exception log is generated for operation and maintenance to troubleshoot.

[0112] S5.2. Distributed multi-replica synchronous write technology is adopted to synchronously store various types of data that have passed verification into the corresponding data partition of the standard material code master database. Specifically, all types of data that pass verification are synchronously stored in the corresponding data partition of the standard material code master database using distributed multi-replica synchronous write technology. Details are as follows: The standard material coding master database adopts a distributed storage architecture, divided into four logical partitions according to data categories: mapping relationship partition, standard coding data partition, stability data partition, and memory data partition. Each partition is configured with a multi-replica strategy, with ≥3 replicas, and each replica is distributed across different physical nodes.

[0113] The write process employs a two-phase commit protocol: In the first phase, the coordinating node sends a pre-commit request to the primary and replica nodes of each partition. Each node writes the data to its local write-ahead log and returns an acknowledgment. In the second phase, after receiving acknowledgments from all primary and replica nodes, the coordinating node sends a commit command. Each node flushes the data in its write-ahead log to the persistent storage layer and synchronously replicates it to the secondary replica nodes. If any primary or replica node fails to commit, the coordinating node sends a rollback command to all pre-committed nodes, and the entire write transaction is rolled back to ensure data consistency.

[0114] After the write operation is complete, each partition updates its data version number. The version number is generated using a monotonically increasing integer or a timestamp, which is used to identify data changes during subsequent incremental synchronization.

[0115] S5.3 Real-time monitoring of data change events in the standard material code master database to trigger incremental update processes; Specifically, the standard material code master database is configured with a change data capture mechanism to monitor data change events in each data partition in real time. When an insert, update, or delete operation occurs, an incremental update process is triggered.

[0116] The structured description of a change event includes the following fields: event type (INSERT / UPDATE / DELETE), affected data partition identifier, affected data record primary key, timestamp of the change, and the updated data version number.

[0117] The incremental update process is triggered when: the number of change events accumulated in any data partition within a preset monitoring window reaches the batch processing threshold, or the time interval since the last incremental push reaches the preset push cycle, whichever comes first. In this embodiment, the batch processing threshold can be set to 100, and the preset push cycle can be set to 5 minutes. This batch triggering strategy balances data real-time performance with system throughput.

[0118] S5.4 Extract the timestamp and version number of the changed data, and push the incremental changed data to the local cache of all related business systems in a synchronized manner; Specifically, once the incremental update process is triggered, the timestamps and version numbers of all changed data in this batch are extracted, and the incremental change data is synchronously pushed to the local cache of all related business systems.

[0119] Related business systems include, but are not limited to, Enterprise Resource Planning (ERP) systems, Supply Chain Management (SRM) systems, Manufacturing Execution System (MES) systems, and data platforms. Each related business system registers a callback address upon initial integration, and the standard material code master database maintains a registration information table for the related business systems.

[0120] The push process employs a publish-subscribe pattern for message queues: incremental changes are encapsulated into standardized message bodies and published to a designated topic on the message bus. Each related business system, acting as a subscriber to this topic, retrieves the incremental message from the message bus, parses it, and updates its local cache. The message body includes: the changed data content, the source data partition identifier, the data version number, and the change timestamp. Each message body carries a message sequence number; the receiver uses this sequence number to deduplicate messages, preventing duplicate consumption and cache inconsistencies.

[0121] S5.5 Perform a data consistency check between the standard material code master database and the cache of each related business system. If data inconsistency is found, trigger a full synchronization correction.

[0122] Specifically, after each incremental push is completed, a data consistency check is performed between the standard material code master database and the caches of each related business system.

[0123] The verification method is as follows: The standard material code master database sends verification requests to each related business system. The verification request carries the data version number of each data partition and the checksum of the data entries. The checksum is calculated using the root hash value of a Merkle tree structure. After sorting the data records in each partition by primary key, the hash value is calculated for each record. A Merkle tree is built from bottom to top, and the root hash value is used as the checksum of that partition.

[0124] Upon receiving a verification request, each related business system calculates a checksum using the same algorithm based on its local cached data and compares it with the checksum in the request. If the checksums match, the data in that partition is considered consistent; if the checksums differ, a Merkle tree is used to further compare and locate the range of inconsistent data records.

[0125] When data inconsistency is detected, a full synchronization correction process is triggered: the standard material code master database pushes the inconsistent data records of that partition to the associated business systems as a full snapshot. The associated business systems then overwrite the corresponding data records in their local caches with the snapshot data from the master database, completing the correction. After the full synchronization correction is completed, a consistency check is performed again until both sides agree.

[0126] Through this step, all output data of this mapping and correction process is reliably persisted to the standard material code master database. Through incremental push and consistency verification mechanisms, the high consistency and real-time synchronization capability of standard code data, mapping relationship data and stability metadata between the master database and various related business systems are guaranteed.

[0127] Those skilled in the art will understand that the process of implementing all or part of the steps of the above embodiments can be carried out by hardware or by a program instructing the relevant hardware.

[0128] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for unified mapping and duplicate correction of material codes supporting multi-source heterogeneous ERP systems, characterized in that, Includes the following steps: S1. Multi-source heterogeneous ERP related data collection: Using multi-protocol data adaptation and access technology, it synchronously collects material basic data, multi-source heterogeneous material coding data and coding full life cycle traceable operation log data from various heterogeneous ERP systems. S2. Multi-source data standardization preprocessing: Using structured and unstructured unified standardization technology, the collected material basic data, multi-source heterogeneous material coding data and operation log data are respectively processed for format unification, outlier removal and time dimension normalization. S3. Intelligent unified mapping construction of material codes: The dual-branch heterogeneous feature decoupling-incremental memory fusion mapping algorithm is adopted to establish the mapping relationship between multi-source heterogeneous material codes and standard material codes. Among them, multi-source heterogeneous material codes that reach the final judgment threshold in terms of comprehensive similarity score and have no conflict are unidirectionally and uniquely mapped to a standard material code. The coding evolution stability factor is introduced to dynamically adjust the comprehensive similarity judgment threshold. S4. Duplicate Detection and Conflict Correction after Mapping: Based on the uniqueness constraint of the standard material coding library, feature fingerprint comparison technology is used to detect duplicate and conflicting items in the mapped code, and hierarchical conflict resolution technology is used to complete automatic correction and manual review and diversion. S5. End-to-end data persistence and incremental update: Adopting distributed data persistence technology, the unified mapping relationship, the corrected standard coding data, coding evolution stability data and incremental feature memory are synchronously stored in the standard material coding master database, and real-time incremental updates are supported.

2. The method for unified mapping and duplicate correction of material codes supporting multi-source heterogeneous ERP systems according to claim 1, characterized in that, In S1, the multi-protocol data adaptation access technology specifically includes: Based on RESTful API interface adaptation, it calls the material data service interface opened by various heterogeneous ERP systems to pull material basic data and multi-source heterogeneous material code data in real time. Based on JDBC database direct connection adaptation, by configuring ERP backend database connection permissions, we can selectively read the original data of multi-source heterogeneous material codes and full life cycle operation logs. Based on Web Service interface adaptation, a data interaction channel is established with the ERP system through SOAP protocol to obtain the externally released multi-source heterogeneous material codes and change records; Based on ETL batch data extraction and adaptation, material data files exported from the ERP system are captured at preset time intervals to complete offline batch data collection of multi-source heterogeneous material codes.

3. The method for unified mapping and duplicate correction of material codes supporting multi-source heterogeneous ERP systems according to claim 1, characterized in that, In step S2, the process of multi-source data standardization preprocessing includes the following steps: S21. Perform field mapping alignment, character set unification, and coding rule standardization on material basic data and multi-source heterogeneous material coding data; S22. Identify and remove invalid data that is missing key attributes, has contradictory data logic, or is redundant and repetitive in the coding of multi-source heterogeneous materials; S23. Convert the operation log time records of various heterogeneous ERP systems into a standard timestamp format to achieve time base unification.

4. The method for unified mapping and duplicate correction of material codes supporting multi-source heterogeneous ERP systems according to claim 1, characterized in that, In S3, the execution process of the dual-branch heterogeneous feature decoupling-incremental memory fusion mapping algorithm includes the following steps: S31. For the preprocessed multi-source heterogeneous material coding data, perform decoupling extraction and quantization coding of structured features and unstructured text features; S32. Perform fast candidate retrieval of encoded structured features based on incremental feature memory; S33. Weighted fusion of the structured features and unstructured text features of the code to be matched and the candidate standard code to calculate the comprehensive similarity score; S34. Combine the coding evolution stability factor to complete the dynamic similarity threshold determination and output the final mapping matching result.

5. The method for unified mapping and duplicate correction of material codes supporting multi-source heterogeneous ERP systems according to claim 4, characterized in that, In step S31, the process of decoupling and extracting structured features from unstructured text features and performing quantization encoding includes the following steps: S31.1 Extracting structured feature vectors of category, specification, material, and unit dimensions from multi-source heterogeneous material coding data. Extract core semantic unstructured text feature vectors from material description texts; S31.2, Use the Locality Sensitive Hashing algorithm to process structured feature vectors Encode to generate a unique structured hash identifier. ; S31.

3. Perform L2 norm normalization on the unstructured text feature vectors to obtain standardized unstructured text feature vectors.

6. The method for unified mapping and duplicate correction of material codes supporting multi-source heterogeneous ERP systems according to claim 4, characterized in that, In step S32, the process of quickly retrieving candidate features based on the incremental feature memory includes the following steps: S32.1, a structured hash identifier for matching multi-source heterogeneous material codes. Use the key as the retrieval key to perform a similarity search in the incremental feature memory; S32.2 Filter out the preset number of candidate standard codes with the highest similarity and proceed to the subsequent fusion matching stage; S32.3, The structured hash identifier of the newly added and successfully matched multi-source heterogeneous material codes. Write the corresponding standard encoding identifier into the incremental feature memory; S32.

4. Perform an overwrite update on entries with the same hash identifier that already exist in the memory.

7. The method for unified mapping and duplicate correction of material codes supporting multi-source heterogeneous ERP systems according to claim 4, characterized in that, In step S33, the process of weightedly fusing the structured features and unstructured text features of the code to be matched and the candidate standard code to calculate the comprehensive similarity score includes the following steps: S33.1 Calculate the structured feature similarity and unstructured text feature similarity between the code of the multi-source heterogeneous material to be matched and each candidate standard code; S33.2 Calculate the dynamic fusion weights of structured features and unstructured text features through a cross-attention mechanism; S33.

3. The similarity scores of structured features and unstructured text features are weighted and summed according to the dynamic fusion weights to obtain the comprehensive similarity score.

8. The method for unified mapping and duplicate correction of material codes supporting multi-source heterogeneous ERP systems according to claim 4, characterized in that, In step S34, the process of determining the dynamic similarity threshold by combining the coding evolution stability factor and outputting the final mapping matching result includes the following steps: S34.1 Extract all change records of the multi-source heterogeneous material codes to be matched within the preset historical period from the standardized operation log data; S34.2 Calculate the three core statistics of the code of the multi-source heterogeneous material to be matched in the historical period: number of changes, average change interval, and cumulative usage time. S34.3 Calculate the coding evolution stability factor using the weighted normalization method. Its value range is strictly limited to [0,1]; S34.4, Based on coding evolution stability factor The final judgment threshold is calculated by comparing the base judgment threshold of the material category to which the code to be matched belongs. Furthermore, it satisfies a monotonically decreasing relationship where higher coding stability leads to a lower final judgment threshold. S34.5, Final Judgment Threshold Apply global upper and lower bound constraints to avoid misjudgments caused by extreme values; S34.

6. Combine the comprehensive similarity score with the final judgment threshold. Perform comparisons and determine the matching results; S34.7, If the coding evolution stability factor If the value is below the preset lower limit, the code of the multi-source heterogeneous material to be matched will be marked as pending review.

9. The method for unified mapping and duplicate correction of material codes supporting multi-source heterogeneous ERP systems according to claim 1, characterized in that, In step S4, the process of detecting duplicates and correcting conflicts after mapping includes the following steps: S4.1 Generate structured feature fingerprints and unstructured text feature fingerprints corresponding to the multi-source heterogeneous material codes after mapping; S4.2 Based on the uniqueness constraint of the standard material coding library, the generated feature fingerprint is compared with the existing standard coding feature fingerprints in the library to identify duplicates with completely identical feature fingerprints and conflicting items with partially matched feature fingerprints. S4.3 For duplicate entries with completely identical feature fingerprints, the newly generated mapping relationship is automatically retained and marked as confirmed, while other redundant mapping entries are deleted. S4.4 For conflict terms with feature fingerprint similarity exceeding a preset conflict threshold, extract the standardized operation log data encoded by both conflicting parties, and calculate their encoding evolution stability factors respectively. ; S4.5, Coding evolution stability factor when the two conflicting parties' codes When the difference exceeds the preset correction threshold, the code with higher stability is automatically selected as the final standard mapping result, while the code with lower stability is marked as discarded. S4.6, Coding evolution stability factor when the two conflicting parties' codes When the difference is less than or equal to the preset correction threshold, the conflict item and the corresponding operation log data will be pushed to the manual review interface. S4.

7. The final mapping relationship, which has been automatically corrected or manually reviewed and confirmed, will be synchronously updated to the standard material code library and the incremental feature memory.

10. The method for unified mapping and duplicate correction of material codes supporting multi-source heterogeneous ERP systems according to claim 1, characterized in that, In S5, the process of end-to-end data persistence and incremental update includes the following steps: S5.1 Classify, package, and format-verify the unified mapping relationship, corrected standard coding data, coding evolution stability data, and incremental feature memory change data generated by this mapping and correction process; S5.

2. Distributed multi-replica synchronous write technology is adopted to synchronously store various types of data that have passed verification into the corresponding data partition of the standard material code master database. S5.3 Real-time monitoring of data change events in the standard material code master database to trigger incremental update processes; S5.4 Extract the timestamp and version number of the changed data, and push the incremental changed data to the local cache of all related business systems in a synchronized manner; S5.5 Perform a data consistency check between the standard material code master database and the cache of each related business system. If data inconsistency is found, trigger a full synchronization correction.

Images