Data field standardized entry method and system based on intelligent verification and synonym mapping

Abstract

The invention relates to the technical field of data management and information management, in particular to a data field standardized entry method and system based on intelligent verification and synonym mapping, and the method comprises the following steps: a construction stage, an entry guide stage, an intelligent verification stage and a decision execution stage. The method has the advantages that different expressions of the same entity are eradicated from the source, data consistency is ensured, and a solid foundation is provided for accurate statistical analysis and data mining. Through intelligent recommendation and prompt, the input workload of the user is reduced, and meanwhile, through non-intrusive guidance instead of forced limitation, the input flexibility is ensured. By establishing a synonym library and an auditing process, data governance is regularized and processed, so that the system can be continuously self-optimized and adapt to new data changes, and the long-term maintenance cost is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data governance and information management, in particular to a data field standardization input method and system based on intelligent verification and synonym mapping. BACKGROUND

[0002] In various information management systems, users fill in text fields (such as unit name, project address, product name, etc.) through forms, which is the main way of data generation. Due to different input habits of users, existence of misspelled words or use of abbreviations / full names, etc., it is easy to cause multiple expressions of the same entity in the system. For example, "Chengkai International School", "Chengkai International School No. 1 Gate" and "Chengkai International College" all refer to the same place.

[0003] The existing technology usually adopts two ways: one is a simple drop-down list, but it cannot cover all possible inputs and has poor flexibility; the other is completely free input, which ensures flexibility but leads to the above-mentioned data inconsistency problem. Although some technologies use keyword matching for prompting, they have insufficient processing capacity for complex situations such as misspelled words, synonyms and inclusion relationships (such as "main name + details"), and cannot fundamentally solve the problem of data standardization. Therefore, there is an urgent need for an intelligent method that can ensure input flexibility while forcibly implementing data standardization from the source. SUMMARY

[0004] The purpose of the present application is to provide a data field standardization input method and system based on intelligent verification and synonym mapping to solve the problems raised in the background.

[0005] To achieve the above purpose, the present application provides the following technical solution: a data field standardization input method based on intelligent verification and synonym mapping, comprising the following steps: Construction stage: establishing a standard field library and a synonym library associated with the standard field; Input guidance stage: in response to the input operation of the user in the input box, real-time fuzzy matching is performed from the standard field library and the synonym library, and the matching results are recommended to the user in the form of a list; Intelligent verification stage: if the user inputs a new field without selecting the recommended item, the similarity of the new field with the existing standard field is calculated; if the similarity is higher than a preset threshold, the user is prompted with similar standard fields and suggested to replace; Decision execution stage: according to the feedback of the user to the prompt or the characteristics of the new field, different operations are performed: If the user agrees to replace, the standard field is input; If the user insists on input, the field is analyzed, and if it can be split into "standard part + detail part", the standard part is associated with the standard field and the detail part is stored in the extended field; Otherwise, the new field will be sent to the review queue, where the administrator will determine whether it should be used as a new standard field or mapped to an existing synonym.

[0006] Preferably, during the construction phase, the synonyms in the thesaurus include, but are not limited to: common misspellings, abbreviations, full names, and descriptions with directional or serial number details from standard fields.

[0007] Preferably, in the intelligent verification stage, the algorithm used to calculate the similarity between the new field and the existing standard field is the Levenshtein edit distance algorithm or a semantic similarity algorithm based on word vectors.

[0008] Preferably, in the decision execution phase, "parsing the field" involves using a predefined delimiter library or regular expressions to parse the input string into a first part representing the core entity and a second part representing supplementary information.

[0009] Preferably, the method constructs a standard field library and a thesaurus, and combines a series of operations in the input guidance, intelligent verification, and decision execution stages to achieve standardized data field input and improve the accuracy and efficiency of data input.

[0010] A standardized data entry system based on intelligent validation and synonym mapping includes: Build modules: used to create a standard field library and a thesaurus associated with the standard fields; Input guidance module: It responds to the user's input in the input box, performs fuzzy matching from the standard field library and the thesaurus in real time, and recommends the matching results to the user in the form of a list; Intelligent verification module: When a user enters a new field without selecting a recommended option, it calculates the similarity between the new field and existing standard fields; if the similarity is higher than a preset threshold, it prompts the user with similar standard fields and suggests replacement. Decision Execution Module: This module performs different operations based on user feedback to prompts or the characteristics of new fields. If the user agrees to the replacement, the standard field is entered. If the user insists on entering the field, it is parsed. If it can be split into "standard part + detailed part", the standard part is associated with the standard field, and the detailed part is stored in the extended field. Otherwise, the new field is sent to the review queue, where the administrator determines whether it should be used as a new standard field or mapped to an existing synonym.

[0011] Preferably, in the construction module, the synonyms in the thesaurus include, but are not limited to: common misspellings, abbreviations, full names, and descriptions with location or serial number details in the standard fields.

[0012] Preferably, in the intelligent verification module, the algorithm used to calculate similarity is the Levenshtein edit distance algorithm or a semantic similarity algorithm based on word vectors.

[0013] Preferably, in the decision execution module, "parsing the field" involves using a predefined delimiter library or regular expressions to parse the input string into a first part representing the core entity and a second part representing supplementary information.

[0014] Preferably, the system establishes standard fields and a thesaurus through a construction module, provides input guidance through an input recommendation module, performs similarity calculation and provides prompts through an intelligent verification module, and executes corresponding operations according to different situations through a decision execution module, thereby completing the standardized entry of data fields and improving the accuracy and efficiency of data entry.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention proposes a standardized data field entry method and system based on intelligent validation and synonym mapping. This eliminates different representations of the same entity from the source, ensuring data consistency and providing a solid foundation for accurate statistical analysis and data mining. Intelligent recommendations and prompts reduce user input workload, while non-intrusive guidance rather than mandatory restrictions ensures input flexibility. By establishing a synonym database and review process, data governance rules are formalized and streamlined, enabling the system to continuously self-optimize, adapt to new data changes, and reduce long-term maintenance costs. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1: This invention provides a technical solution: a standardized data field entry method based on intelligent validation and synonym mapping, comprising the following steps: ① Users start typing in the smart input box of the front-end application (such as a CRM system), such as typing "Chengkai International School Gate 1".

[0019] ② The real-time matching engine immediately works and returns fuzzy matching results (such as a list containing "Chengkai International School"), but the user does not select anything and continues to input the information.

[0020] ③ When the input box loses focus, intelligent verification is triggered. The similarity calculation engine calculates that the similarity between "Chengkai International School Gate 1" and "Chengkai International School" is approximately 85%, which is higher than the prompt threshold (80%).

[0021] ④ The intelligent decision engine generates a prompt based on this. The front end pops up a prompt: "The content you entered is similar to the standard field 'Chengkai International Academy'. It is recommended to use the standard name to maintain data consistency. How do I proceed?", and provides the buttons "Adopt Standard" and "Keep As is".

[0022] ⑤ When the user clicks "Adopt Standard", the system will store the standard field ID (S001) into the database.

[0023] ⑥ If the user clicks "Keep as is", the decision engine will activate the field parsing rules and successfully split the input into the core part "Chengkai International School" and the detailed part "Gate 1". The system will store the standard field ID (S001) in the main field and "Gate 1" in the extended details field.

[0024] ⑦ If the input is a completely new word (such as "Inspur Science and Technology Park"), and its similarity to all content in the existing library is below the threshold, the system will submit it to the "data pool to be reviewed" and notify the administrator to process it.

[0025] Example 2: A data field standardization entry system based on intelligent verification and synonym mapping, comprising: a data support layer, an intelligent service layer, and an application interaction layer.

[0026] ① Data Support Layer: Construction of the standard library, thesaurus, and rule base. This layer is the data foundation of the entire solution, responsible for storing all standard information, mapping relationships, and business rules.

[0027] Standard Fields Library: Stores all approved, unique standard field entries. Each entry contains a unique ID (standard field ID) and a standard name. For example: (S001, Chengkai International School).

[0028] Thesaurus: Associated with the standard field library, it stores all non-standard expressions and their mappings to standard fields. Each record contains a synonym ID, synonym content, and the corresponding standard field ID. For example: (T001, Chengkai International College, S001), (T002, Chengkai International College Gate 1, S001).

[0029] Business rule base: Stores rules used for intelligent judgment.

[0030] Similarity threshold rule: Define similarity thresholds for different scenarios (e.g., prompt threshold = 90%, automatic mapping threshold = 95%).

[0031] Field parsing rule: Define regular expressions or keyword libraries for splitting "core + details" (e.g., including "gate number", "building number", "unit", etc.).

[0032] ② Intelligent service layer: Core computing and decision-making engine. This layer is the brain of the system, encapsulating the core computing logic and providing capabilities to the application layer in the form of API services.

[0033] Real-time matching engine: Based on Trie tree or inverted index technology, achieve millisecond-level fuzzy matching of input keywords with the standard library and thesaurus, providing data for front-end drop-down recommendations.

[0034] Similarity calculation engine: Provide multiple similarity algorithms. By default, use the Levenshtein edit distance algorithm to calculate literal similarity; for higher-order requirements, semantic similarity calculation based on pre-trained models such as BERT can be integrated to understand cases of synonymous words with different expressions like "Digital Harbor" and "Cyberport".

[0035] Intelligent decision-making engine: This is the most core module. It coordinates the invocation of other engines according to the input content and makes the final decision following preset rules. For example, when the similarity calculation result is within a specific range, decide whether to "pass directly", "pop up a prompt", or "transfer to manual review".

[0036] ③ Application interaction layer: User interaction and process execution. This layer is responsible for interacting with users, executing the decisions of the intelligent service layer, and persisting the results.

[0037] Intelligent input box component: This is a reusable component that encapsulates all front-end interaction logic. It not only provides search drop-down functionality but also triggers similarity verification after input completion and displays an intelligent prompt pop-up window.

[0038] Manual review workbench: A unified interface provided for data administrators to process entries in the "pending review data pool". Administrators can perform operations such as "approve as a new standard field", "map to an existing standard field", or "reject as invalid data".

[0039] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A standardized data field entry method based on intelligent validation and synonym mapping, characterized in that: Includes the following steps: Construction phase: Establish a standard field library and a thesaurus associated with the standard fields; Input guidance stage: Responding to the user's input in the input box, performing fuzzy matching from the standard field library and the thesaurus in real time, and recommending the matching results to the user in list form; Intelligent verification phase: If the user enters a new field without selecting a recommended option, the similarity between the new field and the existing standard fields is calculated; if the similarity is higher than a preset threshold, the user is prompted with similar standard fields and suggested to replace them. Decision execution phase: Based on user feedback to the prompts or the characteristics of the new field, perform different operations: If the user agrees to the replacement, then enter the standard fields; If the user insists on entering information, the field is parsed. If it can be split into "standard part + detailed part", the standard part is associated with the standard field, and the detailed part is stored in the extended field. Otherwise, the new field will be sent to the review queue, where the administrator will determine whether it should be used as a new standard field or mapped to an existing synonym.

2. The data field standardization input method based on intelligent verification and synonym mapping according to claim 1, characterized in that: During the construction phase, the synonyms in the thesaurus include, but are not limited to: common misspellings, abbreviations, full names, and descriptions with location or serial number details in the standard fields.

3. The data field standardization input method based on intelligent verification and synonym mapping according to claim 2, characterized in that: During the intelligent verification phase, the algorithm used to calculate the similarity between the new field and the existing standard field is either the Levenshtein edit distance algorithm or a semantic similarity algorithm based on word vectors.

4. The data field standardization input method based on intelligent verification and synonym mapping according to claim 3, characterized in that: During the decision execution phase, "parsing the field" involves using a predefined delimiter library or regular expressions to parse the input string into a first part representing the core entity and a second part representing supplementary information.

5. The data field standardization input method based on intelligent verification and synonym mapping according to claim 4, characterized in that: The method constructs a standard field library and a thesaurus, and combines a series of operations in the input guidance, intelligent verification, and decision execution stages to achieve standardized data field input, thereby improving the accuracy and efficiency of data input.

6. A data field standardization input system based on intelligent verification and synonym mapping, applied to the method described in claim 5, characterized in that: include: Build modules: used to create a standard field library and a thesaurus associated with the standard fields; Input guidance module: It responds to the user's input in the input box, performs fuzzy matching from the standard field library and the thesaurus in real time, and recommends the matching results to the user in the form of a list; Intelligent verification module: When a user enters a new field without selecting a recommended option, it calculates the similarity between the new field and existing standard fields; if the similarity is higher than a preset threshold, it prompts the user with similar standard fields and suggests replacement. Decision Execution Module: This module performs different operations based on user feedback to prompts or the characteristics of new fields. If the user agrees to the replacement, the standard field is entered. If the user insists on entering the field, it is parsed. If it can be split into "standard part + detailed part", the standard part is associated with the standard field, and the detailed part is stored in the extended field. Otherwise, the new field is sent to the review queue, where the administrator determines whether it should be used as a new standard field or mapped to an existing synonym.

7. A data field standardization input system based on intelligent verification and synonym mapping according to claim 6, characterized in that: In the aforementioned construction module, the synonyms in the thesaurus include, but are not limited to: common misspellings, abbreviations, full names, and descriptions with directional or serial number details in the standard fields.

8. A data field standardization input system based on intelligent verification and synonym mapping according to claim 7, characterized in that: In the intelligent verification module, the similarity calculation algorithm is either the Levenshtein edit distance algorithm or a semantic similarity algorithm based on word vectors.

9. A data field standardization input system based on intelligent verification and synonym mapping according to claim 8, characterized in that: In the decision execution module, "parsing the field" involves using a predefined delimiter library or regular expressions to parse the input string into a first part representing the core entity and a second part representing supplementary information.

10. A data field standardization input system based on intelligent verification and synonym mapping according to claim 9, characterized in that: The system establishes standard fields and a thesaurus through a construction module, provides input guidance through an input recommendation module, performs similarity calculation and provides prompts through an intelligent verification module, and executes corresponding operations based on different situations through a decision-making execution module, thereby completing the standardized entry of data fields and improving the accuracy and efficiency of data entry.

Images