Multi-source data field name intelligent matching method and device, equipment and medium

Abstract

The invention provides a multi-source data field name intelligent matching method which comprises the steps that field names of a plurality of input data sources are subjected to standardization processing, and the standardization processing comprises at least one of structure standardization, stop word filtering and length normalization; calculating font similarity, semantic similarity, structural similarity and value similarity between the at least two preprocessed field names; based on the font similarity, the semantic similarity, the structure similarity and the value similarity, obtaining a comprehensive similarity score; and under the condition that the comprehensive similarity score is greater than a preset threshold value, determining that the at least two field names are matched. The invention further provides a multi-source data field name intelligent matching device and equipment and a storage medium.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and more particularly, to a multi-source data field name intelligent matching method, device, equipment and medium. BACKGROUND

[0002] In the field of data governance and information system integration, in order to realize data integration and analysis across business systems, the fields from multiple heterogeneous data sources must be aligned and mapped. However, due to differences in development specifications, design concepts and business backgrounds of different systems, the field naming of their data models is generally inconsistent, which constitutes a major technical obstacle to effective data fusion.

[0003] Firstly, at the level of character form, field naming is not uniform in capitalization specification, inconsistent in abbreviation rules, chaotic in the use of special characters, and contains spelling errors, etc., which leads to the failure of traditional technical methods based on string exact matching. Secondly, at the semantic level, different systems often use different business terms or synonyms to refer to the same business concept, and at the same time, the same field name may carry different business meanings in different context, which brings deep challenges to the accurate mapping of fields. Thirdly, at the structural level, there are significant differences in field naming in terms of vocabulary composition, word segmentation method and syntactic organization, such as the naming of complex business concepts being split and combined in different ways, which requires the matching technology to have basic analysis ability of the naming structure. Finally, at the level of data values and formats, even if the field names are mapped, the underlying data often cannot be directly used due to inconsistencies in measurement units, representation formats or basic data type definitions, which affects the quality of integrated data and the accuracy of subsequent analysis.

[0004] Currently, the industry generally relies on manual methods to sort out and confirm the field mapping relationship. This method not only consumes a large amount of human resources and time cost, but also is inefficient, and is difficult to adapt to the massive and dynamic business environment. At the same time, its accuracy and consistency are highly dependent on the personal experience of the implementers, and the repeatability and standardization degree are insufficient. Therefore, there is an urgent need in the field for a technical solution that can automatically and intelligently match multi-source heterogeneous data field names, in order to overcome the inherent defects of existing manual methods and significantly improve the efficiency, accuracy and scalability of data integration. SUMMARY

[0005] Therefore, the present disclosure provides a multi-source data field name intelligent matching method, device, equipment and medium.

[0006] The first aspect of the present disclosure provides a multi-source data field name intelligent matching method, comprising the following steps: standardizing preprocessing of field names of a plurality of input data sources; calculating glyph similarity, semantic similarity, structural similarity and value similarity between at least two field names after preprocessing; obtaining a comprehensive similarity score based on the glyph similarity, the semantic similarity, the structural similarity and the value similarity; and determining that the at least two field names match in the case that the comprehensive similarity score is greater than a preset threshold.

[0007] According to an embodiment of the present disclosure, the method comprises dynamically adjusting the preset threshold based on the length of the field name; wherein the preset threshold decreases with the increase of the length of the field name.

[0008] According to an embodiment of the present disclosure, the method of calculating the glyph similarity comprises: calculating the Levenshtein distance between the at least two field names to obtain a basic similarity; determining a penalty weight based on the difference of the first characters of the at least two field names; and determining the glyph similarity based on the penalty weight and the basic similarity.

[0009] According to an embodiment of the present disclosure, calculating the semantic similarity comprises: searching the at least two field names as query items in a synonym library, the synonym library recording the mapping relationship of different words, abbreviations and multilingual vocabularies expressing the same semantic concept; determining the semantic similarity according to the search result; wherein if the two field names exist in the same mapping relationship, the semantic similarity value is determined according to the preset mapping relationship; if the two field names do not exist in the same mapping relationship, the semantic similarity is determined as 0.

[0010] According to an embodiment of the present disclosure, the method of calculating the structural similarity comprises: decomposing the at least two field names into word sequences respectively; calculating the weighted average of the Jaccard similarity and the word vector space cosine similarity of the word sequences of the at least two field names to obtain the structural similarity.

[0011] According to an embodiment of the present disclosure, the method of calculating the value similarity comprises: obtaining the data values of the at least two field names; performing numerical distribution inspection and data type consistency inspection on the data values of the at least two field names; determining the value similarity based on the inspection results of the numerical distribution inspection and the data type consistency inspection; wherein the numerical distribution inspection comprises comparing the mean, variance or distribution form of the numerical data corresponding to the fields, and the data type consistency inspection comprises judging the consistency of the data type, format or unit of the fields.

[0012] According to an embodiment of the present disclosure, the standardization preprocessing of the field names of the input multiple data sources comprises: noise processing, missing value processing, structure standardization processing, stop word filtering and length normalization processing on the field names.

[0013] A second aspect of the present disclosure provides a multi-source data field name intelligent matching device, comprising: a preprocessing module configured to perform standardization preprocessing on field names of input multiple data sources; a multi-dimensional similarity calculation module configured to calculate character form similarity, semantic similarity, structure similarity and value similarity between at least two field names after preprocessing; a comprehensive evaluation module configured to obtain the comprehensive similarity score based on the character form similarity, the semantic similarity, the structure similarity and the value similarity; and a matching output module configured to determine that the at least two field names match if the comprehensive similarity score is greater than a preset threshold.

[0014] A third aspect of the present disclosure provides an electronic device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the multi-source data field name intelligent matching method according to any one of the first aspect when executing the program.

[0015] A fourth aspect of the present disclosure provides a computer readable storage medium, having a computer program stored thereon, wherein the computer program is executable by a processor to implement the multi-source data field name intelligent matching method according to any one of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure taken in conjunction with the accompanying drawings, in which:

[0017] Figure 1 A flow chart of the multi-source data field name intelligent matching method according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 2 A block diagram of the multi-source data field name intelligent matching device according to an embodiment of the present disclosure is schematically shown; and

[0019] Figure 3 A block diagram of the electronic device 300 suitable for implementing the multi-source data field name intelligent matching method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0020] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it would be apparent to those skilled in the art that the embodiments can be practiced without these specific details. In other instances, well-known structures and functions have been described in detail in order to avoid obscuring the concepts of the present disclosure.

[0021] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so on, mean the stated features, steps, operations and / or components are present, but do not preclude the presence or addition of one or more other features, steps, operations, components, and / or groups thereof.

[0022] All terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the present description, and should not be interpreted in an idealized or excessively formal manner.

[0023] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted to include at least one of the items enumerated, but not limited to the items enumerated (e.g., "a system having at least one of A, B, and C" should include a system having A alone, a system having B alone, a system having C alone, a system having A and B together, a system having A and C together, a system having B and C together, and / or a system having A, B, and C together, etc.).

[0024] In the embodiments of the present disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) comply with the relevant legal regulations, are used for legal purposes, and do not violate public order and good customs. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data, and to maintain user personal information security, network security and national security.

[0025] Figure 1 A flowchart of a multi-source data field name intelligent matching method according to an embodiment of the present disclosure is schematically shown.

[0026] As shown in Figure 1 , the method includes operations S110-S140.

[0027] In operation S110, the field names of the input multiple data sources are standardized and preprocessed.

[0028] In the embodiments of the present application, the standardization preprocessing of the field names of the input multiple data sources includes noise processing, missing value processing, structure standardization processing, stop word filtering, and length normalization processing.

[0029] The noise processing is used to eliminate random errors and irrelevant characters in the field names, and specifically includes automatically identifying and correcting common spelling errors in the field names, and removing special characters or redundant spaces in the field names that are irrelevant to semantics.

[0030] The missing value processing is used to deal with special cases of missing field names, and specifically includes, for a field that is completely missing in a data source, inferring a possible field name based on business context information or an association relationship with other related fields and performing matching attempts, and if the field cannot be effectively inferred, marking the missing field as a to-be-processed item and recording a log for subsequent manual review.

[0031] The structure standardization processing is used to unify the basic representation form of the field names, and specifically includes uniformly converting various separators, such as underscores, hyphens, or camel case, in the field names into standard separators or spaces to eliminate structural differences caused by different formats.

[0032] The stop word filtering is used to remove common words in the field names that have low contribution to semantics, and specifically includes filtering common suffixes or prefixes, such as “number”, “code”, “identifier”, and “description”, in the field names according to a predefined stop word table to extract a word segment that carries core semantics.

[0033] The length normalization processing is used to truncate long field names to focus on core semantics, and specifically includes setting an upper threshold for the length of the field names, and for a super-long field name that exceeds the threshold, truncating secondary modifying or explanatory parts of the field name by analyzing the word structure of the field name to retain a core semantic segment that has distinguishing power.

[0034] Through the above standardization preprocessing process, noise interference and structural differences in the original field names can be significantly reduced, thereby laying a reliable foundation for subsequent high-precision mixed similarity calculation.

[0035] In operation S120, the glyph similarity, the semantic similarity, the structure similarity, and the value similarity between the at least two preprocessed field names are calculated.

[0036] In the embodiments of the present application, the method of calculating the glyph similarity includes calculating a Levenshtein distance between the at least two field names to obtain a basic similarity, determining a penalty weight based on a difference between first characters of the at least two field names, and determining the glyph similarity based on the penalty weight and the basic similarity.

[0037] The formula for character similarity is expressed as:

[0038] Sedit=1−max(len(a),len(b))ED(a,b)×(1+0.3×Pdiff);

[0039] Among them, P diff The difference in the first letter represents the proportion of the two field names, a and b.

[0040] In this embodiment of the disclosure, calculating semantic similarity includes: using at least two field names as query terms to search in a thesaurus, the thesaurus recording mapping relationships of different words, abbreviations, and multilingual words expressing the same semantic concept; determining semantic similarity based on the search results; wherein, if the two field names exist in the same mapping relationship, a higher semantic similarity value is determined according to a preset mapping relationship; if the two field names do not exist in the same mapping relationship, the semantic similarity is determined to be 0.

[0041] For example, the thesaurus contains a mapping relationship "INCOME" → "income" → "occurrence count", indicating that the two belong to the same semantic concept. Therefore, the system assigns a high semantic similarity value (e.g., 1.0) to these two field names according to preset rules, thus supporting that they are matching fields.

[0042] In this embodiment of the disclosure, the method for calculating structural similarity includes: decomposing at least two field names into word sequences; calculating the weighted average of the Jaccard similarity and the cosine similarity of the word sequences of at least two field names to obtain the structural similarity.

[0043] For example, "customer ID number" can be decomposed into ["customer", "ID number", "number"]. The weighted average of the Jaccard similarity of the word sequence and the cosine similarity of the word vector space can be calculated, with a weight ratio of 3:7.

[0044] In this embodiment of the disclosure, the method for calculating value similarity includes: obtaining data values ​​of at least two field names; performing numerical distribution testing and data type consistency testing on the data values ​​of at least two field names; determining value similarity based on the test results of numerical distribution testing and data type consistency testing; wherein, numerical distribution testing includes comparing the mean, variance, or distribution pattern of the numerical data corresponding to the field, and data type consistency testing includes judging the consistency of the data type, format, or unit of the field.

[0045] For example, when comparing the fields "Sales Revenue (Ten Thousand Yuan)" and "Revenue (Yuan)," the system automatically converts "Ten Thousand Yuan" to "Yuan." Then, it compares the distribution of the converted values ​​(such as mean and variance). If the numerical ranges and statistical characteristics of the two values ​​are highly similar, it determines that their values ​​are highly similar and supports the assumption that they are matching fields.

[0046] For example, when comparing the string field "Transaction Date (YYYYMMDD)" with the date field "Date (YYYY-MM-DD)", the system parses both into a standard date format. By analyzing the parsed date range and distribution, if the time intervals and distribution patterns of the two are found to be consistent, then their values ​​are considered to have a high degree of similarity.

[0047] For example, when both data sources contain a field named "Bank", the system analyzes the actual data content. If one field's values ​​are mostly financial institution names (such as "Bank of China"), while the other field's values ​​are mostly geographical terms (such as "riverbank"), then based on the significant difference in value distribution characteristics, the system determines that the two values ​​have extremely low similarity and are not matching fields.

[0048] In operation S130, a comprehensive similarity score is obtained based on glyph similarity, semantic similarity, structural similarity, and value similarity.

[0049] The similarity values ​​from the four dimensions are combined using a weighted summation method to calculate the overall similarity score:

[0050] Stotal=w1⋅Sedit+w2⋅Ssemantic+w3⋅Sstructural+w4⋅Svalue,

[0051] Where w1, w2, w3, and w4 are the weights of each dimension, and w1+w2+w3+w4=1.

[0052] In operation S140, if the overall similarity score is greater than a preset threshold, at least two field names are determined to match.

[0053] In this embodiment of the disclosure, the preset threshold is a dynamic threshold. Specifically, the preset threshold is dynamically adjusted based on the length of the field name; wherein, the preset threshold decreases as the length of the field name increases.

[0054] This disclosure provides a piecewise function control strategy with a preset threshold, defining a functional relationship between field length L and matching threshold T:

[0055] When L < 5, T = 0.85;

[0056] When 5 ≤ L < 10, T = 0.75 − 0.02 × (L − 5);

[0057] When L≥10, T=0.65−0.01×(L−10).

[0058] The lower limit of the threshold is 0.55 to ensure matching quality.

[0059] The comprehensive similarity score S total Compare with the threshold T, if S tota If the value is greater than or equal to T, then the two field names are considered similar.

[0060] The intelligent matching method for multi-source data field names provided in this disclosure constructs a hybrid similarity model that integrates four-dimensional features of glyphs, semantics, structure, and field values, and combines it with a dynamic threshold adjustment mechanism based on field length. This method achieves high-precision and high-efficiency automatic matching of field names for multi-source heterogeneous data, effectively overcoming the limitations of traditional methods in dealing with synonyms, cross-domain terms, and differences in data values, and providing reliable technical support for data integration and governance.

[0061] Figure 2 A block diagram of a multi-source data field name intelligent matching device according to an embodiment of the present disclosure is shown schematically.

[0062] like Figure 2 As shown, the intelligent matching device 200 for multi-source data field names includes a preprocessing module 210, a multi-dimensional similarity calculation module 220, a comprehensive evaluation module 230, and a matching output module 240.

[0063] The preprocessing module 210 is used to standardize the field names of multiple input data sources.

[0064] The multidimensional similarity calculation module 220 is used to calculate the glyph similarity, semantic similarity, structural similarity and value similarity between at least two field names after preprocessing.

[0065] The comprehensive evaluation module 230 is used to obtain a comprehensive similarity score based on glyph similarity, semantic similarity, structural similarity and value similarity.

[0066] The matching output module 240 is used to determine that at least two field names match when the overall similarity score is greater than a preset threshold.

[0067] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0068] For example, any multiple of the preprocessing module 210, multidimensional similarity calculation module 220, comprehensive evaluation module 230, and matching output module 240 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the preprocessing module 210, multidimensional similarity calculation module 220, comprehensive evaluation module 230, and matching output module 240 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the preprocessing module 210, the multidimensional similarity calculation module 220, the comprehensive evaluation module 230, and the matching output module 240 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0069] It should be noted that the intelligent matching device part for multi-source data field names in the embodiments of this disclosure corresponds to the intelligent matching method part for multi-source data field names in the embodiments of this disclosure. For a detailed description of the intelligent matching device part for multi-source data field names, please refer to the intelligent matching method part for multi-source data field names, which will not be repeated here.

[0070] Figure 3 A block diagram of an electronic device suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0071] like Figure 3 As shown, an electronic device 300 according to an embodiment of this disclosure includes a processor 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage portion 308 into a random access memory (RAM) 303. The processor 301 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 301 may also include onboard memory for caching purposes. The processor 301 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0072] RAM 303 stores various programs and data required for the operation of electronic device 300. Processor 301, ROM 302, and RAM 303 are interconnected via bus 304. Processor 301 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 302 and / or RAM 303. It should be noted that programs may also be stored in one or more memories other than ROM 302 and RAM 303. Processor 301 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0073] According to embodiments of this disclosure, the electronic device 300 may further include an input / output (I / O) interface 305, which is also connected to a bus 304. The electronic device 300 may also include one or more of the following components connected to the input / output (I / O) interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the input / output (I / O) interface 305 as needed. A removable medium 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 310 as needed so that computer programs read from it can be installed into the storage section 308 as needed.

[0074] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by processor 301, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0075] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0076] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0077] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 302 and / or RAM 303 described above and / or one or more memories other than ROM 302 and RAM 303.

[0078] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the multi-source data field name intelligent matching method provided in the embodiments of this disclosure.

[0079] When the computer program is executed by the processor 301, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0080] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via communication section 309, and / or installed from removable medium 311. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0081] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0082] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0083] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A multi-source data field name intelligent matching method, characterized in that, The method comprises the following steps: standardizing and preprocessing field names of multiple input data sources; calculating glyph similarity, semantic similarity, structural similarity and value similarity between at least two field names after preprocessing; obtaining the comprehensive similarity score based on the glyph similarity, the semantic similarity, the structural similarity and the value similarity; determining that the at least two field names match if the comprehensive similarity score is greater than a preset threshold.

2. The method of claim 1, wherein, The method comprises: dynamically adjusting the preset threshold based on the length of the field name; wherein the preset threshold decreases with the increase of the length of the field name.

3. The method according to claim 1 or 2, characterized in that, The method for calculating the glyph similarity comprises: calculating the Levenshtein distance between the at least two field names to obtain a basic similarity; determining a penalty weight based on the difference of the first characters of the at least two field names; determining the glyph similarity based on the penalty weight and the basic similarity.

4. The method according to claim 1 or 2, characterized in that, The method for calculating the semantic similarity comprises: searching the at least two field names as query terms in a synonym library, which records the mapping relationship of different words, abbreviations and multi-language vocabularies expressing the same semantic concept; determining the semantic similarity according to the search result; wherein if the two field names exist in the same mapping relationship, the semantic similarity value is determined according to the preset mapping relationship; if the two field names do not exist in the same mapping relationship, the semantic similarity is determined as 0.

5. The method according to claim 1 or 2, characterized in that, The method for calculating the structural similarity comprises: decomposing the at least two field names into word sequences respectively; calculating the weighted average of the Jaccard similarity and the word vector space cosine similarity of the word sequences of the at least two field names to obtain the structural similarity.

6. The method of claim 1 or 2, wherein, The method for calculating the value similarity comprises: obtaining the data values of the at least two field names; performing numerical distribution test and data type consistency test on the data values of the at least two field names; determining the value similarity based on the test results of the numerical distribution test and the data type consistency test; wherein the numerical distribution test comprises comparing the mean, variance or distribution shape of the numerical data corresponding to the fields, and the data type consistency test comprises consistency judgment on the data type, format or unit of the fields.

7. The method of claim 1, wherein, The standardization preprocessing of the field names of the multiple input data sources comprises: noise processing, missing value processing, structure standardization processing, stop word filtering and length normalization processing on the field names.

8. A multi-source data field name intelligent matching apparatus, characterized by comprising: The method comprises: a preprocessing module for standardizing and preprocessing field names of multiple input data sources; a multi-dimensional similarity calculation module for calculating glyph similarity, semantic similarity, structural similarity and value similarity between at least two field names after preprocessing; a comprehensive evaluation module for obtaining the comprehensive similarity score based on the glyph similarity, the semantic similarity, the structural similarity and the value similarity; a matching output module for determining that the at least two field names match if the comprehensive similarity score is greater than a preset threshold.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the multi-source data field name intelligent matching method as claimed in any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the multi-source data field name intelligent matching method as claimed in any one of claims 1 to 7 when executed by the processor.

Images