Metadata processing method

Abstract

The embodiment of the invention discloses a metadata processing method. The method comprises the following steps: acquiring a data dictionary; the data dictionary comprises N reference metadata in a unified format; the reference metadata comprises Chinese and English names of fields; n is an integer greater than 1; obtaining at least one piece of new metadata in the input file; the new metadata comprises Chinese and English names of fields; performing Chinese and English matching at least based on the Chinese and English name of the at least one piece of new metadata and the Chinese and English names of the N reference metadata to obtain a target association degree of the at least one piece of new metadata; the target association degree is used for representing the association degree between the new metadata and the data dictionary; and determining whether to add the at least one piece of new metadata to the data dictionary or not based on the target association degree of the at least one piece of new metadata. According to the scheme, the matching accuracy can be improved through a Chinese and English matching mode, the repetition condition of the data dictionary is reduced, and the availability rate of the data dictionary is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and relates to but is not limited to a metadata processing method. Background Art

[0002] At present, data modeling is based on the experience of architects. In the data modeling process, the same data has the same name but different meanings, and the same meaning has different names. This can easily lead to data swamps, making it difficult to find and use data.

[0003] How to efficiently use the group's shared data assets, reduce data costs, and increase data utilization efficiency has become a problem that companies are currently facing urgent challenges.

[0004] In the related art, data sharing is generally achieved by creating a data dictionary of an enterprise. However, it is generally achieved only through simple matching, for example, for a new data, it is matched with the data in the data dictionary, and if there is no duplication, the new data is added to the data dictionary.

[0005] It can be seen that the matching method of this solution is relatively simple. It cannot accurately identify the situation where the same data corresponds to different names in different systems, resulting in a large number of duplications in the data dictionary. Summary of the invention

[0006] The present application provides a metadata processing method, which can improve the matching accuracy through Chinese-English matching, reduce the duplication of data dictionaries, and improve the availability of data dictionaries.

[0007] The technical solution of this application is implemented as follows:

[0008] In a first aspect, the present application provides a method for processing metadata, the method comprising:

[0009] Get a data dictionary; the data dictionary includes N reference metadata in a unified format; the reference metadata includes the Chinese and English names of the fields; N is an integer greater than 1;

[0010] Obtain at least one new metadata in the input file; the new metadata includes the English name of the field;

[0011] At least one new metadata is matched in Chinese and English with the Chinese and English names of N reference metadata to obtain a target relevance of at least one new metadata; the target relevance is used to represent the degree of relevance between the new metadata and the data dictionary;

[0012] Based on the target relevance of the at least one new metadata, it is determined whether to add the at least one new metadata to the data dictionary.

[0013] In a second aspect, the present application provides a metadata processing device, the device comprising:

[0014] The first acquisition unit is used to acquire a data dictionary; the data dictionary includes N reference metadata in a unified format; the reference metadata includes the Chinese and English names of the fields; N is an integer greater than 1;

[0015] A second acquisition unit is used to acquire at least one new metadata in the input file; the new metadata includes the Chinese and English names of the fields;

[0016] A processing unit, configured to perform Chinese-English matching based on at least one Chinese and English name of the new metadata and the Chinese and English names of the N reference metadata, to obtain a target relevance of at least one new metadata; the target relevance is used to represent the degree of relevance between the new metadata and the data dictionary;

[0017] The determining unit is configured to determine whether to add the at least one new metadata to the data dictionary based on the target relevance of the at least one new metadata.

[0018] In a third aspect, the present application further provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the metadata processing method provided in the first aspect is implemented.

[0019] In a fourth aspect, the present application further provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the metadata processing method provided in the first aspect above is implemented.

[0020] In a fifth aspect, the present application further provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the metadata processing method provided in the first aspect above.

[0021] Compared with the prior art, the technical solution of this application has the following beneficial effects:

[0022] In the present application, a data dictionary is constructed, which includes multiple reference metadata in a unified format; the data dictionary can be used to unify data from multiple systems. When there is a new input file, or the Chinese and English names of the new metadata in the input file, the Chinese and English names of the new metadata are matched with the Chinese names of the reference metadata to obtain the target correlation between the new metadata and the data dictionary, and whether to add the new metadata to the data dictionary is determined based on the target correlation. By determining the correlation by matching Chinese and English, the accuracy of determining the correlation can be improved by considering the impact of the Chinese and English names of the metadata on the field. For new metadata, the higher the correlation, the smaller the probability of determining that it is added to the data dictionary, and the smaller the correlation, the greater the probability of determining that it is added to the data dictionary. Therefore, the improvement of the accuracy of the correlation can retrieve duplicates in the data dictionary and improve the availability of the data dictionary. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of a first optional flow chart of a metadata processing method provided in an embodiment of the present application;

[0024] Figure 2 A second optional flow chart of the metadata processing method provided in the embodiment of the present application;

[0025] Figure 3 A third optional flow chart of the metadata processing method provided in the embodiment of the present application;

[0026] Figure 4 A fourth optional flow chart of the metadata processing method provided in the embodiment of the present application;

[0027] Figure 5 A fifth optional flow chart of the metadata processing method provided in the embodiment of the present application;

[0028] Figure 6 A sixth optional flow chart of the metadata processing method provided in the embodiment of the present application;

[0029] Figure 7 A seventh optional flow chart of the metadata processing method provided in the embodiment of the present application;

[0030] Figure 8 An eighth optional flow chart of the metadata processing method provided in the embodiment of the present application;

[0031] Fig. 9 A ninth optional flow chart of the metadata processing method provided in the embodiment of the present application;

[0032] Fig.10A schematic diagram of the composition structure of a metadata processing device provided in an embodiment of the present application;

[0033] Fig.11 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0035] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first\second\third" are used only as examples to distinguish different objects, and do not represent a specific order for the objects, nor do they have a limitation on the order of precedence. It is understandable that "first\second\third" can be interchanged with a specific order or order of precedence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0038] The embodiments of the present application provide a metadata processing method, device, equipment, storage medium and computer program product. In practical applications, the metadata processing method can be implemented by a metadata processing device, and each functional entity in the metadata processing device can be collaboratively implemented by hardware resources of electronic equipment (such as vehicle equipment), such as processor computing resources, communication resources (such as for supporting various communication methods such as optical cable and cellular).

[0039] For ease of understanding, the application background of the metadata processing method provided in the embodiment of the present application is exemplarily described.

[0040] Enterprises generally include multiple systems: personnel system, sales system, production management system, material management system, etc. Multiple systems are generally managed and developed independently. During the development process, the corresponding names of the same data in different systems may be different. This results in a large amount of duplicate data in the created data dictionary.

[0041] In a first aspect, an embodiment of the present application provides a method for processing metadata.

[0042] Below, the metadata processing method provided by the application embodiment is described by taking an electronic device as an execution subject. For example, the electronic device can be a server, a computer, or other device with relevant data processing capabilities.

[0043] refer to Figure 1 As shown, the process may include but is not limited to the following S101 to S104.

[0044] S101. The electronic device obtains a data dictionary.

[0045] The data dictionary includes N reference metadata in a unified format; the reference metadata includes the Chinese and English names of the fields; N is an integer greater than 1.

[0046] The embodiment of the present application does not limit the unified format of storing reference metadata in the data dictionary, and can be configured according to actual needs, such as table format, text format, etc.

[0047] Reference metadata refers to metadata stored in the data dictionary, which has a unified data format and has been deduplicated.

[0048] Reference metadata includes the Chinese name of the field and the English name of the field. Of course, reference metadata can also include other information. For example, field value, field value type, field creation time, field modification time, field modification record, field call count, field source system, field original database type, etc.

[0049] S102: The electronic device obtains at least one new metadata in the input file.

[0050] The new metadata includes the Chinese and English names of the fields.

[0051] The input file here can be a new metadata, or a file including multiple new metadata, such as a table file.

[0052] When the input file includes multiple new metadata, the new metadata may be acquired one at a time and then processed accordingly; or the multiple new metadata in the file may be acquired together and processed.

[0053] In practice, since the reference metadata in the data dictionary is the processed metadata, the new metadata here has not been processed yet and may be the same, similar or different from a certain reference metadata in the data dictionary, so the following processing needs to be performed on the at least one new metadata to determine whether to add it to the data dictionary.

[0054] S103: The electronic device performs Chinese-English matching based on at least one Chinese and English name of the new metadata and the Chinese and English names of the N reference metadata to obtain a target relevance of the at least one new metadata.

[0055] The target relevance is used to characterize the degree of relevance between the new metadata and the data dictionary.

[0056] The degree of association may be reflected by similarity, importance, etc. The embodiment of the present application does not limit the method of determining the degree of association, and it can be configured according to actual conditions.

[0057] The target relevance is used to characterize the relevance between the new metadata and the data dictionary. The target relevance can be determined by the relevance between the new metadata and the reference metadata in the data dictionary.

[0058] S103 can be implemented as follows: the electronic device performs Chinese matching based on at least one Chinese name of the new metadata and the Chinese names of N reference metadata, performs English matching based on at least one English name of the new metadata and the English names of N reference metadata, and determines the target relevance of at least one new metadata based on the two matching results.

[0059] S104: The electronic device determines whether to add at least one new metadata to the data dictionary based on the target relevance of the at least one new metadata.

[0060] Since the target relevance can represent the degree of relevance between the new metadata and the data dictionary, it is possible to determine whether to add the at least one new metadata to the data dictionary based on the target relevance of the at least one new metadata. Based on the consideration that the data dictionary cannot be repeated, for the at least one new metadata, the higher the relevance, the lower the probability of determining that one less new metadata is added to the data dictionary, and the lower the relevance, the greater the probability of determining that one less new metadata is added to the data dictionary.

[0061] The metadata processing method provided by this embodiment may include: obtaining a data dictionary; the data dictionary includes N reference metadata in a unified format; the reference metadata includes Chinese and English names of the fields; N is an integer greater than 1; obtaining at least one new metadata in the input file; the new metadata includes Chinese and English names of the fields; performing Chinese and English matching based on at least one new metadata's Chinese and English names and N reference metadata's Chinese and English names, obtaining a target relevance of at least one new metadata; the target relevance is used to characterize the degree of relevance between the new metadata and the data dictionary; based on the target relevance of at least one new metadata, determining whether to add at least one new metadata to the data dictionary.

[0062] In the present application, a data dictionary is constructed, which includes multiple reference metadata in a unified format; the data dictionary can be used to unify data from multiple systems. When there is a new input file, or the Chinese and English names of the new metadata in the input file, the Chinese and English names of the new metadata are matched with the Chinese names of the reference metadata to obtain the target correlation between the new metadata and the data dictionary, and whether to add the new metadata to the data dictionary is determined based on the target correlation. By determining the correlation by matching Chinese and English, the accuracy of determining the correlation can be improved by considering the impact of the Chinese and English names of the metadata on the field. For new metadata, the higher the correlation, the smaller the probability of determining that it is added to the data dictionary, and the smaller the correlation, the greater the probability of determining that it is added to the data dictionary. Therefore, the improvement of the accuracy of the correlation can retrieve duplicates in the data dictionary and improve the availability of the data dictionary.

[0063] Next, the process of obtaining the target relevance of at least one new metadata by performing Chinese-English matching between the electronic device and N reference metadata based on at least one new metadata Chinese-English name in S103 will be described.

[0064] In one possible implementation, reference Figure 2 As shown in the content, the process may include but is not limited to the following S1031 to S1034.

[0065] For each new metadata in the at least one new metadata, the following S1031 to S1033 are executed to obtain the target relevance of each new metadata, and then the following S1034 is executed to obtain the target relevance of the at least one new metadata.

[0066] S1031. The electronic device performs similarity matching between the Chinese name of the field of the new metadata and the Chinese name of the field of each reference metadata in N reference metadata, and determines N Chinese similarities of the new metadata.

[0067] Here, for the Chinese name matching between the new metadata and a reference metadata, a Chinese similarity can be obtained, where a similarity refers to the similarity between the new metadata and a reference metadata. By traversing all reference metadata, N Chinese similarities can be obtained.

[0068] The embodiment of the present application does not limit the method of determining similarity by Chinese matching, and can be configured according to actual needs. For example, the Chinese name can be matched directly, or the Chinese name can be matched after word segmentation.

[0069] S1032: The electronic device matches the English name of the field of the new metadata with the English name of the field of each reference metadata in the N reference metadata, and determines N English similarities of the new metadata.

[0070] The implementation of S1032 is similar to the process of determining the Chinese similarity in S1031, except that the matching is based on the English name. The embodiment of the present application does not limit the method of determining the similarity of English matching, and can be configured according to actual needs. For example, the matching can be performed directly by the English name, or the matching can be performed after extracting the vector of the English name.

[0071] S1033: The electronic device determines a target relevance of the new metadata based on at least the N Chinese similarities and the N English similarities of the new metadata.

[0072] The electronic device determines the new metadata based on at least the N Chinese similarities and the N English similarities of the new metadata, and determines the relevance of the new metadata. Here, there are multiple relevances, and then determines the target relevance among the multiple relevances.

[0073] The target relevance can be one or more. When the target relevance is one, the maximum value of the relevance is generally taken as the target relevance, or the average value of multiple larger relevances is taken as the target relevance.

[0074] When there are multiple target relevances, the relevances may be sorted in descending order, and the first few relevances are taken as the multiple target relevances.

[0075] The embodiment of the present application does not limit the manner of determining the association degree based on the similarity, and can be configured according to actual needs. For example, the association degree can be determined directly based on the similarity, or can also be determined based on the similarity and other information (record information, etc.).

[0076] S1034: For each new metadata in the at least one new metadata, the electronic device determines a target relevance of the at least one new metadata based on the target relevance of each new metadata.

[0077] When the number of the at least one new metadata is one, the electronic device directly determines the target relevance of the new metadata as the relevance of the at least one new metadata.

[0078] In the case that there are multiple at least one new metadata, the electronic device may determine the target relevance of the multiple new metadata as the target relevance of the at least one new metadata by taking an average value, a sum, a weighted average value, or a maximum value.

[0079] In this embodiment, the electronic device first determines N Chinese similarities and N English similarities, and then determines the target association based on the N Chinese similarities and the N English similarities. Since the matching method combining the Chinese and English similarities can simultaneously match from the two perspectives of Chinese name and English name, the accuracy of matching is improved, and the accuracy of association is improved.

[0080] Next, the process of determining the target relevance of the new metadata based on at least the N Chinese similarities and the N English similarities of the new metadata in S1033 is described.

[0081] The process may include but is not limited to implementation 1 or implementation 2 described below.

[0082] Implementation 1. Determine the target relevance of the new metadata based on the N Chinese similarities and the N English similarities of the new metadata.

[0083] Implementation 2: Determine the target relevance of the new metadata based on the N Chinese similarities, the N English similarities of the new metadata and the record information of the N reference metadata.

[0084] Next, the process of determining the target relevance of the new metadata based on the N Chinese similarities and the N English similarities of the new metadata in implementation 1 is described.

[0085] refer to Figure 3 As shown in the content, the process may include but is not limited to the following S301 and S302.

[0086] S301: The electronic device determines N relevances of new metadata based on N Chinese similarities and N English similarities of the new metadata.

[0087] A Chinese similarity is the Chinese similarity between a new metadata and a reference metadata, and an English similarity is the English similarity between a new metadata and a reference metadata.

[0088] Correspondingly, based on the same reference metadata, a relevance corresponding to the reference metadata can be determined based on the Chinese similarity and English similarity corresponding to the reference metadata. By traversing all the reference metadata, N relevances can be obtained.

[0089] The embodiment of the present application does not limit the manner in which a correlation degree corresponding to the reference metadata is determined based on the Chinese similarity and English similarity corresponding to the reference metadata, and can be configured according to actual needs. For example, the Chinese similarity and English similarity corresponding to the reference metadata can be summed, or weighted, to determine a correlation degree corresponding to the reference metadata. For the weighted sum, the corresponding weight can be configured based on actual needs. For example, if Chinese is more important, the weight of the Chinese similarity is configured to be larger, and if English is more important, the weight of the English similarity is configured to be larger.

[0090] S302: The electronic device determines a correlation degree that meets a condition among the N correlation degrees as a target correlation degree of new metadata.

[0091] The conditions include: the first A correlation degrees sorted from large to small; A is greater than or equal to 1.

[0092] In this embodiment, the correlation degree can be obtained directly through the similarity without adding other information, which has the characteristic of simple implementation. In addition, the similarity and the correlation degree are strongly correlated, so the accuracy is also high.

[0093] Next, the process of determining the target relevance of the new metadata based on the N Chinese similarities, N English similarities and N reference metadata record information of the new metadata in Implementation 2 is described.

[0094] refer to Figure 4 As shown in the content, the process may include but is not limited to the following S401 to S404.

[0095] S401: The electronic device determines L similar reference metadata from N reference metadata based on N Chinese similarities and N English similarities of new metadata.

[0096] L is less than N.

[0097] In a possible implementation, the Chinese similarity and the English similarity can be fused to obtain fused similarity, and the first L fused similarities are sorted in descending order to determine as L similar reference metadata. The fusion method can be direct addition or weighted summation.

[0098] It should be noted that, during the fusion process, it is necessary to fuse the Chinese similarity between the new metadata and a reference metadata, and the English similarity between the metadata and the same reference metadata.

[0099] In another possible implementation, M similar reference metadata in Chinese are selected based on Chinese similarity, and M similar reference metadata in English are selected based on English similarity, and the union of the two is determined as L similar reference metadata.

[0100] S402: The electronic device obtains L pieces of similar reference metadata record information from N pieces of reference metadata record information.

[0101] The record information is used to record the modification time, creation time, etc. of the reference metadata. The content of the record information is not limited in the embodiment of the present application and can be configured according to actual conditions.

[0102] For example, the record information may include but is not limited to at least one of the following: the number of calls of the reference metadata, the modification record, and the creation time.

[0103] There is no limitation on the method of obtaining the record information of the reference metadata. For example, the corresponding record information can be directly obtained from the storage location of the similar reference metadata in the data dictionary. Of course, the record information can also be obtained in other ways. For example, the number of calls can be obtained in the data dictionary or in the enterprise data map corresponding to the data dictionary.

[0104] The enterprise data graph may include the association relationship between multiple reference metadata, through which the call counts can be counted to obtain the call count of each reference metadata. The call count here generally refers to the call count in the recent period of time.

[0105] Of course, the number of calls in the enterprise data graph can also be synchronized to the data dictionary to obtain the number of calls through the data dictionary.

[0106] S403: The electronic device determines L relevances of the new metadata based on the record information of the L similar reference metadata, the Chinese similarities of the L similar reference metadata, and the English similarities of the L similar reference metadata.

[0107] Here, a correlation degree (the correlation degree between the new metadata and the similar reference metadata) can be obtained for a similar reference metadata.

[0108] The record information can characterize the importance of the reference metadata to a certain extent. For example, the more important it is, the more it is generally called and modified. Therefore, the relevance can be determined by combining the record information with the similarity.

[0109] The embodiment of the present application does not limit the specific method of combining record information to determine the correlation degree, and can be configured according to actual needs.

[0110] S404: The electronic device determines a correlation degree that meets the conditions among the L correlation degrees as a target correlation degree of the new metadata.

[0111] The conditions include: the first B correlations sorted from largest to smallest; B is greater than or equal to 1.

[0112] In this embodiment, the record information and similarity are combined when determining the relevance, and the important ones can be processed first. That is, when system resources are limited, adding important and dissimilar new metadata to the data dictionary can improve the importance and high frequency of the data dictionary.

[0113] The following describes the process in S401 of the electronic device determining L similar reference metadata from N reference metadata based on the N Chinese similarities and N English similarities of the new metadata.

[0114] refer to Figure 5As shown in the content, the process may include but is not limited to the following S4011 to S4013.

[0115] S4011. The electronic device determines M Chinese-similar reference metadata from N reference metadata based on the N Chinese similarities of the new metadata.

[0116] M is less than N.

[0117] Here, the N Chinese similarities can be directly sorted in descending order, and the first M can be determined as Chinese similarity reference metadata.

[0118] S4012: The electronic device determines M English-similar reference metadata from the N reference metadata based on the N English similarities of the new metadata.

[0119] Here, the N English similarities can be directly sorted in descending order, and the first M can be determined as English similarity reference metadata.

[0120] S4013: The electronic device determines that the L similar reference metadata are a union of the M Chinese similar reference metadata and the N English similar reference metadata.

[0121] L is less than or equal to twice M; L is less than or equal to N.

[0122] That is, for reference metadata with high similarity in both Chinese and English, only one is retained and duplicate removal is required.

[0123] In this implementation, the L similar reference metadata are the union of M English similar reference metadata and M Chinese similar reference metadata. Thus, as long as one of the Chinese or English is similar, it can be determined as similar reference metadata, thus avoiding omission.

[0124] Next, the process of determining L relevance degrees of the new metadata based on the record information of the L similar reference metadata, the Chinese similarities of the L similar reference metadata, and the English similarities of the L similar reference metadata in S403 is described.

[0125] refer to Figure 6 As shown in the content, the process may include but is not limited to the following S4031 to S4034.

[0126] For each of the L similar reference metadata, the following S4031 to S4034 are performed.

[0127] S4031. The electronic device multiplies the Chinese similarity of the similar reference metadata by the Chinese influence weight to obtain the Chinese influence of the similar reference metadata.

[0128] You can configure the Chinese influence weight according to actual needs. For example, if Chinese is important, you can configure a larger Chinese influence weight.

[0129] S4032: The electronic device multiplies the English similarity of the similar reference metadata by the English influence weight to obtain the English influence of the similar reference metadata.

[0130] You can configure the English influence weight according to actual needs. For example, if English is important, you can configure a larger English influence weight.

[0131] S4033: The electronic device determines the influence of the record information corresponding to the record information of the similar reference metadata.

[0132] Here, for different record information, the relationship between the record information influence is predetermined, so when record information with similar reference metadata is obtained, the record information influence corresponding to the record information can be determined by searching or calculating.

[0133] The recorded information includes at least one of the following:

[0134] Reference metadata call count, modification records, and creation time;

[0135] Among them, the more times the call is made, the smaller the impact of the recorded information; the more the records are modified, the smaller the impact of the recorded information; the earlier the creation time, the smaller the impact of the record.

[0136] In the case of multiple record information, different weights may be configured for each record information, and then the total record information influence degree is determined according to the record information influence degrees of the multiple record information.

[0137] S4034: The electronic device sums the Chinese influence of the similar reference metadata, the English influence of the similar reference metadata, and the record information influence to obtain the correlation between the new metadata and the similar reference metadata.

[0138] In this embodiment, when determining the correlation between new metadata and similar reference metadata, the weights of Chinese influence, English influence and record information influence can be adjusted by weight, so that the obtained correlation can be adjusted according to actual needs, with high flexibility and wide application range.

[0139] Next, the process of the electronic device determining whether to add at least one new metadata to the data dictionary based on the target relevance of at least one new metadata in S104 is described.

[0140] This process may include but is not limited to the following case 1 or case 2.

[0141] Case 1: When the number of target relevance is one, how to determine whether to add at least one new metadata to the data dictionary;

[0142] Case 2: When there are multiple target associations, how to determine whether to add at least one new metadata to the data dictionary.

[0143] Next, a process of determining whether to add at least one new metadata to the data dictionary in case 1 when the number of target relevance is one is described.

[0144] If the target relevance belongs to the first relevance range; determining to add at least one new metadata to the data dictionary;

[0145] If the target relevance belongs to the second relevance range, then if at least one new metadata passes the review, it is determined that at least one new metadata is added to the data dictionary; if at least one new metadata fails the review, it is determined not to add at least one new metadata to the data dictionary;

[0146] If the target relevance belongs to the third relevance range, determining not to add the at least one new metadata to the data dictionary;

[0147] The first correlation range, the second correlation range and the third correlation range do not overlap each other, and the correlation in the first correlation range is smaller than that in the second correlation range, and the correlation in the second correlation range is smaller than that in the third correlation range.

[0148] For audit, the audit request can be output by the electronic device for manual audit, and then the electronic device receives the audit result input manually to determine whether the audit is passed. Adding the manual audit process can further improve accuracy and reduce the probability of duplication.

[0149] The embodiment of the present application does not limit the values ​​of the first correlation degree range, the second correlation degree range, and the third correlation degree range, which can be configured according to actual needs.

[0150] For example, the first correlation degree range may be [0-0.7), the second correlation degree range may be [0.7-0.9), and the third correlation degree range may be [0.9-1].

[0151] The implementation process has the characteristics of simple and clear implementation logic.

[0152] Next, the process of determining whether to add at least one new metadata to the data dictionary in case 2 when the number of target relevances is multiple will be described.

[0153] If the multiple target relevances all belong to the first relevance range; determining to add at least one new metadata to the data dictionary;

[0154] If there is a correlation degree belonging to the second correlation degree range among the multiple target correlation degrees, and there is a correlation degree belonging to the first correlation degree range or the third correlation degree range, then if at least one new metadata passes one review, it is determined to add the at least one new metadata to the data dictionary; if at least one new metadata fails one review, it is determined not to add the at least one new metadata to the data dictionary;

[0155] If the multiple target associations all belong to the second association range, then if at least one new metadata passes two reviews, it is determined that the at least one new metadata is added to the data dictionary; if at least one new metadata does not pass two reviews, it is determined not to add the at least one new metadata to the data dictionary;

[0156] If the multiple target associations all belong to the third association range, it is determined not to add at least one new metadata to the data dictionary, and a first reminder message is output; the first reminder message is used to remind the data dictionary to be modified;

[0157] The first correlation range, the second correlation range and the third correlation range do not overlap each other, and the correlation in the first correlation range is smaller than that in the second correlation range, and the correlation in the second correlation range is smaller than that in the third correlation range.

[0158] The first reminder information is used to remind the modification of the data dictionary. If a new metadata is very similar to a reference metadata (belonging to the third correlation range), it is determined that the new metadata is not added to the data dictionary. If a new metadata is very similar to two reference metadata at the same time, it means that the two reference metadata in the data dictionary are also very similar and have a certain probability of duplication. The first reminder information can be used to remind the two reference metadata in the data dictionary to be checked and modified.

[0159] In this embodiment, not only can it be effectively determined whether at least one new metadata needs to be added to the data dictionary, but it can also detect duplication problems in the data dictionary and remind through the first reminder information, thereby further reducing the probability of duplication in the data dictionary.

[0160] Next, the process in S1031 in which the electronic device matches the Chinese name of the field of the new metadata with the Chinese name of the field of each reference metadata in N reference metadata respectively to determine N Chinese similarities of the new metadata is described.

[0161] For a reference metadata, perform the first process:

[0162] refer to Figure 7 As shown in the content, the first processing process may include but is not limited to the following S701 and S702.

[0163] S701: The electronic device performs word segmentation on the Chinese names of the new metadata and the reference metadata respectively through a word segmentation model to determine a first word segmentation result and a second word segmentation result.

[0164] The embodiment of the present application does not specifically limit the word segmentation model, and can be configured according to actual needs. For example, the word segmentation model can be a hidden Markov model (HMM), a maximum entropy model, a conditional random field model, and the like.

[0165] HMM is a probability-based statistical model used to describe the relationship between an observation sequence and a hidden state. In Chinese word segmentation, the HMM model defines a sequence of character states (such as the beginning, middle, and end of a word) and uses the transition probabilities between these states to predict the most likely way to segment a word.

[0166] The Viterbi algorithm is a dynamic programming algorithm used to find the state sequence that is most likely to produce a given observation sequence under the HMM framework. It achieves efficient word segmentation by gradually eliminating impossible paths and eventually finding the global optimal path. The application of this algorithm in Chinese word segmentation includes four steps: initialization, recursion, termination, and backtracking to ensure that the state sequence with the highest probability is found.

[0167] For example, a hidden Markov model and a Viterbi algorithm may be combined as the word segmentation model.

[0168] S702: The electronic device compares the first word segmentation result and the second word segmentation result by using a first similarity algorithm to determine the Chinese similarity between the new metadata and the reference metadata.

[0169] The embodiment of the present application does not limit the type of the first similarity algorithm, and can be configured according to actual needs. For example, the first similarity algorithm can be a Best Match (BM) 25 algorithm, or a cosine similarity algorithm, an Okapi algorithm, and the like.

[0170] The BM25 algorithm is a classic information retrieval algorithm used to calculate the similarity between query terms and documents. Its core idea is to evaluate relevance by calculating factors such as the frequency of occurrence of query terms in documents, inverse document frequency, and document length.

[0171] In this embodiment, the Chinese names of the fields are first analyzed and sorted to unify the various descriptions, and then similarity matching is performed using a similarity algorithm. In this way, the influence of different people's expression order on the same thing can be effectively identified, so the obtained Chinese similarity is more accurate.

[0172] The following describes the process in S1032 where the electronic device matches the English name of the field of the new metadata with the English name of the field of each reference metadata in N reference metadata to determine N English similarities of the new metadata.

[0173] For a reference metadata, perform the second process: reference Figure 8 As shown in the content, the second processing process may include but is not limited to the following S801 and S802.

[0174] S801: The electronic device extracts features of the English names of the new metadata and the reference metadata respectively by using a vector space model to determine a first feature vector and a second feature vector.

[0175] In a possible implementation, the vector space model may be an Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) model.

[0176] S802: The electronic device compares the first feature vector and the second feature vector by using a second similarity algorithm to determine English similarity between the new metadata and the reference metadata.

[0177] The embodiment of the present application does not limit the type of the second similarity algorithm, and can be configured according to actual needs. For example, the second similarity algorithm can be a locality-sensitive hashing (LSH) or a Min Hash algorithm, etc.

[0178] The LSH algorithm hashes items in a data set into buckets so that similar items are mapped to the same bucket with a high probability, thereby achieving fast data search. LSH is particularly suitable for dealing with neighbor search problems in high-dimensional data sets, such as finding similar objects in images, texts, or other types of large-scale data.

[0179] In this embodiment, the feature extraction of the field English name is first performed, because the feature can reflect the essence, so as to unify various descriptions, and then similarity matching is performed through the similarity algorithm. In this way, the influence of different people on different expressions of the same thing can be effectively identified, so the obtained English similarity is more accurate.

[0180] The metadata processing method provided in the embodiment of the present application may further include: when it is determined to add at least one new metadata to the data dictionary, how to add it.

[0181] In a possible implementation, for a new metadata, if the new metadata has only one piece of data, the new metadata is directly added to the data dictionary.

[0182] In a possible implementation, for a new metadata, if the new metadata has multiple data, the new metadata corresponding to the primary key in the multiple new metadata is added to the data dictionary. Data is generally stored in the form of key-value pairs, and some primary keys are generally specified. Here, the new metadata corresponding to the primary key can be added to the data dictionary. This can effectively prevent duplication problems and make the metadata added to the data dictionary the desired metadata.

[0183] In the case where there is no primary key, new metadata with more important system sources and databases can be added to the data dictionary based on the system sources of multiple new metadata and the corresponding databases.

[0184] Below, taking a specific embodiment as an example, the processing process of metadata provided by the embodiment of the present application is explained.

[0185] At present, data modeling is based on the experience of architects. In the data modeling process, there are cases where the same name has different meanings, synonyms have different names, and field types and lengths vary greatly, which can easily lead to data swamps and make it difficult to find and use data. Often, more than a dozen people are needed to confirm the use of a piece of data. How to better standardize business system development and data development engineers to be more scientific and reasonable when designing data, so that business departments can efficiently use the group's shared data assets, reduce data costs, and increase data usage efficiency has become an urgent problem that enterprises need to solve.

[0186] The metadata processing process may include but is not limited to the following steps 1 to 5.

[0187] Step 1: Sort out the data dictionary of the current enterprise data lake, sort out various systems, various data sources, structured data and unstructured data, and perform corresponding analysis on unstructured data, and obtain unified standard metadata through OAI-PMH.

[0188] Various systems include but are not limited to enterprise systems, business systems, personnel systems, financial systems, file systems, and tracking systems.

[0189] A basic metadata dictionary is built based on structured and unstructured metadata information. The metadata dictionary may contain tens of thousands or even hundreds of thousands of fields.

[0190] Step 2: Build an enterprise data map based on the current data field usage of data lineage. The enterprise data map includes metadata operation information such as the data corresponding system and database type, data storage type, current data volume, current data call times (call times in 1 month and 6 months), and current data storage volume.

[0191] Step 3. Input: data table creation statement, including the field English name, field Chinese name, field type, and field length.

[0192] When creating a table in the database, you can also perform a quality check to preliminarily determine whether it complies with the database table creation specifications.

[0193] Step 4: Automatically disassemble metadata information, compare English similarity, compare Chinese similarity, select different algorithms based on the characteristics of Chinese and English, and calculate the similarity respectively.

[0194] Select the current usage of the five fields with the highest English name similarity, the number of calls in the past month, the number of calls in the past six months, the modification records, and the field creation time.

[0195] Select the current usage of the five fields with the highest Chinese name similarity, the number of calls in the past month, the number of calls in the past six months, the modification records, and the field creation time.

[0196] The metadata similarity calculation method in OAI-PMH is based on the vector space model, and the CRITIC analysis method is used to extract the English name feature vector of the field.

[0197] The field English name is similarity calculated using the LSH method to calculate the similarity value A of the field English name.

[0198] The Chinese name of the field is segmented using the HMM+Viterbi algorithm method, and the BM25 method is used to calculate the similarity value B of the Chinese name of the field.

[0199] Step 5: Import the Chinese and English similarity of the fields and the corresponding data into the data lake for real-time online analysis. This can fully utilize the advantages of distributed computing and tidal characteristics, reuse existing computing resources, and greatly reduce computing resource costs.

[0200] As old systems are taken offline and new systems are updated, it is also necessary to determine the life cycle of the corresponding system of the existing data. Some data that have been marked as being ready to be taken offline in the near future will not be included in the calculation and will be multiplied by different weights S=A×Q1+B×Q2+C.

[0201] Among them, S is equivalent to the above-mentioned relevance, A is equivalent to the above-mentioned English similarity, B is equivalent to the above-mentioned Chinese similarity, and C is equivalent to the above-mentioned record information influence.

[0202] The total similarity between the column field and the existing data dictionary is calculated by multiplying the similarity between the field English name and the field Chinese name by the corresponding weight value. Then the last possible similar historical metadata document is matched.

[0203] Select the three metadata fields with the highest matching degree between this field and the historical metadata information.

[0204] If the similarity of the Chinese and English in the field is both lower than 0.7, it is reasonable to add it to the data dictionary under the corresponding topic; if one of the Chinese and English in the field is between 0.7-0.85, it needs to be reviewed by an architect, and the new field needs to be added to the data dictionary directory; if both the Chinese and English in the field are between 0.85-0.9, it needs to be submitted to two or more architects for review and the new field needs to be added to the data dictionary directory; if both are above 0.9, it means that the table creation statement needs to be adjusted to ensure that there is no data between 0.9-1 and use the original data dictionary.

[0205] Related technologies all use an algorithm to directly parse the Chinese and English in a field without distinguishing between the Chinese and English matching similarities. This embodiment selects different word segmentation and similarity algorithm models based on the characteristics of Chinese and English, and makes a comprehensive judgment based on the data life cycle, which greatly enhances the scientific rationality of the judgment. Finally, the data type and data length are unified as much as possible, and the cycle is repeated to help enterprises achieve the unification of metadata roots, etc.

[0206] The algorithm model machine focuses on the GPU to store the summary calculation after comparing the similarity of the fields in the distributed data lake, using the CPU advantage to greatly reduce the computing cost.

[0207] Furthermore, the similarity of the entire table can be derived based on the similarity of the fields to see whether there are duplicate tables and the storage space occupied by the duplicate tables, helping enterprises reduce data costs.

[0208] In short, reference Fig. 9 As shown in the content, the metadata processing process may include but is not limited to the following S901 to S910.

[0209] S901, determine English similarity.

[0210] S902: Determine Chinese similarity.

[0211] S903. Import into the data lake for processing.

[0212] S904: Real-time data calculation.

[0213] S905: Determine the similarity between Chinese and English.

[0214] S906: Initiate a judgment process.

[0215] S907. By adding a data dictionary.

[0216] S908, failed, redesigned.

[0217] S909, Review.

[0218] If the audit succeeds, execute S910; if the audit fails, execute S908.

[0219] S910. Passed, added to the data dictionary.

[0220] In a second aspect, an embodiment of the present application provides a metadata processing device.

[0221] refer to Fig.10 As shown, the metadata processing device 100 includes a first acquisition unit 1001 , a second acquisition unit 1002 , a processing unit 1003 and a determination unit 1004 .

[0222] in:

[0223] The first acquisition unit 1101 is used to acquire a data dictionary; the data dictionary includes N reference metadata in a unified format; the reference metadata includes the Chinese and English names of the fields; N is an integer greater than 1;

[0224] The second acquisition unit 1002 is used to acquire at least one new metadata in the input file; the new metadata includes the Chinese and English names of the fields;

[0225] Processing unit 1003, configured to perform Chinese-English matching based on at least one Chinese and English name of the new metadata and the Chinese and English names of the N reference metadata, to obtain a target relevance of at least one new metadata; the target relevance is used to represent the degree of relevance between the new metadata and the data dictionary;

[0226] The determining unit 1004 is configured to determine whether to add the at least one new metadata to the data dictionary based on the target relevance of the at least one new metadata.

[0227] In some embodiments, the processing unit 1003 is further used to: for each new metadata in at least one new metadata, perform similarity matching between the Chinese name of the field of the new metadata and the Chinese name of the field of each reference metadata in N reference metadata, and determine N Chinese similarities of the new metadata; match the English name of the field of the new metadata with the English name of the field of each reference metadata in N reference metadata, and determine N English similarities of the new metadata; determine the target relevance of the new metadata based on at least the N Chinese similarities and the N English similarities of the new metadata; for each new metadata in at least one new metadata, determine the target relevance of at least one new metadata based on the target relevance of each new metadata.

[0228] In some embodiments, the processing unit 1003 is further configured to: determine a target relevance of the new metadata based on the N Chinese similarities and the N English similarities of the new metadata;

[0229] Alternatively, the target relevance of the new metadata is determined based on the N Chinese similarities, the N English similarities of the new metadata, and the record information of the N reference metadata.

[0230] In some embodiments, the processing unit 1003 is also used to: determine N associations of the new metadata based on the N Chinese similarities and N English similarities of the new metadata; determine the association that meets the conditions among the N associations as the target association of the new metadata; the conditions include: the first A associations sorted from large to small; A is greater than or equal to 1.

[0231] In some embodiments, the processing unit 1003 is also used to: determine L similar reference metadata from N reference metadata based on N Chinese similarities and N English similarities of the new metadata; L is less than N; obtain record information of L similar reference metadata from the record information of the N reference metadata; determine L associations of the new metadata based on the record information of the L similar reference metadata, the Chinese similarities of the L similar reference metadata, and the English similarities of the L similar reference metadata; determine the association that meets the conditions among the L associations as the target association of the new metadata; the conditions include: the first B associations sorted from large to small according to the associations; B is greater than or equal to 1.

[0232] In some embodiments, the processing unit 1003 is further used to: determine M Chinese similar reference metadata from N reference metadata based on the N Chinese similarities of the new metadata; M is less than N; determine M English similar reference metadata from N reference metadata based on the N English similarities of the new metadata; determine L similar reference metadata as the union of M Chinese similar reference metadata and N English similar reference metadata; L is less than or equal to twice M; L is less than or equal to N.

[0233] In some embodiments, the processing unit 1003 is also used to: perform the following processing for each similar reference metadata in the L similar reference metadata: multiply the Chinese similarity of the similar reference metadata by the Chinese influence weight to obtain the Chinese influence of the similar reference metadata; multiply the English similarity of the similar reference metadata by the English influence weight to obtain the English influence of the similar reference metadata; determine the record information influence corresponding to the record information of the similar reference metadata; sum the Chinese influence of the similar reference metadata, the English influence of the similar reference metadata, and the record information influence to obtain the association between the new metadata and the similar reference metadata.

[0234] In some embodiments, the record information includes at least one of the following: the number of calls of the reference metadata, modification records, and creation time; wherein, the more the number of calls, the smaller the impact of the record information; the more the modification records, the smaller the impact of the record information; the earlier the creation time, the smaller the impact of the record.

[0235] In some embodiments, when the number of target relevance is one, the determining unit 1004 is further configured to:

[0236] If the target relevance belongs to the first relevance range; determining to add at least one new metadata to the data dictionary;

[0237] If the target relevance belongs to the second relevance range, then if at least one new metadata passes the review, it is determined that at least one new metadata is added to the data dictionary; if at least one new metadata fails the review, it is determined not to add at least one new metadata to the data dictionary;

[0238] If the target relevance belongs to the third relevance range, determining not to add the at least one new metadata to the data dictionary;

[0239] The first correlation range, the second correlation range and the third correlation range do not overlap each other, and the correlation in the first correlation range is smaller than that in the second correlation range, and the correlation in the second correlation range is smaller than that in the third correlation range.

[0240] In some embodiments, when there are multiple target associations, the determining unit 1004 is further configured to:

[0241] If the multiple target relevances all belong to the first relevance range; determining to add at least one new metadata to the data dictionary;

[0242] If there is a correlation degree belonging to the second correlation degree range among the multiple target correlation degrees, and there is a correlation degree belonging to the first correlation degree range or the third correlation degree range, then if at least one new metadata passes one review, it is determined to add the at least one new metadata to the data dictionary; if at least one new metadata fails one review, it is determined not to add the at least one new metadata to the data dictionary;

[0243] If the multiple target associations all belong to the second association range, then if at least one new metadata passes two reviews, it is determined that the at least one new metadata is added to the data dictionary; if at least one new metadata does not pass two reviews, it is determined not to add the at least one new metadata to the data dictionary;

[0244] If the multiple target associations all belong to the third association range, it is determined not to add at least one new metadata to the data dictionary, and a first reminder message is output; the first reminder message is used to remind the data dictionary to be modified;

[0245] The first correlation range, the second correlation range and the third correlation range do not overlap each other, and the correlation in the first correlation range is smaller than that in the second correlation range, and the correlation in the second correlation range is smaller than that in the third correlation range.

[0246] In some embodiments, the processing unit 1003 is also used to: perform a first process on a reference metadata: perform word segmentation on the Chinese names of the new metadata and the reference metadata respectively through a word segmentation model to determine a first word segmentation result and a second word segmentation result; compare the first word segmentation result and the second word segmentation result through a first similarity algorithm to determine the Chinese similarity between the new metadata and the reference metadata; traverse N reference metadata, perform the first process on each reference metadata, and obtain N Chinese similarities of the new metadata.

[0247] In some embodiments, the processing unit 1003 is further configured to:

[0248] For a reference metadata, the second processing is performed: the English names of the new metadata and the reference metadata are respectively feature extracted by a vector space model to determine a first feature vector and a second feature vector; the first feature vector and the second feature vector are compared by a second similarity algorithm to determine the English similarity between the new metadata and the reference metadata.

[0249] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0250] It should be noted that in the embodiment of the present application, if the above-mentioned vehicle skylight adjustment method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0251] Fig.11 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Fig.11 As shown, the hardware entity of the electronic device 110 includes: a processor 1101 and a memory 1102, wherein the memory 1102 stores a computer program that can be run on the processor 1101, and the processor 1101 implements the steps in the method of any of the above embodiments when executing the program.

[0252] The memory 1102 stores computer programs that can be run on the processor. The memory 1102 is configured to store instructions and applications executable by the processor 1101. It can also cache data to be processed or processed by the processor 801 and various modules in the electronic device 110 (for example, image data, audio data, voice communication data, and video communication data). This can be implemented through flash memory (FLASH) or random access memory.

[0253] When the processor 1101 executes the program, the steps of any of the above metadata processing methods are implemented. The processor 1101 generally controls the overall operation of the electronic device 110.

[0254] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the metadata processing method of any of the above embodiments.

[0255] In one embodiment, the processor is configured to execute:

[0256] Acquire a data dictionary; the data dictionary includes N reference metadata in a unified format; the reference metadata includes the Chinese and English names of the fields; N is an integer greater than 1; acquire at least one new metadata in an input file; the new metadata includes the Chinese and English names of the fields; perform Chinese and English matching based on at least one new metadata's Chinese and English names with the N reference metadata's Chinese and English names, and obtain a target relevance of at least one new metadata; the target relevance is used to characterize the degree of relevance between the new metadata and the data dictionary; determine whether to add at least one new metadata to the data dictionary based on the target relevance of at least one new metadata.

[0257] In another embodiment, the processor includes a central processing unit and a graphics processing unit;

[0258] The central processing unit is used to execute: obtaining a data dictionary; the data dictionary includes N reference metadata in a unified format; the reference metadata includes the Chinese and English names of the fields; N is an integer greater than 1; obtaining at least one new metadata in the input file; the new metadata includes the Chinese and English names of the fields;

[0259] The graphics processor is used to execute: performing Chinese and English matching based on at least one Chinese and English name of the new metadata and the Chinese and English names of N reference metadata to obtain a target relevance of at least one new metadata; the target relevance is used to represent the degree of relevance between the new metadata and the data dictionary;

[0260] The central processor is further configured to determine whether to add the at least one new metadata to the data dictionary based on the target relevance of the at least one new metadata.

[0261] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0262] The processor may be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the function of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0263] The above-mentioned computer storage medium / memory can be a read-only memory, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0264] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes some or all of the steps in the above method.

[0265] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0266] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0267] The above are only implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for processing metadata, characterized in that: The method comprises: Obtain a data dictionary; the data dictionary includes N reference metadata in a unified format; the reference metadata includes the Chinese and English names of the fields; N is an integer greater than 1; Obtain at least one new metadata in the input file; the new metadata includes the English name of the field; At least based on matching the Chinese and English names of the at least one new metadata with the Chinese and English names of the N reference metadata, a target relevance of the at least one new metadata is obtained; the target relevance is used to represent the degree of relevance between the new metadata and the data dictionary; Based on the target relevance of the at least one new metadata, it is determined whether to add the at least one new metadata to the data dictionary.

2. The method according to claim 1, characterized in that The step of performing Chinese-English matching based on at least one of the Chinese and English names of the at least one new metadata and the Chinese and English names of the N reference metadata to obtain a target relevance of the at least one new metadata includes: For each of the at least one new metadata, respectively, the Chinese name of the field of the new metadata is matched with the Chinese name of the field of each of the N reference metadata to determine N Chinese similarities of the new metadata; Matching the English name of the field of the new metadata with the English name of the field of each of the N reference metadata respectively, and determining N English similarities of the new metadata; Determining a target relevance of the new metadata based on at least N Chinese similarities and N English similarities of the new metadata; For each of the at least one new metadata, a target relevance of the at least one new metadata is determined based on the target relevance of each of the new metadata.

3. The method according to claim 2, characterized in that The determining the target relevance of the new metadata based at least on the N Chinese similarities and the N English similarities of the new metadata includes: Determining a target relevance of the new metadata based on the N Chinese similarities and the N English similarities of the new metadata; or, Based on the N Chinese similarities, the N English similarities of the new metadata and the record information of the N reference metadata, a target relevance of the new metadata is determined.

4. The method according to claim 3, characterized in that: The determining the target relevance of the new metadata based on the N Chinese similarities and the N English similarities of the new metadata includes: Determining N relevances of the new metadata based on the N Chinese similarities and the N English similarities of the new metadata; The relevance degree that meets the conditions among the N relevance degrees is determined as the target relevance degree of the new metadata; the conditions include: the first A relevance degrees after being sorted from large to small in terms of relevance; and A is greater than or equal to 1.

5. The method according to claim 3, characterized in that: The determining the target relevance of the new metadata based on the N Chinese similarities, the N English similarities of the new metadata and the record information of the N reference metadata includes: Based on the N Chinese similarities and the N English similarities of the new metadata, determining L similar reference metadata from the N reference metadata; the L is less than the N; Acquire the record information of the L similar reference metadata from the record information of the N reference metadata; Determining L relevances of the new metadata based on the record information of the L similar reference metadata, the Chinese similarities of the L similar reference metadata, and the English similarities of the L similar reference metadata; The relevance degree that meets the conditions among the L relevance degrees is determined as the target relevance degree of the new metadata; the conditions include: the first B relevance degrees sorted from large to small; B is greater than or equal to 1.

6. The method according to claim 5, characterized in that The determining L similar reference metadata from the N reference metadata based on the N Chinese similarities and the N English similarities of the new metadata includes: Based on the N Chinese similarities of the new metadata, determining M Chinese similar reference metadata from the N reference metadata; wherein M is less than N; Based on the N English similarities of the new metadata, determining M English similar reference metadata from the N reference metadata; Determine that the L similar reference metadata are the union of the M Chinese similar reference metadata and the N English similar reference metadata; the L is less than or equal to twice the M; the L is less than or equal to the N.

7. The method according to claim 5, characterized in that The determining of the L relevances of the new metadata based on the record information of the L similar reference metadata, the Chinese similarities of the L similar reference metadata, and the English similarities of the L similar reference metadata includes: For each of the L similar reference metadata, the following process is performed: Multiplying the Chinese similarity of the similar reference metadata by the Chinese influence weight to obtain the Chinese influence of the similar reference metadata; Multiplying the English similarity of the similar reference metadata by the English influence weight to obtain the English influence of the similar reference metadata; Determining the impact of the record information corresponding to the record information of the similar reference metadata; The Chinese influence of the similar reference metadata, the English influence of the similar reference metadata, and the record information influence are summed to obtain the correlation between the new metadata and the similar reference metadata.

8. The method according to claim 7, characterized in that The recorded information includes at least one of the following: The call count, modification record, and creation time of the reference metadata; Among them, the more the calling times are, the smaller the impact of the record information is; the more the modification records are, the smaller the impact of the record information is; the earlier the creation time is, the smaller the impact of the record is.

9. The method according to any one of claims 1 to 8, characterized in that: In a case where the number of the target relevance is one, determining whether to add the at least one new metadata to the data dictionary based on the target relevance of the at least one new metadata includes: If the target relevance belongs to the first relevance range; determining to add the at least one new metadata to the data dictionary; If the target relevance belongs to the second relevance range, then if the at least one new metadata passes the review, it is determined to add the at least one new metadata to the data dictionary; if the at least one new metadata fails the review, it is determined not to add the at least one new metadata to the data dictionary; If the target relevance belongs to a third relevance range, determining not to add the at least one new metadata to the data dictionary; Among them, the first association degree range, the second association degree range and the third association degree range do not overlap with each other, and the association degree in the first association degree range is smaller than the association degree in the second association degree range, and the association degree in the second association degree range is smaller than the association degree in the third association degree range.

10. The method according to any one of claims 1 to 8, characterized in that: In the case that the number of the target relevances is plural, determining whether to add the at least one new metadata to the data dictionary based on the target relevance of the at least one new metadata includes: If the multiple target associations all belong to the first association range, then determining to add the at least one new metadata to the data dictionary; If the multiple target associations include an association that belongs to the second association range, and include an association that belongs to the first association range or the third association range, then if the at least one new metadata passes one review, determining to add the at least one new metadata to the data dictionary; if the at least one new metadata does not pass one review, determining not to add the at least one new metadata to the data dictionary; If the multiple target associations all belong to the second association range, then if the at least one new metadata passes two reviews, it is determined to add the at least one new metadata to the data dictionary; if the at least one new metadata does not pass two reviews, it is determined not to add the at least one new metadata to the data dictionary; If the multiple target associations all belong to the third association range, it is determined not to add the at least one new metadata to the data dictionary, and a first reminder message is output; the first reminder message is used to remind the data dictionary to be modified; Among them, the first association degree range, the second association degree range and the third association degree range do not overlap with each other, and the association degree in the first association degree range is smaller than the association degree in the second association degree range, and the association degree in the second association degree range is smaller than the association degree in the third association degree range.

Images