Metadata repeatability analysis method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a metadata repeatability analysis method and device, equipment and a storage medium. The method comprises the steps of determining an adaptive dictionary identifier of target metadata under each attribute from a pre-constructed metadata industry dictionary according to attribute word segmentation of the target metadata under each attribute, the metadata industry dictionary is composed of a mapping relationship between each word segmentation identifier under each industry word meaning and a dictionary identifier corresponding to the industry word meaning; and according to the adaptive dictionary identifier of the target metadata under each attribute, determining repeated metadata of the target metadata from a pre-constructed metadata attribute word meaning bucket, the metadata attribute word meaning bucket being composed of a mapping relationship between a dictionary identifier corresponding to each industry word meaning and a metadata identifier set where each segmented word under the industry word meaning is located. According to the embodiment of the invention, real-time analysis of metadata repeatability is realized, memory resource consumption during metadata repeatability analysis is reduced, and comprehensiveness and accuracy of metadata repeatability analysis are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and specifically to a metadata repeatability analysis method, device, equipment and storage medium. Background Art

[0002] With the development of information technology and the advancement of enterprise digital transformation, metadata management has become increasingly important as key data related to data source definition, target definition, conversion rules, etc. generated during the construction of data warehouses. For metadata management, metadata duplication analysis is a core function that enables customers to easily identify and review duplicate metadata in business systems and improve metadata quality.

[0003] Traditional metadata duplication analysis solutions usually require all metadata to be loaded into memory to determine whether the specific attribute values ​​of each two metadata under the same attribute are consistent, so as to analyze whether each metadata is repeated. However, when all metadata are loaded into memory for comparison, the memory usage will increase sharply, resulting in huge memory resource consumption, and metadata with inconsistent attribute values ​​but similar meanings cannot be identified, which greatly affects the accurate analysis of metadata duplication. Summary of the invention

[0004] The embodiments of the present application provide a metadata repeatability analysis method, apparatus, device and storage medium to achieve real-time analysis of metadata repeatability, significantly reduce memory resource consumption during metadata repeatability analysis, and improve the comprehensiveness and accuracy of metadata repeatability analysis.

[0005] In a first aspect, an embodiment of the present application provides a metadata repeatability analysis method, the method comprising:

[0006] According to the attribute segmentation of the target metadata under each attribute, determining the adaptation dictionary identifier of the target metadata under each attribute from the pre-built metadata industry dictionary, wherein the metadata industry dictionary is composed of the mapping relationship between each segmentation identifier under each industry word meaning and the dictionary identifier corresponding to the industry word meaning;

[0007] According to the adaptation dictionary identifier of the target metadata under each attribute, the repeated metadata of the target metadata is determined from the pre-constructed metadata attribute word sense bucket, and the metadata attribute word sense bucket is composed of the mapping relationship between the dictionary identifier corresponding to each industry word sense and the metadata identifier set where each word segment under the industry word sense is located.

[0008] In a second aspect, an embodiment of the present application provides a metadata repeatability analysis device, the device comprising:

[0009] According to the attribute segmentation of the target metadata under each attribute, determining the adaptation dictionary identifier of the target metadata under each attribute from the pre-built metadata industry dictionary, wherein the metadata industry dictionary is composed of the mapping relationship between each segmentation identifier under each industry word meaning and the dictionary identifier corresponding to the industry word meaning;

[0010] According to the adaptation dictionary identifier of the target metadata under each attribute, the repeated metadata of the target metadata is determined from the pre-constructed metadata attribute word sense bucket, and the metadata attribute word sense bucket is composed of the mapping relationship between the dictionary identifier corresponding to each industry word sense and the metadata identifier set where each word segment under the industry word sense is located.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0012] A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the metadata repeatability analysis method provided in the first aspect of the present application.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the metadata repeatability analysis method provided in the first aspect of the present application.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the metadata repeatability analysis method provided in the first aspect of the present application.

[0015] The embodiment of the present application provides a metadata repetitive analysis method, device, equipment and storage medium. By analyzing the meaning of relevant segmentation words in the metadata industry, the corresponding metadata industry dictionary can be constructed by using the mapping relationship between each segmentation mark under each industry word meaning and the dictionary mark corresponding to the industry word meaning. Moreover, the corresponding metadata attribute word meaning bucket is constructed by using the mapping relationship between the dictionary mark corresponding to each industry word meaning and the metadata mark set where each segmentation word under the industry word meaning is located. Then, for any target metadata, according to the attribute segmentation of the target metadata under each attribute, the adaptation dictionary mark of the target metadata under each attribute can be determined from the metadata industry dictionary, and the metadata mark set corresponding to the adaptation dictionary mark can be determined from the metadata attribute word meaning bucket to obtain the repeated metadata of the target metadata, thereby realizing the real-time analysis of metadata repetitiveness, without the need to uniformly load all metadata into the memory to judge the repetitiveness with the target metadata, so as to significantly reduce the memory resource consumption during metadata repetitiveness analysis. Moreover, through the metadata industry dictionary and the metadata attribute word meaning bucket, the metadata where multiple segmentation words under the same word meaning are located can be associated to accurately identify multiple metadata with similar meanings as repeated metadata, thereby improving the comprehensiveness and accuracy of metadata repetitiveness analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A flowchart of a metadata repeatability analysis method shown in an embodiment of the present application;

[0018] Figure 2 A flowchart of a method for constructing a metadata industry dictionary according to an embodiment of the present application;

[0019] Figure 3 A flowchart of a method for constructing metadata attribute meaning buckets according to an embodiment of the present application;

[0020] Figure 4 This is a principle block diagram of a metadata repeatability analysis device shown in an embodiment of the present application;

[0021] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0024] In order to solve the problem that traditional metadata repeatability analysis solutions occupy a large amount of memory and cause great memory resource consumption, the embodiment of the present application designs a new metadata repeatability analysis solution. By analyzing the meanings of relevant segmentations in the metadata industry, the mapping relationship between each segmentation identifier under each industry segmentation meaning and the dictionary identifier corresponding to the industry segmentation meaning is constructed into a corresponding metadata industry dictionary, and the mapping relationship between the dictionary identifier corresponding to each industry segmentation meaning and the metadata identifier set where each segmentation under the industry segmentation meaning is located is constructed into a corresponding metadata attribute meaning bucket. Then, with the help of the metadata industry dictionary and the metadata attribute meaning bucket, all metadata that is completely consistent with any metadata or has a similar meaning can be fully identified, thereby realizing real-time analysis of metadata repeatability, significantly reducing memory resource consumption during metadata repeatability analysis, and improving the comprehensiveness and accuracy of metadata repeatability analysis.

[0025] Figure 1 This is a flowchart of a metadata repeatability analysis method shown in an embodiment of the present application. Figure 1 , the method may include the following steps:

[0026] S110, according to the attribute segmentation of the target metadata under each attribute, determine the adaptation dictionary identifier of the target metadata under each attribute from the pre-constructed metadata industry dictionary, and the metadata industry dictionary is composed of the mapping relationship between each segmentation identifier under each industry word meaning and the dictionary identifier corresponding to the industry word meaning.

[0027] Among them, the target metadata can be any metadata used to describe various business data in various business systems. Moreover, each metadata can usually fully describe the relevant business data from multiple angles. Then, in order to accurately analyze the repeatability between various metadata, it is usually necessary to perform repeatability analysis on the data content of each metadata from multiple angles, that is, it is first necessary to extract the data content from each metadata from each angle. In this application, various attributes of metadata can be used to represent the various angles that each metadata faces when describing relevant business data. Exemplarily, the various attributes of metadata can be the Chinese name, English name, business version, storage location, occupied space, etc. of the metadata.

[0028] Moreover, considering that the specific description content of each metadata for the relevant business data under each attribute is composed of various segmented words. Then, the present application can analyze the repetitiveness between the metadata by judging the similarity between the various segmented words contained in the specific description content of the relevant business data under the same attribute of each metadata, so as to ensure the accuracy of the metadata repetitiveness analysis. As for the different segmented words contained in each metadata under each attribute, it is usually possible to judge whether the different segmented words are similar by analyzing whether the meanings expressed by the different segmented words are the same.

[0029] From the above content, it can be seen that in order to analyze metadata duplication efficiently and conveniently, this application can perform statistics and word meaning analysis on various word segments involved in the metadata industry to build a corresponding metadata industry dictionary. In the metadata industry dictionary, multiple word segments with the same word meaning can be associated together, so that the word segments in other metadata that are similar to the word segment of any metadata under each attribute can be quickly and effectively found later.

[0030] Specifically, the present application can perform word segmentation processing on each existing metadata to obtain various word segmentations involved in the metadata industry. Then, by performing word meaning analysis on each word segmentation, the word segmentations with the same word meaning can be grouped together to obtain the word segmentations under each industry word meaning. Among them, the industry word meaning can represent the corresponding word meanings of various word segmentations involved in the metadata industry.

[0031] It can be understood that, for each industry word meaning, each segmentation under the industry word meaning is a similar segmentation, so the metadata of each segmentation under the industry word meaning will become duplicate metadata of each other. Therefore, for each segmentation under each industry word meaning, the present application can conduct a comprehensive analysis of each segmentation under the industry word meaning to randomly generate a unique comprehensive identification information for the industry word meaning as the dictionary identifier corresponding to the industry word meaning. Therefore, the present application can configure a corresponding mapping relationship between each segmentation identifier under each industry word meaning and the dictionary identifier corresponding to the industry word meaning, so as to construct a corresponding metadata industry dictionary, so that the metadata industry dictionary can use each dictionary identifier to divide each segmentation under a certain industry word meaning represented by each dictionary identifier together, so as to indicate that each segmentation identifier under any industry word meaning in the metadata industry dictionary can be mapped to the same dictionary identifier. Subsequently, by analyzing the relationship between the dictionary identifier corresponding to each industry word meaning and the metadata where each segmentation under the industry word meaning is located, the corresponding duplicate metadata can be quickly found.

[0032] The word segmentation identifier in the present application may be reference information that can uniquely represent the word segmentation, and may include but is not limited to the annotation code of the word segmentation, a hash code generated for the word segmentation using a hash algorithm, etc.

[0033] Therefore, when performing repetitive analysis on any target metadata, the present application will first separate the specific description content in the target metadata when used to describe the relevant business data under each attribute. Then, by performing corresponding word segmentation processing on the specific description content of the target metadata when used to describe the relevant business data under each attribute, the attribute word segmentation of the target metadata under each attribute can be obtained. Among them, the number of attribute word segmentations of the target metadata under each attribute can be one or more, and there is no limitation on this.

[0034] Furthermore, for each attribute participle of the target metadata under each attribute, the present application can perform corresponding processing on each attribute participle to determine the identification information of each attribute participle. Then, in the pre-built metadata industry dictionary, the dictionary identifier after mapping the identification information of each attribute participle under each attribute can be determined as the adaptation dictionary identifier of the target metadata under each attribute. At this time, the adaptation dictionary identifier after each attribute participle is mapped can correspond to all participles belonging to the same industry meaning as the attribute participle, that is, all similar participles of the attribute participle, so as to assist in the subsequent search for the metadata where all the above similar participles are located as duplicate metadata of the target metadata.

[0035] S120, according to the adaptation dictionary identifier of the target metadata under each attribute, determine the repeated metadata of the target metadata from the pre-constructed metadata attribute word meaning bucket, and the metadata attribute word meaning bucket is composed of the mapping relationship between the dictionary identifier corresponding to each industry word meaning and the metadata identifier set where each word segment under the industry word meaning is located.

[0036] On the basis that the specific description content of each metadata for the related business data under each attribute is composed of various participles, if any two metadata have similar participles under a certain attribute, it can be indicated that the two metadata have a certain degree of duplication.

[0037] The above-mentioned metadata industry dictionary records the mapping relationship between each word segment identifier under each industry word meaning and the dictionary identifier corresponding to the industry word meaning, but does not record the actual information of the metadata where each word segment is located under each industry word meaning, so it is impossible to find the repeated metadata of similar word segments.

[0038] Therefore, in addition to the metadata industry dictionary, the present application can also analyze the word segmentation identifier of each existing metadata under each attribute, and determine the dictionary identifier of the word segmentation identifier of each metadata under each attribute after being mapped in the above-mentioned metadata industry dictionary, to associate the metadata with the dictionary identifier. In this way, under the dictionary identifier corresponding to each industry word meaning, all metadata where each word segmentation under the industry word meaning is located can be associated, and a corresponding metadata identifier set can be formed. At this point, it can be determined that each metadata in the metadata identifier set associated with the dictionary identifier corresponding to each industry word meaning has a similar word segmentation under the industry word meaning under a certain attribute, indicating that each metadata in the metadata identifier set has a certain degree of duplication.

[0039] Therefore, the present application can configure a corresponding mapping relationship between the dictionary identifier corresponding to each industry word meaning and the metadata identifier set where each word segment under the industry word meaning is located, to construct the corresponding metadata attribute word meaning bucket, and to divide the metadata that are repeated to a certain extent according to the dictionary identifier corresponding to each industry word meaning, so as to facilitate the subsequent efficient and accurate analysis of metadata repeatability.

[0040] In this application, for the target metadata, after determining the adaptation dictionary identifier of the target metadata under each attribute, the metadata identifier set after mapping the adaptation dictionary identifier under each attribute can be determined in the pre-constructed metadata attribute word meaning bucket, so that each metadata in the metadata identifier set has similar word segments under the same word meaning as the target metadata under a certain attribute. Therefore, this application can use each metadata in the metadata identifier set as duplicate metadata of the target metadata, thereby realizing real-time analysis of metadata repeatability, without the need to uniformly load all metadata into the memory to determine the repeatability with the target metadata, so as to significantly reduce the memory resource consumption during metadata repeatability analysis.

[0041] In some implementable embodiments, in order to accurately analyze the repetitiveness of each metadata under different attributes, the present application can divide the metadata attribute meaning bucket into metadata attribute meaning sub-buckets under each attribute. That is, for each attribute, the metadata and the dictionary identifier can be associated under the attribute by analyzing the existing word segmentation identifier of each metadata under the attribute, and determining the dictionary identifier of the word segmentation identifier of each metadata under the attribute after being mapped in the above-mentioned metadata industry dictionary. In this way, under the dictionary identifier corresponding to each industry word meaning under the attribute, all metadata where each word segment under the industry word meaning under the attribute is located can be associated, and a corresponding metadata identifier set can be formed. Then, by configuring a corresponding mapping relationship between the dictionary identifier corresponding to each industry word meaning under the attribute and the metadata identifier set where each word segment under the industry word meaning under the attribute is located, a metadata attribute meaning sub-bucket under the attribute is constructed.

[0042] In the same way as above, the metadata attribute word meaning sub-bucket under each attribute can be constructed. Moreover, each metadata in the metadata identifier set associated with the dictionary identifier corresponding to each industry word meaning recorded in the metadata attribute word meaning sub-bucket under each attribute is a similar word segment under the industry word meaning under the attribute, indicating that each metadata in the metadata identifier set has a certain degree of duplication under the attribute.

[0043] Then, for the repeated metadata of the target metadata, the present application can determine it in the following manner: for each attribute, according to the adaptation dictionary identifier of the target metadata under the attribute, search for the corresponding target metadata identifier set from the metadata attribute meaning sub-bucket under the attribute; according to the metadata identifier in the target metadata identifier set, determine the repeated metadata of the target metadata under the attribute.

[0044] That is to say, for each attribute, after determining the adaptation dictionary identifier of the target metadata under the attribute, the target metadata identifier set after the adaptation dictionary identifier under the attribute is mapped can be determined in the metadata attribute meaning sub-bucket under the attribute, so that each metadata in the target metadata identifier set has similar word segments under the same meaning as the target metadata under the attribute. Therefore, the present application can obtain each metadata in the target metadata identifier set under each attribute by parsing each metadata identifier in the target metadata identifier set under the attribute, as the repeated metadata of the target metadata under the attribute.

[0045] In the same manner as above, the repeated metadata of the target metadata under each attribute can be determined.

[0046] In addition, since each metadata may have similar segmentation under different attributes, the target metadata may have the same repeated metadata under different attributes. That is, each repeated metadata of the target metadata may have similar segmentation with the target metadata under one or more attributes, so that the repetition degree between the target metadata and each repeated metadata is different.

[0047] In this application, in order to accurately analyze the repetition between each metadata, this application can pre-set the corresponding weight for each attribute by analyzing the influence of each attribute on the metadata repetition analysis. Furthermore, after determining the repeated metadata of the target metadata, this application can also determine the repetition between the target metadata and the repeated metadata according to the associated attribute weight of the repeated metadata.

[0048] Specifically, for each repeated metadata of the target metadata, the present application may analyze under which attributes similar segmentations exist between the repeated metadata and the target metadata, and use these attributes as associated attributes of the repeated metadata in the present application.

[0049] Exemplarily, when the metadata attribute meaning bucket is divided into metadata attribute meaning sub-buckets under each attribute, the present application can determine the metadata attribute meaning sub-buckets under which each repeated metadata of the target metadata is found, and then use these attributes as associated attributes of the repeated metadata.

[0050] Then, the present application can determine the weights of each associated attribute of each repeated metadata and the number of similar word segments that exist between the repeated metadata and the target metadata under each associated attribute. For each repeated metadata, the number of similar word segments that exist under each associated attribute of the repeated metadata can be weighted and summed according to the weights of each associated attribute of the repeated metadata to obtain the degree of repetition between the target metadata and the repeated metadata. Thus, the degree of repetition between the target metadata and each repeated metadata can be calculated, and a corresponding metadata repetition analysis report can be generated according to the repetition to accurately describe the repetition of the target metadata to the user.

[0051] The technical solution provided by the embodiment of the present application can construct a corresponding metadata industry dictionary by analyzing the meaning of the relevant word segments in the metadata industry and using the mapping relationship between each word segment identifier under each industry word meaning and the dictionary identifier corresponding to the industry word meaning. Moreover, the corresponding metadata attribute word meaning bucket is constructed by using the mapping relationship between the dictionary identifier corresponding to each industry word meaning and the metadata identifier set where each word segment under the industry word meaning is located. Then, for any target metadata, according to the attribute word segmentation of the target metadata under each attribute, the adaptation dictionary identifier of the target metadata under each attribute can be determined from the metadata industry dictionary, and the metadata identifier set corresponding to the adaptation dictionary identifier can be determined from the metadata attribute word meaning bucket to obtain the repeated metadata of the target metadata, thereby realizing real-time analysis of metadata repetitiveness, without the need to uniformly load all metadata into the memory to judge the repetitiveness with the target metadata, so as to significantly reduce the memory resource consumption during metadata repetitiveness analysis. Moreover, through the metadata industry dictionary and the metadata attribute word meaning bucket, the metadata where multiple word segments under the same word meaning are located can be associated to accurately identify multiple metadata with similar meanings as repeated metadata, thereby improving the comprehensiveness and accuracy of metadata repetitiveness analysis.

[0052] As an optional implementation method in this application, this application mainly uses pre-built metadata industry dictionaries and metadata attribute word meaning buckets to comprehensively identify all metadata that have word segments that are completely consistent or have similar meanings to any metadata, thereby realizing real-time analysis of metadata repeatability, significantly reducing memory resource consumption during metadata repeatability analysis, and improving the comprehensiveness and accuracy of metadata repeatability analysis. Therefore, in order to ensure the accuracy of metadata repeatability analysis, this application first needs to build a complete and accurate metadata industry dictionary and metadata attribute word meaning bucket.

[0053] Next, the present application may provide detailed explanations on the specific construction processes of the metadata industry dictionary and metadata attribute word meaning buckets respectively.

[0054] Figure 2 A flowchart of a method for constructing a metadata industry dictionary according to an embodiment of the present application.

[0055] like Figure 2 As shown, the method may specifically include the following steps:

[0056] S210, for each industry word meaning group read in batches, determine each word segmentation identifier in the industry word meaning group, and generate a corresponding dictionary identifier for the industry word meaning group.

[0057] In order to accurately analyze the word segmentation information under each industry word in the metadata industry, this application will first read the relevant industry words in the metadata industry from the database in batches. For example, up to 500 relevant industry words can be read in each batch, so as to continuously obtain various word segments in the metadata industry.

[0058] Then, by performing corresponding statistics and semantic analysis on each segmented word read in each batch, the segmented words under the same industry semantics can be grouped together to obtain multiple segmented words under each industry semantics, thereby forming multiple industry semantics groups. Among them, each industry semantics can include multiple segmented words under the same industry semantics.

[0059] It is understandable that, for each industry word meaning group, each word segment in the industry word meaning group is a similar word segment representing the same industry word meaning, so each metadata where each word segment in the industry word meaning group is located will become each other's duplicate metadata.

[0060] Therefore, for each industry word meaning group read in batches, the present application can first perform corresponding identification processing on each word segment in the industry word meaning group, such as marking a corresponding identification code for each word segment or using a corresponding hash algorithm to generate a hash code for each word segment, etc., to obtain the identification information of each word segment in the industry word meaning group. Moreover, since each word segment in each industry word meaning group can express the same industry word meaning. Then, the present application can perform a comprehensive analysis on each word segment in each industry word meaning group to randomly generate a unique comprehensive identification information for the industry word meaning group as the dictionary identifier corresponding to the industry word meaning represented by the industry word meaning group.

[0061] S220: construct a corresponding metadata industry dictionary according to the mapping relationship between each word segmentation identifier in the industry word meaning group and the dictionary identifier generated for the industry word meaning group.

[0062] For each industry word meaning group, after determining the identification information of each word segment in the industry word meaning group and the dictionary identifier corresponding to the industry word meaning represented by the industry word meaning group, the present application can configure a corresponding mapping relationship between each word segment identifier in the industry word meaning group and the dictionary identifier corresponding to the industry word meaning represented by the industry word meaning group, so as to construct a corresponding metadata industry dictionary, so that the metadata industry dictionary can use each dictionary identifier to divide each word segment under each industry word meaning together, so as to indicate that each word segment identifier under any industry word meaning used in the metadata industry dictionary can be mapped to the same dictionary identifier. Subsequently, by analyzing the relationship between the dictionary identifier corresponding to each industry word meaning and the metadata where each word segment under the industry word meaning is located, the corresponding duplicate metadata can be quickly found.

[0063] In an exemplary solution, the present application can use a key-value pair approach to configure a corresponding mapping relationship between each word segmentation identifier in each industry word meaning group and the dictionary identifier corresponding to the industry word meaning represented by the industry word meaning group. That is, for each industry word meaning group, each word segmentation identifier in the industry word meaning group can be used as a key value, and the dictionary identifier corresponding to the industry word meaning group can be used as a value value to form a key-value pair and store it in the corresponding hash table to construct the corresponding metadata industry dictionary.

[0064] Considering that some of the word segments in each metadata may not belong to the word segmentation in the metadata industry, some of the word segmentations in each metadata will not exist in the metadata industry dictionary. Then, in order to efficiently and accurately analyze whether the attribute word segmentation under each attribute of the target metadata has been stored in the metadata industry dictionary, this application will also construct a bit array when constructing the metadata industry dictionary to indicate whether each word segmentation exists in the metadata industry dictionary.

[0065] Specifically, when constructing the corresponding metadata industry dictionary, for each industry word meaning group, the present application can also determine the index position of each word segment in the industry word meaning group in the constructed bit array, and update the bit mark at the index position to the stored mark to determine whether the attribute segment under each attribute of the target metadata has been stored in the metadata industry dictionary.

[0066] That is, when storing the mapping relationship between each word segment identifier in each industry word meaning group and the dictionary identifier corresponding to the industry word meaning group in the metadata industry dictionary, the present application can also use a corresponding hash function to calculate the position of each word segment in each industry word meaning group in the constructed bit array as the index position in the present application.

[0067] In the initial case, the bit marker of each position in the constructed bit array is a non-stored marker, for example, the non-stored marker can be "0". Then, after determining the index position of each word segment in each industry word meaning group in the constructed bit array, the bit marker of the index position can be updated to a stored marker, for example, the stored marker can be "1", to indicate that each word segment in each industry word meaning group has been stored in the metadata industry dictionary. Then, when subsequently analyzing the adaptation dictionary marker after the attribute segmentation mapping of the target metadata under each attribute from the metadata industry dictionary, the present application can first determine the index position of the attribute segmentation of the target metadata under each attribute in the bit array, and determine whether the bit marker at the index position is a stored marker, so as to determine whether there is an adaptation dictionary marker after the attribute segmentation mapping of the target metadata under each attribute in the metadata industry dictionary.

[0068] Figure 3 The flowchart of the method for constructing metadata attribute word meaning buckets shown in the embodiment of the present application. Figure 3 As shown, the method may specifically include the following steps:

[0069] S310 , for each historical metadata that has been read, performing word segmentation processing on the associated text under each attribute of the historical metadata to obtain attribute word segmentation of each attribute of the historical metadata.

[0070] In order to ensure the comprehensiveness of metadata repeatability analysis, this application needs to associate all existing metadata with dictionary identifiers corresponding to different industry word meanings. Therefore, this application will first continuously read the corresponding table column structure from the database to continuously obtain the corresponding historical metadata. Then, for each historical metadata that has been read, this application can first split out the specific description content of the historical metadata used to describe the relevant business data under each attribute as the associated text of the historical metadata under each attribute. Furthermore, by performing corresponding word segmentation processing on the associated text of the historical metadata under each attribute and removing the corresponding stop words, the attribute segmentation of the historical metadata under each attribute can be obtained.

[0071] S320, according to the attribute segmentation of the historical metadata under each attribute, determine the adaptation dictionary identifier under each attribute of the historical metadata from the metadata industry dictionary.

[0072] For each attribute segmentation under each attribute of the historical metadata, the present application can perform corresponding processing on each attribute segmentation to determine the identification information of each attribute segmentation. Then, in the pre-built metadata industry dictionary, the dictionary identification mapped with the identification information of each attribute segmentation under each attribute can be determined as the adaptation dictionary identification under each attribute of the historical metadata.

[0073] In order to quickly and effectively analyze the adaptation dictionary identifier of the historical metadata under each attribute, when a bit array has been constructed in the metadata industry dictionary, the present application can determine the adaptation dictionary identifier of the historical metadata under each attribute in the following manner: for each attribute, determine the target index position of the attribute segmentation of the historical metadata under the attribute in the bit array; if the bit identifier of the target index position is a stored identifier, determine the mapping dictionary identifier of the attribute segmentation identifier of the historical metadata under the attribute from the metadata industry dictionary as the adaptation dictionary identifier of the historical metadata under the attribute; if the bit identifier of the target index position is a non-stored identifier, determine that the adaptation dictionary identifier of the historical metadata under the attribute is empty.

[0074] That is, after determining the attribute segmentation of the historical metadata under each attribute, the present application will first determine whether the attribute segmentation of the historical metadata under each attribute has been stored in the metadata industry dictionary. Specifically, for each attribute, the present application can use a corresponding hash function to calculate the position of each attribute segmentation of the historical metadata under the attribute in the constructed bit array as the target index position in the present application, and determine whether the bit mark at the target index position is a stored mark or an unstored mark.

[0075] If the bit mark of the target index position is a stored mark, it means that the corresponding attribute segmentation of the historical metadata under the attribute has been stored in the metadata industry dictionary. Then, the present application can determine the dictionary mark after the attribute segmentation mark of the historical metadata under the attribute is mapped from the metadata industry dictionary as the adaptation dictionary mark of the historical metadata under the attribute. If the bit mark of the target index position is a non-stored mark, it means that the corresponding attribute segmentation of the historical metadata under the attribute has not been stored in the metadata industry dictionary. Then, the present application can directly determine that the adaptation dictionary mark of the historical metadata under the attribute is empty without searching from the metadata industry dictionary, thereby ensuring the efficient determination of the adaptation dictionary mark of the historical metadata under each attribute.

[0076] S330: Add the identification information of the historical metadata to the metadata identification set corresponding to the adaptation dictionary identification under each attribute of the historical metadata.

[0077] After determining the adaptation dictionary identifier of the historical metadata under each attribute, the present application may associate the adaptation dictionary identifier with the identification information of the historical metadata to indicate the mapping relationship between each dictionary identifier and each historical metadata.

[0078] In this application, since the above-mentioned metadata industry dictionary records the mapping relationship between each word segmentation identifier under each industry word meaning and the dictionary identifier corresponding to the industry word meaning, but does not record the actual information of the metadata where each word segmentation under each industry word meaning is located, it is impossible to find the repeated metadata of similar word segments. Therefore, for each dictionary identifier existing in the metadata industry dictionary, this application can pre-configure a metadata identifier set for each dictionary identifier. In the initial case, the metadata identifier set corresponding to each dictionary identifier is empty.

[0079] Then, for each historical metadata, after determining the adaptation dictionary identifier of the historical metadata under each attribute, the identification information of the historical metadata can be added to the metadata identifier set corresponding to the adaptation dictionary identifier of the historical metadata under each attribute, so that each metadata in the metadata identifier set has similar participles with the same meaning under the corresponding attribute, and ensures that each metadata in the metadata identifier set is a duplicate metadata of each other.

[0080] As the identification information of each historical metadata is continuously added to the metadata identification set corresponding to the corresponding dictionary identification, all metadata of each word segment under the industry word meaning can be associated under the dictionary identification corresponding to each industry word meaning to form a corresponding metadata identification set.

[0081] S340: Construct corresponding metadata attribute word meaning buckets according to the metadata identifier sets corresponding to the dictionary identifiers under each attribute.

[0082] After completing the analysis of historical metadata, the metadata identifier set corresponding to each dictionary identifier can be obtained under each attribute. Therefore, according to the metadata identifier set corresponding to each dictionary identifier under each attribute, a corresponding mapping relationship can be configured between each dictionary identifier and the metadata identifier set corresponding to the dictionary identifier under each attribute to construct the metadata attribute meaning sub-bucket under each attribute. By merging the metadata attribute meaning sub-buckets under each attribute, the corresponding metadata attribute meaning bucket can be constructed.

[0083] Figure 4 This is a block diagram of a metadata repeatability analysis device shown in an embodiment of the present application. Figure 4 As shown, the apparatus 400 may include:

[0084] The dictionary identifier determination module 410 is used to determine the adaptation dictionary identifier of the target metadata under each attribute from the pre-built metadata industry dictionary according to the attribute segmentation under each attribute of the target metadata, wherein the metadata industry dictionary is composed of the mapping relationship between each segmentation identifier under each industry word meaning and the dictionary identifier corresponding to the industry word meaning;

[0085] The duplicate metadata determination module 420 is used to determine the duplicate metadata of the target metadata from the pre-constructed metadata attribute word meaning bucket according to the adaptation dictionary identifier under each attribute of the target metadata. The metadata attribute word meaning bucket is composed of the mapping relationship between the dictionary identifier corresponding to each industry word meaning and the metadata identifier set where each word segment under the industry word meaning is located.

[0086] In some implementations, the metadata attribute word meaning bucket includes metadata attribute word meaning sub-buckets under each attribute. The repeated metadata determination module 420 can be specifically used to:

[0087] For each attribute, according to the adaptation dictionary identifier of the target metadata under the attribute, a corresponding target metadata identifier set is searched from the metadata attribute meaning sub-bucket under the attribute;

[0088] According to the metadata identifier in the target metadata identifier set, duplicate metadata of the target metadata under the attribute is determined.

[0089] In some implementations, the metadata repeatability analysis device 400 may further include a repeatability determination module. The repeatability determination module may be used to:

[0090] The degree of repetition between the target metadata and the repeated metadata is determined according to the associated attribute weight of the repeated metadata.

[0091] In some implementations, the metadata repeatability analysis device 400 may further include an industry dictionary building module. The industry dictionary building module may be used to:

[0092] For each industry word meaning group read in batches, determine each word segmentation identifier in the industry word meaning group, and generate a corresponding dictionary identifier for the industry word meaning group;

[0093] According to the mapping relationship between each word segmentation identifier in the industry word meaning group and the dictionary identifier generated for the industry word meaning group, a corresponding metadata industry dictionary is constructed.

[0094] In some implementations, the industry dictionary building module can also be used to:

[0095] The index position of each word segment in the industry word meaning group in the constructed bit array is determined, and the bit mark at the index position is updated to a stored mark to determine whether the attribute word segment under each attribute of the target metadata has been stored in the metadata industry dictionary.

[0096] In some implementations, the metadata repeatability analysis device 400 may further include an attribute word meaning bucket construction module. The attribute word meaning bucket construction module may be used to:

[0097] For each historical metadata that has been read, performing word segmentation processing on the associated text of the historical metadata under each attribute to obtain the attribute word segmentation of the historical metadata under each attribute;

[0098] According to the attribute segmentation of the historical metadata under each attribute, determining the adaptation dictionary identifier of the historical metadata under each attribute from the metadata industry dictionary;

[0099] Adding the identification information of the historical metadata to the metadata identification set corresponding to the adaptation dictionary identification under each attribute of the historical metadata;

[0100] According to the metadata identifier set corresponding to each dictionary identifier under each attribute, the corresponding metadata attribute word meaning bucket is constructed.

[0101] In some implementations, the attribute word meaning bucket construction module can be specifically used for:

[0102] For each attribute, determining a target index position of the attribute segmentation of the historical metadata under the attribute in the bit array;

[0103] If the bit mark of the target index position is a stored mark, determining a mapping dictionary mark of the attribute segmentation mark of the historical metadata under the attribute from the metadata industry dictionary as an adaptation dictionary mark of the historical metadata under the attribute;

[0104] If the bit mark of the target index position is a non-stored mark, it is determined that the adaptation dictionary mark of the historical metadata under the attribute is empty.

[0105] In the embodiment of the present application, by analyzing the meaning of the relevant word segments in the metadata industry, the mapping relationship between each word segment identifier under each industry word segment and the dictionary identifier corresponding to the industry word segment can be used to construct the corresponding metadata industry dictionary. Moreover, the mapping relationship between the dictionary identifier corresponding to each industry word segment and the metadata identifier set where each word segment under the industry word segment is located is used to construct the corresponding metadata attribute word segment bucket. Then, for any target metadata, according to the attribute word segment of the target metadata under each attribute, the adaptation dictionary identifier of the target metadata under each attribute can be determined from the metadata industry dictionary, and the metadata identifier set corresponding to the adaptation dictionary identifier can be determined from the metadata attribute word segment bucket to obtain the repeated metadata of the target metadata, thereby realizing the real-time analysis of metadata repeatability, without the need to uniformly load all metadata into the memory to judge the repeatability with the target metadata, so as to significantly reduce the memory resource consumption during metadata repeatability analysis. Moreover, through the metadata industry dictionary and the metadata attribute word segment bucket, the metadata where multiple word segments under the same meaning are located can be associated to accurately identify multiple metadata with similar meanings as repeated metadata, thereby improving the comprehensiveness and accuracy of metadata repeatability analysis.

[0106] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 4 The device 400 shown can execute any method embodiment in the present application, and the aforementioned and other operations and / or functions of each module in the device 400 are respectively for implementing the corresponding processes in each method in the embodiment of the present application. For the sake of brevity, they will not be repeated here.

[0107] The above describes the device 400 of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0108] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present application.

[0109] like Figure 5 As shown, the electronic device 500 may include:

[0110] The memory 510 and the processor 520, the memory 510 is used to store the computer program and transmit the program code to the processor 520. In other words, the processor 520 can call and run the computer program from the memory 510 to implement the method in the embodiment of the present application.

[0111] For example, the processor 520 may be configured to execute the above method embodiments according to instructions in the computer program.

[0112] In some embodiments of the present application, the processor 520 may include but is not limited to:

[0113] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0114] In some embodiments of the present application, the memory 510 includes but is not limited to:

[0115] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0116] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 510 and executed by the processor 520 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0117] like Figure 5 As shown, the electronic device may also include:

[0118] The transceiver 530 may be connected to the processor 520 or the memory 510 .

[0119] The processor 520 may control the transceiver 530 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 530 may include a transmitter and a receiver. The transceiver 530 may further include an antenna, and the number of antennas may be one or more.

[0120] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0121] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0122] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0123] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0124] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0125] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0126] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A metadata repeatability analysis method, characterized in that: include: According to the attribute segmentation of the target metadata under each attribute, determining the adaptation dictionary identifier of the target metadata under each attribute from the pre-built metadata industry dictionary, wherein the metadata industry dictionary is composed of the mapping relationship between each segmentation identifier under each industry word meaning and the dictionary identifier corresponding to the industry word meaning; According to the adaptation dictionary identifier of the target metadata under each attribute, the repeated metadata of the target metadata is determined from the pre-constructed metadata attribute word sense bucket, and the metadata attribute word sense bucket is composed of the mapping relationship between the dictionary identifier corresponding to each industry word sense and the metadata identifier set where each word segment under the industry word sense is located.

2. The method according to claim 1, characterized in that The metadata attribute word meaning bucket includes metadata attribute word meaning sub-buckets under each attribute; The step of determining the repeated metadata of the target metadata from the pre-constructed metadata attribute word meaning bucket according to the adaptation dictionary identifier of the target metadata under each attribute includes: For each attribute, according to the adaptation dictionary identifier of the target metadata under the attribute, a corresponding target metadata identifier set is searched from the metadata attribute meaning sub-bucket under the attribute; According to the metadata identifier in the target metadata identifier set, duplicate metadata of the target metadata under the attribute is determined.

3. The method according to claim 1, characterized in that The method further comprises: The degree of repetition between the target metadata and the repeated metadata is determined according to the associated attribute weight of the repeated metadata.

4. The method according to claim 1, characterized in that: The metadata industry dictionary is constructed in the following way: For each industry word meaning group read in batches, determine each word segmentation identifier in the industry word meaning group, and generate a corresponding dictionary identifier for the industry word meaning group; According to the mapping relationship between each word segmentation identifier in the industry word meaning group and the dictionary identifier generated for the industry word meaning group, a corresponding metadata industry dictionary is constructed.

5. The method according to claim 4, characterized in that When constructing a corresponding metadata industry dictionary according to the mapping relationship between each word segmentation identifier in the industry word meaning group and the dictionary identifier generated for the industry word meaning group, it also includes: The index position of each word segment in the industry word meaning group in the constructed bit array is determined, and the bit mark at the index position is updated to a stored mark to determine whether the attribute word segment under each attribute of the target metadata has been stored in the metadata industry dictionary.

6. The method according to claim 1, characterized in that The metadata attribute word meaning bucket is constructed in the following way: For each historical metadata that has been read, performing word segmentation processing on the associated text of the historical metadata under each attribute to obtain the attribute word segmentation of the historical metadata under each attribute; According to the attribute segmentation of the historical metadata under each attribute, determining the adaptation dictionary identifier of the historical metadata under each attribute from the metadata industry dictionary; Adding the identification information of the historical metadata to the metadata identification set corresponding to the adaptation dictionary identification under each attribute of the historical metadata; According to the metadata identifier set corresponding to each dictionary identifier under each attribute, the corresponding metadata attribute word meaning bucket is constructed.

7. The method according to claim 6, characterized in that The metadata industry dictionary includes a constructed bit array, and the method of determining the adaptation dictionary identifier of the historical metadata under each attribute from the metadata industry dictionary according to the attribute segmentation of the historical metadata under each attribute includes: For each attribute, determining a target index position of the attribute segmentation of the historical metadata under the attribute in the bit array; If the bit mark of the target index position is a stored mark, determining a mapping dictionary mark of the attribute segmentation mark of the historical metadata under the attribute from the metadata industry dictionary as an adaptation dictionary mark of the historical metadata under the attribute; If the bit mark of the target index position is a non-stored mark, it is determined that the adaptation dictionary mark of the historical metadata under the attribute is empty.

8. A metadata repeatability analysis device, characterized in that: include: A dictionary identifier determination module is used to determine the adaptation dictionary identifier of the target metadata under each attribute from a pre-built metadata industry dictionary according to the attribute segmentation under each attribute of the target metadata, wherein the metadata industry dictionary is composed of a mapping relationship between each segmentation identifier under each industry word meaning and a dictionary identifier corresponding to the industry word meaning; The duplicate metadata determination module is used to determine the duplicate metadata of the target metadata from the pre-constructed metadata attribute word meaning bucket according to the adaptation dictionary identifier under each attribute of the target metadata. The metadata attribute word meaning bucket is composed of the mapping relationship between the dictionary identifier corresponding to each industry word meaning and the metadata identifier set where each word segment under the industry word meaning is located.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the metadata repeatability analysis method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program enables a computer to execute the metadata repeatability analysis method according to any one of claims 1 to 7.

Images