Semantic type determination method and device for table data and storage medium

Abstract

The invention discloses a semantic type determination method and device for table data and a storage medium. The method comprises the following steps: processing predefined N types of semantic type data, and generating a reference feature value set corresponding to each type of semantic type data; performing feature extraction on data in a target column of the table by using X feature extraction operations in sequence to obtain a target feature value set corresponding to the target column, and detecting the similarity between the target feature value set obtained by each feature extraction operation and each reference feature value set; and determining the semantic type of the target column and / or the semantic type of each cell in the target column according to the similarity between the target feature value set obtained by each feature extraction operation and each reference feature value set. The technical problems that in the prior art, a large number of regular expressions need to be constructed to determine the semantic type of the table data, and consequently the semantic type recognition process of the table data is high in complexity and low in efficiency are solved.

Description

Technical Field

[0001] This application relates to the field of financial technology, and more specifically, to a method, apparatus, and storage medium for determining the semantic type of tabular data. Background Technology

[0002] Identifying the semantic type of tabular data relies on constructing a large number of regular expressions. To determine the semantic type of a column of data in a table, one or more regular expressions need to be created for each possible semantic type. These regular expressions are used to scan the data in the column to determine how many values ​​match a given semantic type. By repeating this process for each semantic type, the semantic type corresponding to the regular expression with the highest matching rate is identified as the semantic type of the target column. A large number of regular expressions need to be constructed to determine the semantic type of tabular data.

[0003] However, building and maintaining a large number of regular expressions requires significant time and effort, is costly, and demands substantial computing resources, especially when the semantic types are numerous and complex, making the process extremely cumbersome. The regular expression matching process requires multiple scans of the table data, and the semantic type recognition process is highly complex, consuming significant computing resources and reducing recognition efficiency.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method, apparatus, and storage medium for determining the semantic type of tabular data, which at least solves the technical problem in the prior art that the semantic type identification process of tabular data is highly complex and inefficient due to the need to construct a large number of regular expressions to determine the semantic type of tabular data.

[0006] According to one aspect of the embodiments of this application, a method for determining the semantic type of tabular data is provided, comprising: processing N predefined semantic type data to generate a reference feature value set corresponding to each semantic type data, wherein N is an integer greater than 1; sequentially using X feature extraction operations to extract features from the data in the target column of the table to obtain a target feature value set corresponding to the target column, and detecting the similarity between the target feature value set obtained by each feature extraction operation and each reference feature value set, wherein X is an integer greater than 1; determining the semantic type of the target column and / or the semantic type of each cell in the target column based on the similarity between the target feature value set obtained by each feature extraction operation and each reference feature value set.

[0007] Optionally, X feature extraction operations are used sequentially to extract features from the data in the target column of the table, obtaining a set of target feature values ​​corresponding to the target column. The similarity between the target feature value set obtained by each feature extraction operation and each reference feature value set is detected, including: if the similarity between the target feature value set obtained by the first feature extraction operation and at least one reference feature value set is greater than or equal to a preset threshold, the feature extraction operation is terminated; if the similarity between the target feature value set obtained by the first feature extraction operation and each reference feature value set is less than the preset threshold, the next feature extraction operation is used sequentially to extract the target feature value set corresponding to the target column, until the similarity between the target feature value set extracted by using a feature extraction operation and at least one reference feature value set is greater than or equal to the preset threshold, the feature extraction operation is terminated.

[0008] Optionally, the method further includes: if, after performing X feature extraction operations, no similarity is detected between the target feature value set extracted from the target column by any feature extraction operation and at least one reference feature value set that is greater than or equal to a preset threshold, then the semantic type of the target column and / or the semantic type of each cell in the target column belongs to an unknown semantic type.

[0009] Optionally, the semantic type of the target column and / or the semantic type of each cell in the target column are determined based on the similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set, including: if the similarity between the target feature value set obtained from the j-th feature extraction operation and at least one reference feature value set is greater than or equal to a preset threshold, then the reference feature value set with the highest similarity to the target feature value set is determined as the set with the highest reference value, where j is a positive integer less than or equal to X; the semantic type data corresponding to the set with the highest reference value is taken as the target data; and the semantic type of the target column and / or the semantic type of each cell in the target column are determined based on the semantic type of the target data.

[0010] Optionally, the X feature extraction operations include at least the following feature extraction operations: a first feature extraction operation, used to sort and remove duplicate strings from all data in the target column, and generate a set of target feature values ​​corresponding to the target column based on the strings obtained after sorting and removing duplicates; a second feature extraction operation, used to extract characters from preset positions of each data in the target column, sort and remove duplicates from all extracted characters, and generate a set of target feature values ​​corresponding to the target column based on the characters in the sorted and extracted positions.

[0011] Optionally, detecting the similarity between the target feature value set obtained in each feature extraction operation and each reference feature value set includes: constructing a binary array based on the target feature value set and the h-th reference feature value set, wherein each value in the binary array is used to characterize whether each character corresponding to the target feature value set is the same as the character corresponding to the h-th reference feature value set, and h is a positive integer less than or equal to N; obtaining the length of each feature value in the target feature value set and the length of each feature value in the h-th reference feature value set; and determining the similarity between the target feature value set and the h-th reference feature value set based on the binary array, the length of each feature value in the target feature value set, and the length of each feature value in the h-th reference feature value set.

[0012] Optionally, a binary array is constructed based on the target feature value set and the h-th reference feature value set, including: if the i-th character in the target feature value set is found to be the same as the character corresponding to any feature value in the h-th reference feature value set, then the i-th value in the binary array is set to 1, where i is an integer greater than or equal to 1; if the i-th character in the target feature value set is found to be different from all the characters corresponding to the h-th reference feature value set, then the i-th value in the binary array is set to 0.

[0013] Optionally, after obtaining the length of each feature value in the target feature value set and the length of each feature value in the h-th reference feature value set, the method further includes: constructing a first array based on the length of each feature value in the target feature value set, wherein the i-th value in the first array represents the length of the feature value generated based on the i-th character of the target column; and constructing a second array based on the length of each feature value in the h-th reference feature value set, wherein the i-th value in the second array represents the length of the feature value generated based on the i-th character of the h-th semantic type data, and the h-th semantic type data is the semantic type data corresponding to the h-th reference feature value set.

[0014] Optionally, the similarity between the target feature set and the h-th reference feature set is determined based on the binary array, the length of each feature value in the target feature set, and the length of each feature value in the h-th reference feature set, including: determining the similarity between the target feature set and the h-th reference feature set based on the ratio of the binary array to the first array and the ratio of the binary array to the second array.

[0015] According to another aspect of the embodiments of this application, a semantic type determination device for tabular data is also provided, comprising: a first processing unit, configured to process N predefined semantic type data to generate a reference feature value set corresponding to each semantic type data, wherein N is an integer greater than 1; a second processing unit, configured to sequentially use X feature extraction operations to extract features from the data in the target column of the table to obtain a target feature value set corresponding to the target column, and detect the similarity between the target feature value set obtained by each feature extraction operation and each reference feature value set, wherein X is an integer greater than 1; and a third processing unit, configured to determine the semantic type of the target column and / or the semantic type of each cell in the target column based on the similarity between the target feature value set obtained by each feature extraction operation and each reference feature value set.

[0016] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located performs the above-described method for determining the semantic type of tabular data.

[0017] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the above-described semantic type determination method for tabular data.

[0018] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implement the above-described method for determining the semantic type of tabular data.

[0019] In this application, firstly, N predefined semantic data types are processed to generate a reference feature value set corresponding to each semantic data type, where N is an integer greater than 1. Then, X feature extraction operations are sequentially used to extract features from the data in the target column of the table, obtaining a target feature value set corresponding to the target column. The similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set is then detected, where X is an integer greater than 1. Finally, based on the similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set, the semantic type of the target column and / or the semantic type of each cell in the target column is determined.

[0020] As described above, by predefining N semantic types and generating corresponding reference feature value sets, this application avoids the need to construct a large number of regular expressions for each semantic type, thereby simplifying the recognition process, reducing dependence on computing resources, and lowering the complexity of the recognition process, enabling the data processing system to quickly process large amounts of data. Furthermore, by using X feature extraction operations to analyze the target column data, this application can capture data features from multiple perspectives, improving the accuracy of recognition. The similarity calculation between the target feature value set generated by the feature extraction operations and the reference feature value set not only reduces the need for manual intervention but also improves the reliability of the recognition results. This application improves recognition efficiency while ensuring the accuracy of the recognition results, thus solving the technical problem in existing technologies where the construction of a large number of regular expressions to determine the semantic type of tabular data leads to high complexity and low efficiency in the semantic type recognition process. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1 This is a flowchart of an optional method for determining the semantic type of tabular data according to an embodiment of this application;

[0023] Figure 2 This is a schematic diagram of an optional semantic type determination device for tabular data according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] According to an embodiment of this application, a method embodiment for determining the semantic type of tabular data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] According to the embodiments of this application, a data processing system can be used as the execution subject of the semantic type determination method for tabular data in the embodiments of this application. The system can be a software system or an embedded system combining software and hardware. Of course, the execution subject of the method in the embodiments of this application can also be other forms of execution subject, such as devices, equipment, etc. It should be known by those skilled in the art that this application does not particularly limit the specific form of the execution subject.

[0028] Figure 1 This is a semantic type determination method for tabular data according to embodiments of this application, such as... Figure 1 As shown, the method includes the following steps:

[0029] Step S101: Process the predefined N semantic data types to generate a set of reference feature values ​​corresponding to each semantic data type, where N is an integer greater than 1.

[0030] Optionally, processing N predefined semantic data types to generate a reference feature value set corresponding to each semantic data type can involve feature extraction and processing of known data with clearly defined semantic types. The N semantic data types can be predefined, such as age, sales quantity, and inventory count, each with corresponding data features. Through feature extraction techniques such as extraction, sorting, and deduplication, each semantic data type can be converted into a reference feature value set. This reference feature value set will serve as a reference standard in subsequent identification processes, used for comparison with the feature value set of the target column.

[0031] This application embodiment can generate a representative set of reference feature values. This set of reference feature values ​​facilitates the data processing system's ability to quickly and accurately identify the semantic type of the target column. The set of reference feature values ​​can be stored in the data processing system for subsequent semantic type identification operations.

[0032] Step S102: Use X feature extraction operations in sequence to extract features from the data in the target column of the table, obtain the target feature value set corresponding to the target column, and detect the similarity between the target feature value set obtained by each feature extraction operation and each reference feature value set, where X is an integer greater than 1.

[0033] Optionally, X feature extraction operations can be used sequentially to extract features from the data in the target column of the table, resulting in a set of target feature values ​​for that column. This can refer to the process of performing feature extraction multiple times on a single column of data in the table. Specifically, the data processing system can apply X feature extraction operations sequentially, each extracting a set of feature values ​​from the target column to form a target feature value set. Feature extraction operations can include, but are not limited to, extraction, sorting, deduplication, and character extraction at specific positions. The data processing system can extract features from data from different perspectives, thus facilitating a more comprehensive description of the semantic type of the target column.

[0034] After feature extraction, the data processing system can detect the similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set. The reference feature value set can be pre-generated and used to represent feature value sets of known semantic types. By detecting the similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set, the data processing system can determine which known semantic type the target column's semantic type is closest to. Similarity calculation can be based on various methods; for example, it can construct a binary array to compare character matching and combine this with the length of the feature values ​​to calculate precision and recall, ultimately yielding a similarity value. This ensures the accuracy and efficiency of semantic type recognition.

[0035] Step S103: Based on the similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set, determine the semantic type of the target column and / or the semantic type of each cell in the target column.

[0036] Optionally, embodiments of this application can determine the overall semantic type of a target column or the specific semantic type of each cell by comparing the similarity between the feature value set of the target column and a reference feature value set of known semantic types. Specifically, the data processing system can sequentially use multiple feature extraction operations to generate a target feature value set and calculate the similarity between the target feature value set and a predefined reference feature value set. The similarity calculation can be achieved by constructing a binary array, calculating precision, and recall. Finally, the data processing system can determine the semantic type of the target column based on the similarity level, or further refine it to the semantic type of each cell. If the similarity is higher than a preset threshold, the semantic type of the target column or cell is considered to be consistent with the semantic type corresponding to the reference feature value set. If all feature extraction operations fail to find a semantic type with a similarity higher than the threshold, it is determined to be an unknown type. Through multiple feature extraction operations and similarity calculations, the accuracy and efficiency of semantic type recognition are improved.

[0037] Unlike existing technologies that construct regular expressions to determine the semantic type of tabular data, this application's embodiments eliminate the need to construct any regular expressions. Instead, it processes predefined N semantic types of data to generate a reference feature value set corresponding to each semantic type, which is a single data processing operation. Furthermore, for tabular data requiring semantic type determination, such as target column data, X feature extraction operations can be sequentially used to extract the target feature value set corresponding to the target column. Finally, based on the similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set, the semantic type of the target column and / or the semantic type of each cell in the target column is determined. This feature similarity comparison method can be directly implemented without constructing regular expressions, thereby reducing the difficulty of semantic type identification of tabular data and improving identification efficiency.

[0038] In one optional embodiment, X feature extraction operations are sequentially used to extract features from the data in the target column of the table, obtaining a set of target feature values ​​corresponding to the target column, and detecting the similarity between the target feature value set obtained by each feature extraction operation and each reference feature value set, including: if the data processing system detects that the similarity between the target feature value set obtained by the first feature extraction operation and at least one reference feature value set is greater than or equal to a preset threshold, the feature extraction operation ends; if the similarity between the target feature value set obtained by the first feature extraction operation and each reference feature value set is less than the preset threshold, the next feature extraction operation is sequentially attempted to extract the target feature value set corresponding to the target column, until the similarity between the target feature value set extracted using a feature extraction operation and at least one reference feature value set is greater than or equal to the preset threshold, the feature extraction operation ends.

[0039] Optionally, when performing semantic type recognition, the data processing system sequentially uses multiple feature extraction operations to generate a set of feature values ​​for the target column and compares its similarity with a predefined set of reference feature values. Specifically, the data processing system first processes the target column using a first feature extraction operation to generate a target feature value set. This first feature extraction operation can be, but is not limited to, extraction, sorting, and deduplication. Then, the data processing system sequentially calculates the similarity of this target feature value set with all reference feature value sets. If the similarity between the target feature value set and at least one reference feature value set is greater than or equal to a preset threshold, the data processing system immediately terminates the feature extraction operation, determining that the semantic type of the target column corresponds to the semantic type of the reference feature value set with the highest similarity. This allows for rapid determination of the semantic type and avoids unnecessary calculations.

[0040] If the similarity between the target feature value set obtained by the first feature extraction operation and all reference feature value sets is less than a preset threshold, the data processing system will not immediately give up. Instead, it will continue to try the next feature extraction operation, such as character extraction at a specific position. The data processing system will use X feature extraction operations in sequence until it finds a feature extraction operation whose target feature value set has a similarity greater than or equal to at least one reference feature value set. This allows the data processing system to find the most suitable feature extraction operation among the X operations, thereby improving the accuracy and reliability of semantic type recognition.

[0041] In this embodiment, the X feature extraction operations are not parallel operations, but serial operations. As long as the target feature value set obtained by one feature extraction operation has a similarity to at least one reference feature value set greater than or equal to a preset threshold, the feature extraction operation can be terminated. This can minimize the number of feature extraction operations and improve overall efficiency.

[0042] In an optional embodiment, the method further includes: if, after performing X feature extraction operations, the data processing system does not detect that the similarity between the target feature value set extracted from the target column by any feature extraction operation and at least one reference feature value set is greater than or equal to a preset threshold, then it determines that the semantic type of the target column and / or the semantic type of each cell in the target column belongs to an unknown semantic type.

[0043] Optionally, after executing all X preset feature extraction operations, the data processing system will perform a final judgment. If, among the X feature extraction operations, the similarity between the target feature value set generated and the reference feature value set reaches or exceeds a preset threshold, the data processing system will enter a special processing flow. Specifically, the data processing system will determine that the semantic type of the target column cannot be accurately matched using existing feature extraction methods and the reference feature value set. The data processing system can mark the semantic type of the target column and / or the semantic type of each cell in the target column as an unknown semantic type. Tabular data can contain various complex semantic types, some of which may not be predefined or difficult to identify through preset feature extraction operations. By marking unmatchable semantic types as unknown, the accuracy of identification is ensured, while unnecessary waste of computational resources is avoided. The data processing system not only avoids the potential risks of misidentification but also provides clear prompts for subsequent data analysis and processing. Simultaneously, it provides direction for the subsequent optimization and expansion of the data processing system, such as improving the system's recognition capabilities by adding new semantic types or improving feature extraction operations.

[0044] In one optional embodiment, determining the semantic type of the target column and / or the semantic type of each cell in the target column based on the similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set includes: if the data processing system detects that the similarity between the target feature value set obtained from the j-th feature extraction operation and at least one reference feature value set is greater than or equal to a preset threshold, then the reference feature value set with the highest similarity to the target feature value set is determined as the set with the highest reference value, where j is a positive integer less than or equal to X. Then, the semantic type data corresponding to the set with the highest reference value is used as the target data, and the semantic type of the target column and / or the semantic type of each cell in the target column is determined based on the semantic type of the target data.

[0045] Optionally, when performing semantic type recognition, the data processing system will sequentially process the target column using X feature extraction operations. When the data processing system performs the j-th feature extraction operation, if it detects that the similarity between the target feature value set generated by the j-th feature extraction operation and at least one reference feature value set is greater than or equal to a preset threshold, it will proceed to the next step of judgment. The data processing system will select the reference feature value set with the highest similarity from all matching reference feature value sets as the set with the highest reference value, so that the data processing system can select the semantic type that best matches the characteristics of the target column data from multiple possible semantic types.

[0046] After identifying the set with the highest reference value, the data processing system uses the semantic type data corresponding to this set as the target data. Based on the semantic type of this target data, the system ultimately determines the overall semantic type of the target column, or further refines it to the specific semantic type of each cell within the target column. This not only ensures the accuracy of semantic type recognition but also improves the flexibility and reliability of recognition through a combination of N feature extraction techniques. The data processing system can efficiently process complex data while providing accurate semantic information for subsequent data analysis and applications.

[0047] In one optional embodiment, the X feature extraction operations include at least the following feature extraction operations: a first feature extraction operation, used to sort and remove duplicate string elements of all data in the target column, and generate a target feature value set corresponding to the target column based on the string obtained after sorting and removing duplicate strings; a second feature extraction operation, used to extract characters from a preset position of each data in the target column, sort and remove duplicate characters from all extracted characters, and generate a target feature value set corresponding to the target column based on the characters in the sorted and extracted characters.

[0048] Optionally, the X feature extraction operations in the embodiments of this application include at least the following two specific operations:

[0049] The first feature extraction operation is based on processing the string elements of all data in the target column. Specifically, the data processing system can extract the string elements of each cell in the target column, sort all the string elements in the target column, and remove duplicate characters to obtain a sorted and deduplicated string. This string can reflect the overall characteristics of the data in the target column. The data processing system will then use the string obtained after sorting and deduplication to generate a set of target feature values ​​corresponding to the target column.

[0050] Table 1

[0051]

[0052] For example, Table 1 might be the first column of a master table storing string data types. While the database system directly indicates that the data type of this column is string, it doesn't provide a direct indication of its semantic type. If the target column in Table 1 contains the strings "EU", "EU", "YZ", and "SA", then after sorting and deduplication, the resulting target feature value set is the string "aesuyz". This effectively captures the overall character distribution characteristics of the target column data and is suitable for identifying semantic types with fixed character combinations or sets.

[0053] Table 2

[0054]

[0055] For example, Table 2 might be a master table storing all the strings of abbreviations of mainland names. The feature value obtained from this column is the feature value for the semantic type of mainland name abbreviations. After using the first feature extraction operation, the string "acefnosu" is obtained. After generating feature values ​​for both column 1 and the semantic type of mainland name abbreviations, the feature values ​​can be compared to obtain a similarity value.

[0056] The second type of feature extraction operation focuses on extracting characters from preset positions in each data point of the target column. The data processing system can extract characters from specific positions in each cell, such as the first character, the second character, etc., then sort and remove duplicates from the extracted characters, and finally generate a set of target feature values.

[0057] For example, in column 1 of Table 1, which serves as the target column, the second feature extraction operation can extract the leftmost characters "E", "E", "Y", and "S" from each row. After sorting and removing duplicate characters, the string "eys" is obtained as the target feature value set. The number of feature value sets that can be generated depends on the length of the longest content contained in the column. Using "U", "U", "Z", and "A" from the second position from the left generates "auz". Generating the target feature value set for the values ​​in column 2 using the second feature extraction operation yields "aenos" and "acfnsu", which can capture character features at specific positions in the target column data. This is suitable for identifying semantic types with specific positional patterns, such as abbreviations or codes.

[0058] This application embodiment combines two feature extraction operations—a first feature extraction operation and a second feature extraction operation—to enable the data processing system to analyze target column data from different perspectives, thereby more accurately identifying its semantic type.

[0059] In one optional embodiment, detecting the similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set includes: the data processing system constructing a binary array based on the target feature value set and the h-th reference feature value set, wherein each value in the binary array is used to characterize whether each character corresponding to the target feature value set is the same as the character corresponding to the h-th reference feature value set, and h is a positive integer less than or equal to N. Then, the length of each feature value in the target feature value set and the length of each feature value in the h-th reference feature value set are obtained. Finally, the similarity between the target feature value set and the h-th reference feature value set is determined based on the binary array, the length of each feature value in the target feature value set, and the length of each feature value in the h-th reference feature value set.

[0060] Optionally, the data processing system can calculate the similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set by constructing a binary array. Specifically, the data processing system can first perform character matching between the target feature value set and the h-th reference feature value set. The data processing system can check whether each character in the target feature value set is the same as any character in the h-th reference feature value set. If they are the same, the corresponding position in the binary array is marked as 1; otherwise, it is marked as 0.

[0061] Optionally, after constructing the binary array, the data processing system can obtain the length of each feature value in the target feature value set and the length of each feature value in the h-th reference feature value set. The feature value length information can be used to calculate precision P and recall E. Precision P can represent the proportion of characters in the target feature value set that match the reference feature value set, i.e., the ratio of the number of 1s in the binary array to the length of the target feature value set. Recall E can represent the proportion of characters in the reference feature value set that match the target feature value set, i.e., the ratio of the number of 1s in the binary array to the length of the reference feature value set. By comprehensively considering the proportion of matching characters and the set length, the similarity between the target feature value set and the reference feature value set can be evaluated more accurately, thus providing a reliable basis for semantic type recognition.

[0062] In one optional embodiment, a binary array is constructed based on the target feature value set and the h-th reference feature value set, including: if the data processing system detects that the i-th character in the target feature value set is the same as the character corresponding to any feature value in the h-th reference feature value set, then the i-th value in the binary array is set to 1, where i is an integer greater than or equal to 1. If the data processing system detects that the i-th character in the target feature value set is different from all characters corresponding to the h-th reference feature value set, then the i-th value in the binary array is set to 0.

[0063] Optionally, the data processing system evaluates the similarity between the target feature value set and the reference feature value set by constructing a binary array. Specifically, the data processing system can examine each character in the target feature value set one by one and compare it with all characters in the h-th reference feature value set. If the i-th character in the target feature value set is the same as the character corresponding to any feature value in the h-th reference feature value set, the data processing system can set the i-th value in the binary array to 1, indicating that the i-th character in the target feature value set has a match in the h-th reference feature value set, thus providing a positive signal for subsequent similarity calculations. For example, assuming the target feature value set is "abc" and the reference feature value set is "abcd", then the characters "a", "b", and "c" in the target feature value set will all find matches in the reference feature value set, and the corresponding binary array values ​​will be set to 1.

[0064] Conversely, if the i-th character in the target feature set is completely different from all characters in the h-th reference feature set, the data processing system can set the i-th value in the binary array to 0, indicating that the i-th character in the target feature set has no match in the h-th reference feature set, thus providing a negative signal for subsequent similarity calculations. For example, assuming the target feature set is "abc" and the reference feature set is "def", then the characters "a", "b", and "c" in the target set will not find matches in the reference feature set, and the corresponding binary array value will be set to 0. In this way, the binary array can clearly record the character matching between the target feature set and the reference feature set, providing basic data for subsequent similarity calculations.

[0065] In an optional embodiment, after obtaining the length of each feature value in the target feature value set and the length of each feature value in the h-th reference feature value set, the method further includes: the data processing system constructing a first array based on the length of each feature value in the target feature value set, wherein the i-th value in the first array represents the length of the feature value generated based on the i-th character of the target column. Then, a second array is constructed based on the length of each feature value in the h-th reference feature value set, wherein the i-th value in the second array represents the length of the feature value generated based on the i-th character of the h-th semantic type data, and the h-th semantic type data is the semantic type data corresponding to the h-th reference feature value set.

[0066] Optionally, the data processing system records the length information of each feature value in the target feature value set and the reference feature value set by constructing a first array and a second array. First, the data processing system can construct an array, called the first array, based on the length of each feature value in the target feature value set. Specifically, the i-th value in the first array represents the length of the feature value generated by the i-th character in the target column. For example, if the target feature value set is a set of strings generated by some feature extraction operation from each character in the target column, then each value in the first array is the length of that string. The first array provides reference information on the length of the target column feature values ​​for subsequent similarity calculations.

[0067] Furthermore, the data processing system constructs another array, called the second array, based on the length of each feature value in the h-th reference feature value set. The i-th value in the second array represents the length of the feature value generated by the i-th character in the h-th semantic type data. The h-th semantic type data is a set of data of known semantic types corresponding to the h-th reference feature value set. By constructing the second array, the data processing system can record the length information of each feature value in the reference feature value set, facilitating subsequent similarity comparisons with the target feature value set.

[0068] The construction of the first and second arrays enables the data processing system to quantitatively compare the target column and the reference semantic type from the perspective of feature value length, thereby more accurately assessing their similarity.

[0069] For example, a binary value array D[i] can be defined. If the i-th character in the target column's feature value matches any character in the target semantic type's feature value, then D[i] is 1; otherwise, it is 0. A first array N[i] can be defined, representing the length of the feature value generated from the i-th character position from the left in the target column. For feature values ​​generated using the first feature extraction operation, N[i] is not an array but a single value representing the length of a single target column feature value. A second array R[i] can be defined, representing the length of the feature value generated from the i-th character position from the left in the target semantic type. For feature values ​​generated using the first feature extraction operation, R[i] is not an array but a single value representing the length of a single target semantic type feature value.

[0070] The formula for calculating accuracy P can be found in Formula 1:

[0071] (1)

[0072] The formula for calculating recall rate E can be found in Formula 2:

[0073] (2)

[0074] In one optional embodiment, determining the similarity between the target feature set and the h-th reference feature set based on the binary array, the length of each feature value in the target feature set, and the length of each feature value in the h-th reference feature set includes: the data processing system can determine the similarity between the target feature set and the h-th reference feature set based on the ratio of the binary array to the first array and the ratio of the binary array to the second array.

[0075] Optionally, the binary array can record whether each character in the target feature value set matches a character in the h-th reference feature value set. The first array records the length of each feature value in the target feature value set, while the second array records the length of each feature value in the h-th reference feature value set. The data processing system can obtain the precision P, i.e., the proportion of matching characters in the target feature value set, by calculating the ratio of the number of 1s in the binary array to the length of the corresponding feature value in the first array. Simultaneously, the recall E, i.e., the proportion of matching characters in the reference feature value set, can be obtained by calculating the ratio of the number of 1s in the binary array to the length of the corresponding feature value in the second array.

[0076] Based on precision and recall, the data processing system can further calculate the similarity between the target feature set and the h-th reference feature set. (Similarity) The calculation formula can be found in Formula 3:

[0077] (3)

[0078] Where P can be precision, which is the ratio of the number of characters in the target feature set that match the reference feature set to the total number of characters in the target feature set; E can be recall, which is the ratio of the number of characters in the target feature set that match the reference feature set to the total number of characters in the reference feature set; b can be an adjustment parameter used to balance the relative importance of precision and recall. When b increases, the influence of recall E increases; when b decreases, the influence of precision P increases. Formula 3 can combine precision and recall, providing a single numerical value to measure the similarity between the target feature set and the reference feature set. When b=0, F[0]=P. When b=1, Formula 3 simplifies to the standard F1 score, which is the harmonic mean of precision and recall. By adjusting the value of b, the similarity calculation can be optimized according to the needs of specific application scenarios. Formula 3 can be used to determine the similarity between the target feature set and the h-th reference feature set, thereby helping to identify the semantic type of tabular data.

[0079] The embodiments of this application can generate a feature value set only once for a single target column, requiring only one scan, which minimizes resource usage and reduces costs significantly. It can also use a larger dataset for semantic type identification, resulting in higher accuracy.

[0080] See Figure 2 According to another aspect of the embodiments of this application, a semantic type determination device for tabular data is also provided, including: a first processing unit 21, a second processing unit 22 and a third processing unit 23.

[0081] The first processing unit 21 is used to process N predefined semantic type data to generate a reference feature value set corresponding to each semantic type data, where N is an integer greater than 1; the second processing unit 22 is used to sequentially use X feature extraction operations to extract features from the data in the target column of the table to obtain the target feature value set corresponding to the target column, and to detect the similarity between the target feature value set obtained by each feature extraction operation and each reference feature value set, where X is an integer greater than 1; the third processing unit 23 is used to determine the semantic type of the target column and / or the semantic type of each cell in the target column based on the similarity between the target feature value set obtained by each feature extraction operation and each reference feature value set.

[0082] Optionally, the second processing unit 22 includes: a first processing subunit, configured to terminate the feature extraction operation when the similarity between the target feature value set obtained by the first feature extraction operation and at least one reference feature value set is greater than or equal to a preset threshold; and a second processing subunit, configured to sequentially attempt to extract the target feature value set corresponding to the target column using the next feature extraction operation when the similarity between the target feature value set obtained by the first feature extraction operation and each reference feature value set is less than a preset threshold, until the similarity between the target feature value set extracted using a feature extraction operation and at least one reference feature value set is greater than or equal to a preset threshold, and then terminate the feature extraction operation.

[0083] Optionally, the semantic type determination device for tabular data further includes: a fourth processing unit, configured to determine the semantic type of the target column and / or the semantic type of each cell in the target column as an unknown semantic type if, after performing X feature extraction operations, no similarity is detected between the target feature value set extracted from the target column by any feature extraction operation and at least one reference feature value set, and the similarity is greater than or equal to a preset threshold.

[0084] Optionally, the third processing unit 23 includes: a third processing subunit, configured to determine the reference feature set with the highest similarity to the target feature set as the set with the highest reference value if the similarity between the target feature value set obtained from the j-th feature extraction operation and at least one reference feature value set is greater than or equal to a preset threshold, where j is a positive integer less than or equal to X; a fourth processing subunit, configured to use the semantic type data corresponding to the set with the highest reference value as the target data; and a fifth processing subunit, configured to determine the semantic type of the target column and / or the semantic type of each cell in the target column based on the semantic type of the target data.

[0085] Optionally, the second processing unit 22 includes: a first feature extraction subunit, used to sort and remove duplicate strings from all data in the target column, and generate a set of target feature values ​​corresponding to the target column based on the strings obtained after sorting and removing duplicate strings; and a second feature extraction subunit, used to extract characters from preset positions of each data in the target column, sort and remove duplicate characters from all extracted characters, and generate a set of target feature values ​​corresponding to the target column based on the characters in the sorted and extracted subunits.

[0086] Optionally, the second processing unit 22 further includes: an array construction subunit, used to construct a binary array based on the target feature value set and the h-th reference feature value set, wherein each value in the binary array is used to characterize whether each character corresponding to the target feature value set is the same as the character corresponding to the h-th reference feature value set, and h is a positive integer less than or equal to N; a length acquisition subunit, used to acquire the length of each feature value in the target feature value set and the length of each feature value in the h-th reference feature value set; and a similarity determination subunit, used to determine the similarity between the target feature value set and the h-th reference feature value set based on the binary array, the length of each feature value in the target feature value set, and the length of each feature value in the h-th reference feature value set.

[0087] Optionally, the array construction sub-unit includes: a first setting module, used to set the i-th value in the binary array to 1 if the i-th character in the target feature value set is the same as the character corresponding to any feature value in the h-th reference feature value set, where i is an integer greater than or equal to 1; and a second setting module, used to set the i-th value in the binary array to 0 if the i-th character in the target feature value set is different from all the characters corresponding to the h-th reference feature value set.

[0088] Optionally, the semantic type determination device for tabular data further includes: a first construction unit, configured to construct a first array based on the length of each feature value in the target feature value set, wherein the i-th value in the first array represents the length of the feature value generated based on the i-th character of the target column; and a second construction unit, configured to construct a second array based on the length of each feature value in the h-th reference feature value set, wherein the i-th value in the second array represents the length of the feature value generated based on the i-th character in the h-th semantic type data, and the h-th semantic type data is the semantic type data corresponding to the h-th reference feature value set.

[0089] Optionally, the similarity determination subunit includes: a similarity determination module, used to determine the similarity between the target feature value set and the h-th reference feature value set based on the ratio of the binary array to the first array and the ratio of the binary array to the second array.

[0090] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located performs the above-described method for determining the semantic type of tabular data.

[0091] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the above-described semantic type determination method for tabular data.

[0092] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implement the above-described method for determining the semantic type of tabular data.

[0093] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0094] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0096] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0097] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0098] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0099] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for determining the semantic type of tabular data, characterized in that, include: Process N predefined semantic data types to generate a set of reference feature values ​​for each semantic data type, where N is an integer greater than 1; X feature extraction operations are used sequentially to extract features from the data in the target column of the table to obtain the target feature value set corresponding to the target column, and the similarity between the target feature value set obtained by each feature extraction operation and each reference feature value set is detected, where X is an integer greater than 1; Based on the similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set, the semantic type of the target column and / or the semantic type of each cell in the target column are determined.

2. The method of claim 1, wherein, X feature extraction operations are sequentially applied to extract features from the data in the target column of the table, resulting in a set of target feature values ​​corresponding to the target column. The similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set is then detected, including: If the similarity between the target feature value set obtained by the first feature extraction operation and at least one reference feature value set is greater than or equal to a preset threshold, the feature extraction operation is terminated. If the similarity between the target feature value set obtained by the first feature extraction operation and each reference feature value set is less than a preset threshold, then the next feature extraction operation is used sequentially to extract the target feature value set corresponding to the target column until the similarity between the target feature value set extracted by the first feature extraction operation and at least one reference feature value set is greater than or equal to a preset threshold, then the feature extraction operation ends.

3. The method of claim 1, wherein, The method further includes: If, after performing the X feature extraction operations, no similarity is detected between the target feature value set extracted from the target column by any feature extraction operation and at least one reference feature value set that is greater than or equal to a preset threshold, then the semantic type of the target column and / or the semantic type of each cell in the target column is determined to be an unknown semantic type.

4. The method according to claim 2, characterized in that, Based on the similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set, the semantic type of the target column and / or the semantic type of each cell in the target column is determined, including: If the similarity between the target feature value set obtained from the j-th feature extraction operation and at least one reference feature value set is greater than or equal to a preset threshold, then the reference feature value set with the highest similarity to the target feature value set is determined as the set with the highest reference value, where j is a positive integer less than or equal to X; The semantic type data corresponding to the set with the highest reference value is taken as the target data; The semantic type of the target column and / or the semantic type of each cell in the target column are determined based on the semantic type of the target data.

5. The method according to claim 2, characterized in that, The X types of feature extraction operations include at least the following feature extraction operations: The first feature extraction operation is used to sort and remove duplicate strings from all data in the target column, and generate a set of target feature values ​​corresponding to the target column based on the strings obtained after sorting and removing duplicate strings. The second feature extraction operation is used to extract characters from preset positions of each data in the target column, sort all extracted characters and remove duplicates, and generate a set of target feature values ​​corresponding to the target column based on the sorted and segmented characters.

6. The method of claim 1, wherein, Detect the similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set, including: Based on the target feature value set and the h-th reference feature value set, a binary array is constructed, wherein each value in the binary array is used to characterize whether each character corresponding to the target feature value set is the same as the character corresponding to the h-th reference feature value set, and h is a positive integer less than or equal to N; Obtain the length of each feature value in the target feature value set and the length of each feature value in the h-th reference feature value set; The similarity between the target feature set and the h-th reference feature set is determined based on the binary array, the length of each feature value in the target feature set, and the length of each feature value in the h-th reference feature set.

7. The method of claim 6, wherein, Based on the target feature value set and the h-th reference feature value set, a binary array is constructed, including: If the i-th character in the target feature value set is found to be the same as the character corresponding to any feature value in the h-th reference feature value set, then the i-th value in the binary array is set to 1, where i is an integer greater than or equal to 1; If the i-th character in the target feature value set is found to be different from all characters in the h-th reference feature value set, then the i-th value in the binary array is set to 0.

8. The method of claim 6, wherein, After obtaining the length of each feature value in the target feature value set and the length of each feature value in the h-th reference feature value set, the method further includes: A first array is constructed based on the length of each feature value in the target feature value set, wherein the i-th value in the first array represents the length of the feature value generated based on the i-th character of the target column; A second array is constructed based on the length of each feature value in the h-th reference feature value set, wherein the i-th value of the second array represents the length of the feature value generated based on the i-th character in the h-th semantic type data, and the h-th semantic type data is the semantic type data corresponding to the h-th reference feature value set.

9. The method of claim 8, wherein, Based on the binary array, the length of each feature value in the target feature value set, and the length of each feature value in the h-th reference feature value set, the similarity between the target feature value set and the h-th reference feature value set is determined, including: The similarity between the target feature set and the h-th reference feature set is determined based on the ratio of the binary array to the first array and the ratio of the binary array to the second array.

10. A device for determining the semantic type of tabular data, characterized in that, include: The first processing unit is used to process N predefined semantic type data and generate a set of reference feature values ​​corresponding to each semantic type data, where N is an integer greater than 1; The second processing unit is used to sequentially use X feature extraction operations to extract features from the data in the target column of the table, obtain the target feature value set corresponding to the target column, and detect the similarity between the target feature value set obtained by each feature extraction operation and each reference feature value set, where X is an integer greater than 1; The third processing unit is used to determine the semantic type of the target column and / or the semantic type of each cell in the target column based on the similarity between the target feature value set obtained from each feature extraction operation and each reference feature value set.

11. A computer readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device in which the computer-readable storage medium is located performs the semantic type determination method for tabular data as described in any one of claims 1 to 9.

12. An electronic device, comprising: It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the semantic type determination method for tabular data as described in any one of claims 1 to 9.

13. A computer program product, characterised in that, It includes a computer program or instructions that, when executed by a processor, implement the semantic type determination method for tabular data according to any one of claims 1 to 9.

Images