A method and system for constructing query of enterprise-level data sets

Through data analysis model and fuzzy logic reasoning, the integration and maintenance plan of data sets is determined, which solves the problem of high maintenance costs in the enterprise-level data set construction query system, and achieves fast and selective data maintenance and user experience guarantee.

CN118606362BActive Publication Date: 2024-11-29BEIJING FIBERHOME TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410827344.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-25
Publication Date
2024-11-29
Estimated Expiration
2044-06-25

AI Technical Summary

Technical Problem

When the database structure and data content increase, the enterprise-level data set construction query system needs to be updated frequently, resulting in increased maintenance costs and time, and the existing technology is difficult to effectively reduce maintenance burden.

Method used

By collecting field feature information and table feature information, establishing a data analysis model, generating integrated evaluation coefficients, and comparing them with the integration threshold, determining the integrated table, combining user index frequency and table update frequency, using fuzzy logic to determine the maintenance plan, and perform fuzzy Reasoning to determine the category table maintenance results.

Benefits of technology

It avoids the high cost of real-time data maintenance, reduces maintenance time, improves maintenance selectivity, and ensures rapid maintenance and user experience of data collections.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118606362B_ABST
    Figure CN118606362B_ABST
Patent Text Reader

Abstract

The present invention discloses an enterprise-level data set construction query method and system, which relates to the field of data integration and is used to solve the problem of huge maintenance costs. The method collects field feature information and table feature information, establishes a data analysis model, generates an integration evaluation coefficient, obtains the integration evaluation coefficient, and compares and analyzes it with an integration threshold to obtain a comparison result. According to the comparison result, an integratable comparison table is determined, and the integratable comparison table is counted. All tables of the data set are integrated according to the numerical values, and each category table is obtained to obtain index feature information and data frequency information. Through data processing, user index frequency and table update frequency are obtained, and fuzzy logic is used to determine the maintenance results of each category table according to the user index frequency and the table update frequency. Fuzzy reasoning is performed according to fuzzy rules to determine the category table maintenance plan, and targeted data in the data set is regularly maintained by category, so as to quickly perform maintenance and ensure user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data integration, and more specifically, to an enterprise-level data set construction query method and system. Background Art

[0002] The enterprise-level data set construction query method and system is a Web-based application with a user-friendly interface that allows users to quickly query and browse metadata information in the data set by keyword search, data type filtering, etc. The system can also provide metadata management functions, allowing administrators to manage and maintain metadata information in the data set. At the same time, the system should have permission management functions to ensure that only authorized users can access and modify information in the data set.

[0003] The prior art has the following deficiencies:

[0004] At present, the enterprise-level data collection construction query system is designed to help enterprises better complete the data integration in the database. By establishing and maintaining data collections, enterprises can better understand and manage their data assets, thereby achieving the goal of data integration;

[0005] Data sets need to be updated and maintained in a timely manner. As the database structure and data content increase, the data sets need to be continuously updated, which will increase the maintenance cost and workload of the enterprise, consume energy and increase the maintenance time. Therefore, an enterprise-level data set construction query method and system are proposed.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0007] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides an enterprise-level data set construction query method and system, which solves the problems raised in the above-mentioned background technology by using different product inspection methods.

[0008] To achieve the above object, the present invention provides the following technical solution: an enterprise-level data set construction query method and system, comprising S1: collecting field feature information and table feature information, and performing data preprocessing on them, including data type conversion, missing value processing, data standardization, feature extraction, etc., to obtain the sum of field length deviations, the sum of field position distance differences, table foreign key constraint deviations and the number of table shared fields;

[0009] S2: Obtain field feature information and table feature information to establish a data analysis model and generate an integrated evaluation coefficient;

[0010] S3: Obtain the integration evaluation coefficient, and compare and analyze it with the integration threshold to obtain the comparison result. According to the comparison result, determine the comparison table that can be integrated, count the comparison tables that can be integrated, and integrate all the tables in the data set according to the value size;

[0011] S4: Obtain each category table, obtain index feature information and data frequency information, and obtain user index frequency and table update frequency through data processing;

[0012] S5: using fuzzy logic to determine the maintenance results of each category table according to the user index frequency and table update frequency;

[0013] S6: Perform fuzzy reasoning based on fuzzy rules and determine the category table maintenance plan.

[0014] In a preferred embodiment, the field characteristic information includes the sum of field length deviations and the sum of field position distance differences;

[0015] Subtract the lengths of the two fields to get the absolute value of the length difference, and then sum them up to get the total field length deviation , where i is the number of fields in the table, and i=1, 2, 3, 4, ... N, and N is the total number of fields in the table;

[0016] According to the index position of the field in the table, count the index from 0, subtract the position index of the two fields, take the absolute value to get the distance difference, and then sum it up to get the total distance difference of the field position. .

[0017] In a preferred embodiment, the table characteristic information includes table foreign key constraint deviation and the number of table shared fields;

[0018] Get the foreign key constraint information of the two tables, compare the foreign key constraints of the two tables, check whether there are mismatched foreign key constraints, and get the table foreign key constraint deviation , where g is the number of comparison tables, and g=1, 2, 3, 4, ... M, and M is the total number of comparison tables;

[0019] Through all the field names of the two tables, including all the field names in the table and all the field names in the comparison table, and then count the number of fields that exist in the two tables, the number of shared fields in the table is obtained through set calculation. .

[0020] In a preferred embodiment, the sum of field length deviations is obtained: , the sum of field location distance differences , table foreign key constraint deviation And the number of shared fields in the table , establish data analysis model, generate integrated evaluation coefficient , based on the formula:

[0021]

[0022] In the formula, is the similarity evaluation coefficient, , , as well as The sum of field length deviations , the sum of field location distance differences , table foreign key constraint deviation And the number of shared fields in the table The preset scaling factor of , , as well as Both are greater than 0.

[0023] In a preferred embodiment, the integrated evaluation coefficient is obtained Finally, the integrated evaluation coefficient is compared and analyzed with the continuously iterated integrated threshold;

[0024] If the integration evaluation coefficient is greater than or equal to the integration threshold, the comparison table is marked as an integration-capable comparison table, and an integration signal is generated;

[0025] If the integration evaluation coefficient is less than the similarity threshold, the comparison table is marked as an unintegrable comparison table and an end signal is generated;

[0026] When multiple calculations are performed, integrated evaluation coefficient results of multiple tables and comparison tables are obtained, and the tables and comparison tables are integrated according to the results of the integrable comparison tables.

[0027] In a preferred embodiment, the index feature information includes the user index frequency. The user index frequency is obtained by counting the number of index operations performed by users on the table within a fixed time and then calculating the ratio. , where j is the number of table categories, j=1, 2, 3, 4, ... Z, and Z is the total number of table categories;

[0028] Data frequency information includes table update frequency, which is obtained by counting the number of updates of data or fields in the table within a fixed time period and then calculating the ratio. .

[0029] In a preferred embodiment, user index frequency and table update frequency are defined as input variables, and they are divided into different fuzzy sets respectively;

[0030] The maintenance results of each category table are defined as output variables and divided into fuzzy sets;

[0031] Formulate fuzzy rules to describe the impact of user index frequency and table update frequency on the maintenance results of each category table;

[0032] Perform fuzzy reasoning based on fuzzy rules and determine the category table maintenance plan.

[0033] An enterprise-level data set construction query system includes a collection module, a processing module, an integration module, a fuzzy maintenance module, and an output window:

[0034] The collection module is used to collect field characteristic information and table characteristic information, and send the obtained field length deviation sum, field position distance difference sum, table foreign key constraint deviation and table shared field quantity to the processing module;

[0035] The processing module is used to receive the field characteristic information and the table characteristic information to establish a data analysis model, generate an integrated evaluation coefficient, and send it to the integration module;

[0036] The integration module is used to receive the integration evaluation coefficient and compare and analyze it with the integration threshold to obtain the comparison result. According to the comparison result, the integration comparison table is determined, the integration comparison table is counted, and all tables of the data set are integrated according to the value size, and the integrated category tables are sent to the fuzzy maintenance module;

[0037] The fuzzy maintenance module is used to receive each category table, obtain index feature information and data frequency information, and use fuzzy logic to determine the results of each category table and send them to the output window;

[0038] The output window is used to receive the results of each category table and report them.

[0039] Technical effects and advantages of the present invention:

[0040] 1. The present invention establishes a data analysis model through field feature information and table feature information to obtain an integration evaluation coefficient, compares it with the integration threshold to determine the integrated comparison table, and counts it, and integrates all tables of the data set according to the value size, avoiding the problem of real-time data maintenance and high maintenance cost, and reducing maintenance time.

[0041] 2. The present invention formulates a set of fuzzy rules for fuzzy reasoning through user index frequency and table update frequency to determine the category table maintenance plan. While ensuring the reduction of maintenance costs, it increases the selectivity of maintenance. It can perform targeted and regular maintenance on the data in the data set, thereby quickly performing maintenance and ensuring user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The present invention is a method flow chart of a method for constructing a query method for an enterprise-level data set.

[0043] Figure 2 A schematic diagram of a module for constructing a query system for an enterprise-level data set according to the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Example 1

[0046] See also Figure 1 , a method for building queries for enterprise-level data sets. Data sets are usually expressed as containing multiple data tables. For data maintenance, automated tools are used to monitor changes in data sets and automatically update them to reduce manual intervention and workload, or to monitor changes in data set structures, including new tables, new fields, field type changes, etc., and update data sets in a timely manner to reflect these changes. This is real-time data maintenance, which relatively reduces energy consumption, but there is still a problem of high maintenance costs. After integrating data into categories, maintaining a single category based on category scores is a targeted maintenance operation method. The specific operation process is as follows:

[0047] S1: Collect field feature information and table feature information, and perform data preprocessing on them, including data type conversion, missing value processing, data standardization, feature extraction, etc., to obtain the sum of field length deviations, the sum of field position distance differences, table foreign key constraint deviations, and the number of table shared fields;

[0048] The field characteristic information includes the sum of field length deviations and the sum of field position distance differences, and the table characteristic information includes the table foreign key constraint deviation and the number of table shared fields;

[0049] The logic for obtaining the sum of field length deviations is to subtract the lengths of the two fields, obtain the absolute value of the length difference, and then perform sum calculation. It can be understood that the two fields belong to different data fields. Then the length difference is normalized, such as dividing it by the maximum length of the field to obtain the relative value of the length deviation. Assuming that the length of field 1 is , the length of field 2 is , through the formula:

[0050]

[0051] Get field length deviation , and then sum them up , get the sum of field length deviations ;

[0052] It should be noted that when the sum of field length deviations is larger, it means that the total fields in the two tables are less similar, and the two tables are more difficult to judge and integrate, where i is the number of fields in the table, and i=1, 2, 3, 4, ... N, N is the total number of fields in the table;

[0053] The logic for obtaining the sum of the field position distance difference is the index position of the field in the table. The index is counted from 0, the position index of the two fields is subtracted, the absolute value is taken to get the distance difference, and then the sum is calculated. Assume that the position index of field 1 is , the position index of field 2 is , through the formula:

[0054]

[0055] Get the field position distance difference , and then sum them up , get the total field length deviation ;

[0056] It should be noted that when the sum of the field position distance differences is larger, it means that all the fields in the two tables are less similar, and it is more difficult to determine the integration of the two tables;

[0057] It can be understood that the table contains multiple fields. When comparing fields, the fields in one table are always compared and analyzed with the fields in the comparison table to obtain the field length deviation and field position distance difference, thereby obtaining the sum of the field length deviation and the sum of the field position distance difference;

[0058] The logic of obtaining table foreign key constraint deviation is to obtain the foreign key constraint information of the two tables, including constraint name, associated table, associated field, etc., compare the foreign key constraints of the two tables, and check whether there are mismatched foreign key constraints. Assume that the foreign key constraint list of Table 1 is , the foreign key constraint list of table 2 is , through the formula:

[0059]

[0060] Get table foreign key constraint deviation , where g is the number of comparison tables, and g=1, 2, 3, 4, ... M, and M is the total number of comparison tables;

[0061] Table foreign key constraint deviation refers to the difference between the foreign key constraints of two tables. Foreign key constraint is a constraint in the database, which is used to maintain the association between tables. In the database, the foreign key of one table points to the primary key of another table to ensure the integrity and consistency of the data. Table foreign key constraint deviation can be determined by comparing the foreign key constraints of the two tables. Generally, the difference in foreign key constraints includes the following aspects: the difference in the number of foreign key constraints, the number of foreign key constraints between the two tables is different, and the difference in the attributes of foreign key constraints: the attributes of foreign key constraints do not match, including constraint names, associated tables, associated fields, etc.

[0062] It should be noted that when the deviation of the table foreign key constraint is larger, it means that the difference in the association relationship between the two tables is larger, and the two metadata are more difficult to judge and integrate;

[0063] The logic for obtaining the number of shared fields in the tables is to obtain all the field names of the two tables respectively, including all the field names in the table and all the field names in the comparison table, and then count the number of fields that exist in the two tables to obtain the field set of table 1 and the field set of table 2. , get the number of shared fields in the table ;

[0064] It should be noted that when the number of shared fields in the tables is greater, it means that the field similarity between the two tables is higher, and the possibility of table integration is greater;

[0065] S2: Obtain field feature information and table feature information to establish a data analysis model and generate an integrated evaluation coefficient;

[0066] Get the total field length deviation , the sum of field location distance differences , table foreign key constraint deviation And the number of shared fields in the table , establish data analysis model, generate integrated evaluation coefficient , based on the formula:

[0067]

[0068] In the formula, is the similarity evaluation coefficient, , , as well as The sum of field length deviations , the sum of field location distance differences , table foreign key constraint deviation And the number of shared fields in the table The preset scaling factor of , , as well as All are greater than 0;

[0069] Among them, the sum of field length deviations , the sum of field location distance differences , table foreign key constraint deviation And the number of shared fields in the table , are all data-based manifestations of whether the direct expression table and the comparison table can be integrated;

[0070] From the formula, we can see that the lower the sum of field length deviation, field position distance difference and table foreign key constraint deviation, the more likely the table and the comparison table are to be integrated, and the higher the integration evaluation coefficient. On the contrary, if the number of shared fields in the table is smaller, the less likely the table and the comparison table are to be integrated, and the lower the integration evaluation coefficient.

[0071] S3: Obtain the integration evaluation coefficient, and compare and analyze it with the integration threshold to obtain the comparison result. According to the comparison result, determine the comparison table that can be integrated, count the comparison tables that can be integrated, and integrate all the tables in the data set according to the value size;

[0072] The logic of obtaining the integration threshold is to collect a distribution sample set of the historical data set database, divide the data set into a training set and a test set, set the evaluation index and clustering algorithm, and in each round of cross-validation, train the model on the training set and evaluate the model performance on the test set. Then, adjust the integration threshold according to the performance of the validation set. Therefore, the integration threshold is constantly updated.

[0073] In the present invention, clustering algorithm is a type of unsupervised learning algorithm used to divide tables in a data set into groups or clusters with similarities; a common one is K-means clustering, which divides the tables in a data set into K clusters, so that the distance between each table and the center point (centroid) of its cluster is minimized, and finally the integration between the tables is measured by the Euclidean distance, so as to set the integration threshold;

[0074] Get the integrated evaluation coefficient Finally, the integrated evaluation coefficient is compared and analyzed with the continuously iterated integrated threshold;

[0075] If the integration evaluation coefficient is greater than or equal to the integration threshold, the comparison table is marked as an integration-capable comparison table, and an integration signal is generated;

[0076] If the integration evaluation coefficient is less than the similarity threshold, the comparison table is marked as an unintegrable comparison table and an end signal is generated;

[0077] When multiple calculations are performed, the integrated evaluation coefficient results of multiple tables and comparison tables are obtained. The tables and comparison tables are integrated according to the results of the integrated comparison tables. Considering the actual situation, when is greater than or equal to the integration threshold, and is also greater than or equal to the integration threshold, then and Compare the values, if Greater than , then the table is integrated into comparison table 1, otherwise, the table is integrated into comparison table 2, and so on, all data sets are integrated into the table;

[0078] Understandably, The integrated evaluation coefficients of the representative table and the comparative table 1, The integrated evaluation coefficient of the representative table and the comparison table 2, then the sum of the tables included in the data set is M+1;

[0079] For example, through calculation, there are three comparison tables, namely B, C, and D. Compare and analyze B, C, D and table A. Figure 1 As shown:

[0080]

[0081] Figure 1

[0082] As shown in the figure, all the integration evaluation coefficients in the above figure are greater than or equal to the integration threshold. Then, we compare their values ​​and find that the integration evaluation coefficient of A and C is the highest. Then, we integrate table A with table C. We find that the integration evaluation coefficient of B and D is the highest. Then, we integrate table B with table D. All tables in the data set are integrated to complete the quick maintenance.

[0083] The present invention establishes a data analysis model through field feature information and table feature information to obtain an integration evaluation coefficient, compares it with an integration threshold to determine an integrable comparison table, and counts it, and integrates all tables of the data set according to the size of the value, thereby avoiding the problem of real-time data maintenance and high maintenance cost, and reducing maintenance time.

[0084] Example 2

[0085] In the first embodiment of the present invention, an example is given to illustrate the operation strategy of integrating the tables in the data set after obtaining the field characteristic information and the table characteristic information, and the table that can be integrated that is greater than or equal to the integration threshold is calculated in detail, and the operation method of integrating the tables is performed according to the numerical value of the integration evaluation coefficient; however, in the first embodiment, only the tables are integrated, and how to maintain all the tables in a single category has not been deeply explored. Obviously, although this approach can reduce the maintenance time, it is difficult to quickly perform maintenance and ensure user use; in view of the above problems, the second embodiment of the present invention is further refined;

[0086] After integrating the categories of all tables in the data set, the category tables covering the categories are obtained, and then the category scores are scored to perform targeted maintenance on the single category;

[0087] S4: Obtain each category table, obtain index feature information and data frequency information, and obtain user index frequency and table update frequency through data processing;

[0088] The frequency refers to the ratio of the number of user indexes and table updates in a time domain to the time domain. Usually, a time domain is divided into a standard time length, which is a fixed time and is usually set by the user according to business needs. It will not be repeated here.

[0089] The logic for obtaining the user index frequency is to count the number of index operations performed by users on the table within a fixed time, and then calculate the ratio. The user index frequency is obtained by dividing the number of index operations within a fixed time by the fixed time ratio. , where j is the number of table categories, j=1, 2, 3, 4, ... Z, and Z is the total number of table categories;

[0090] It should be noted that when the user index frequency is higher, it means that the fields or data covered in the table of this category have more user index times, and the category table needs to be maintained more;

[0091] The logic for obtaining the table update frequency is to count the number of updates of the data or fields in the table within the same time period as the above fixed time. It can be understood that the operation of adding a new field or table to this category can be understood as the number of table updates, and then the ratio is calculated. The table update frequency is obtained by dividing the number of updates of the data or fields in the table within the fixed time by the fixed time. ;

[0092] It should be noted that when the table is updated more frequently, it means that the fields or data covered in the table of this category are updated faster, and the category table needs to be maintained more frequently.

[0093] The division of the tables in each category is consistent with the above-mentioned embodiment 1, and will not be repeated here;

[0094] S5: using fuzzy logic to determine the maintenance results of each category table according to the user index frequency and table update frequency;

[0095] For example, "High", "Low", "Medium" for user index frequency, "Update Quickly", "UpdateRegularly", "Update Slowly" for table update frequency;

[0096] Formulate a set of fuzzy rules to describe the impact of different input variables on output variables. The definition of rules can be based on professional knowledge or obtained through data analysis and experiments. For example:

[0097] Mark the user index frequency as X, the table update frequency as U, and the category table maintenance result as R_Maintenance;

[0098] Then we can define:

[0099] Rule 1: IF (X is High) AND (U is Update Quickly) THEN (R_Maintenanceis Many)

[0100] Rule 2: IF (U is Low) AND (U is Update Slowly) THEN (R_Maintenance isFew) ...

[0102] S6: Perform fuzzy reasoning based on fuzzy rules to determine the category table maintenance plan;

[0103] It should be noted that the division of fuzzy sets can be adjusted according to actual conditions. For example, although this embodiment takes three fuzzy sets as examples, the user index frequency and table update frequency can actually be divided into more than three sets to facilitate better precise adjustment according to different situations.

[0104] Furthermore, for the judgment of the maintenance rate of user index frequency and table update frequency, a threshold can be set according to the actual situation. For example, when the user index frequency exceeds 70%, it is marked as "High", and when the table update frequency is higher than 75%, it is marked as "Update Quickly", etc., which will not be elaborated here.

[0105] The calculated category table maintenance results are divided into "level 1", "level 2" and "level 3" according to "Many", "Average" and "Few". "Level 1" means that the data in this category table needs to be maintained at present, which has the highest priority. "Level 3" means that this category does not need to be maintained temporarily, etc. The specific maintenance activation time is not limited and is set by the user according to business needs, which will not be elaborated here.

[0106] The present invention formulates a set of fuzzy rules for fuzzy reasoning through user index frequency and table update frequency to determine the category table maintenance plan. While ensuring the reduction of maintenance costs, it increases the selectivity of maintenance. It can perform targeted and regular maintenance on the data in the data set according to categories, thereby quickly performing maintenance and ensuring user experience.

[0107] Example 3

[0108] See also Figure 2 , an enterprise-level data set building query system, including acquisition module, processing module, integration module, fuzzy maintenance module, output window:

[0109] The collection module is used to collect field characteristic information and table characteristic information, and send the obtained field length deviation sum, field position distance difference sum, table foreign key constraint deviation and table shared field quantity to the processing module;

[0110] The processing module is used to receive the field characteristic information and the table characteristic information to establish a data analysis model, generate an integrated evaluation coefficient, and send it to the integration module;

[0111] The integration module is used to receive the integration evaluation coefficient and compare and analyze it with the integration threshold to obtain the comparison result. According to the comparison result, the integration comparison table is determined, the integration comparison table is counted, and all tables of the data set are integrated according to the value size, and the integrated category tables are sent to the fuzzy maintenance module;

[0112] The fuzzy maintenance module is used to receive each category table, obtain index feature information and data frequency information, and use fuzzy logic to determine the results of each category table and send them to the output window;

[0113] The output window is used to receive the results of each category table and report them.

[0114] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0115] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0116] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0117] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0118] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0119] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0122] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0123] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for constructing a query for an enterprise-level data set, characterized in that: S1: Collect field feature information and table feature information, and perform data preprocessing on them, including data type conversion, missing value processing, data standardization, feature extraction, etc., to obtain the sum of field length deviations, the sum of field position distance differences, table foreign key constraint deviations, and the number of table shared fields; S2: Obtain field feature information and table feature information to establish a data analysis model and generate an integrated evaluation coefficient; S3: Obtain the integration evaluation coefficient, and compare and analyze it with the integration threshold to obtain the comparison result. According to the comparison result, determine the comparison table that can be integrated, count the comparison tables that can be integrated, and integrate all the tables in the data set according to the value size; S4: Obtain each category table, obtain index feature information and data frequency information, and obtain user index frequency and table update frequency through data processing; S5: using fuzzy logic to determine the maintenance results of each category table according to the user index frequency and table update frequency; S6: Perform fuzzy reasoning based on fuzzy rules to determine the category table maintenance plan; The field characteristic information includes the sum of field length deviations and the sum of field position distance differences; By subtracting the lengths of the two fields, we get the absolute value of the length difference, and then summing them up to get the total field length deviation Hde. i , where i is the number of fields in the table, and i = 1, 2, 3, 4, ... N, and N is the total number of fields in the table; According to the index position of the field in the table, count the index from 0, subtract the position index of the two fields, take the absolute value to get the distance difference, and then sum it up to get the total field position distance difference Hdi i ; Table characteristic information includes table foreign key constraint deviation and the number of table shared fields; Get the foreign key constraint information of the two tables, compare the foreign key constraints of the two tables, check whether there are mismatched foreign key constraints, and get the deviation of the table foreign key constraints g , where g is the number of comparison tables, and g = 1, 2, 3, 4, ... M, M is the total number of comparison tables; Through all the field names of the two tables, including all the field names in the table and all the field names in the comparison table, and then count the number of fields that exist in the two tables, the number of shared fields in the table is obtained through set calculation Sh g ; Get the total field length deviation Hde i , the sum of field position distance differences Hdi i , table foreign key constraint deviation g And the number of shared fields in the table Sh g , establish data analysis model, generate integrated evaluation coefficient The formula is: In the formula, is the similarity evaluation coefficient, ω1, ω2, ω3 and ω4 are the sum of field length deviations Hde i , the sum of field position distance differences Hdi i , table foreign key constraint deviation g And the number of shared fields in the table Sh g The preset proportional coefficient is , and ω1, ω2, ω3 and ω4 are all greater than 0.

2. The enterprise-level data set construction query method according to claim 1, characterized in that: Get the integrated evaluation coefficient Finally, the integrated evaluation coefficient is compared and analyzed with the continuously iterated integrated threshold; If the integration evaluation coefficient is greater than or equal to the integration threshold, the comparison table is marked as an integration-capable comparison table, and an integration signal is generated; If the integration evaluation coefficient is less than the similarity threshold, the comparison table is marked as an unintegrable comparison table and an end signal is generated; When multiple calculations are performed, integrated evaluation coefficient results of multiple tables and comparison tables are obtained, and the tables and comparison tables are integrated according to the results of the integrable comparison tables.

3. The enterprise-level data set construction query method according to claim 1, characterized in that: Index feature information includes user index frequency. The user index frequency Fr is obtained by counting the number of index operations performed by users on the table within a fixed time and then calculating the ratio. j , where j is the number of table categories, j = 1, 2, 3, 4, ... Z, and Z is the total number of table categories; Data frequency information includes table update frequency. The table update frequency Ta is obtained by counting the number of updates of data or fields in the table within a fixed time period and then calculating the ratio. j .

4. The enterprise-level data set construction query method according to claim 3, characterized in that: User index frequency and table update frequency are defined as input variables, and they are divided into different fuzzy sets respectively; The maintenance results of each category table are defined as output variables and divided into fuzzy sets; Formulate fuzzy rules to describe the impact of user index frequency and table update frequency on the maintenance results of each category table; Perform fuzzy reasoning based on fuzzy rules and determine the category table maintenance plan.

5. An enterprise-level data set construction query system, used to implement an enterprise-level data set construction query method according to any one of claims 1 to 4, characterized in that: Including acquisition module, processing module, integration module, fuzzy maintenance module, output window: The collection module is used to collect field characteristic information and table characteristic information, and send the obtained field length deviation sum, field position distance difference sum, table foreign key constraint deviation and table shared field quantity to the processing module; The processing module is used to receive the field characteristic information and the table characteristic information to establish a data analysis model, generate an integrated evaluation coefficient, and send it to the integration module; The integration module is used to receive the integration evaluation coefficient and compare and analyze it with the integration threshold to obtain the comparison result. According to the comparison result, the integration comparison table is determined, the integration comparison table is counted, and all tables of the data set are integrated according to the value size, and the integrated category tables are sent to the fuzzy maintenance module; The fuzzy maintenance module is used to receive each category table, obtain index feature information and data frequency information, and use fuzzy logic to determine the results of each category table and send them to the output window; The output window is used to receive the results of each category table and report them.

Citation Information

Patent Citations

  • Standard address library collecting and building method and system

    CN117112707A

  • Intelligent classification retrieval method and system for intelligent literature information

    CN118069851A