A method and system for identifying inter-table relationships based on intelligent algorithms

Abstract

The application provides a kind of inter-table relationship identification method and system based on intelligent algorithm, belong to data processing technical field, specifically include: the table excluding invalid table and duplicate table is regarded as correlation evaluation table, and according to the table title of correlation evaluation table, the effective field of correlation evaluation table is obtained by excluding invalid field, and according to effective field and jaccard coefficient, the association coefficient of correlation evaluation table and other correlation evaluation table and the determination of associated table are carried out;According to the association coefficient of correlation evaluation table and other correlation evaluation table and the quantity of associated table, the determination of data reliability requirement of correlation evaluation table is carried out, and according to data reliability requirement, it is determined whether the table quality analysis of the correlation evaluation table is needed, so as to realize the correlation management of table.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a method and system for identifying inter-table relationships based on intelligent algorithms. Background Technology

[0002] By establishing a data platform, data resources can be aggregated and users' basic data resource needs can be met. However, due to the large amount of tabular data, without analysis of inter-table relationships, it is difficult for general business personnel to understand and use the data, resulting in problems such as difficulty in searching, understanding, and applying the data.

[0003] To identify relationships between tables, existing technologies often determine these relationships by matching the similarity of table fields. Specifically, invention patent CN202310867293.2, "A Method for Establishing Relationships Between Tables Through Field Content," utilizes the characteristics of different table field content to determine the relationships between tables based on the similarity and relevance of the field content, thus improving table similarity assessment. However, it suffers from the following technical problems:

[0004] Existing technical solutions neglect the identification of invalid fields in tables. When performing relationship matching, if there are invalid fields in the table title, it is not only impossible to accurately determine the relationship between different tables, but also makes it difficult to guarantee the efficiency of relationship analysis.

[0005] Existing technical solutions neglect the identification of invalid tables when performing relationship analysis. When there are multiple similar tables based on the table title, if the missing content or blank tables cannot be identified, the relationship between different tables cannot be accurately identified.

[0006] To address the aforementioned technical problems, this invention provides a method and system for identifying inter-table relationships based on intelligent algorithms. Summary of the Invention

[0007] To achieve the objectives of this invention, the following technical solution is adopted:

[0008] According to one aspect of the present invention, a method for identifying inter-table relationships based on intelligent algorithms is provided.

[0009] A method for identifying inter-table relationships based on intelligent algorithms, characterized in that it specifically includes:

[0010] S1 determines the category of the table based on the recognition results of the table's title, and determines similar tables based on the table's category and the recognition results of the title keywords of different tables;

[0011] S2 identifies duplicate tables and suspected problematic content in the similar tables based on the table content of the similar tables, and determines the table quality of the similar tables and filters invalid tables based on the identification results of the suspected problematic content, the table update time, and the table content.

[0012] S3 uses the table that excludes invalid and duplicate tables as the correlation evaluation table, and excludes invalid fields according to the table title of the correlation evaluation table to obtain the valid fields of the correlation evaluation table. Based on the valid fields and the Jaccard coefficient, the correlation coefficient between the correlation evaluation table and other correlation evaluation tables is determined, and the correlation tables are identified.

[0013] S4 determines the data reliability requirements of the association evaluation table based on the correlation coefficient between the association evaluation table and other association evaluation tables and the number of association tables, and determines whether table quality analysis of the association evaluation table is required based on the data reliability requirements.

[0014] The beneficial effects of this invention are as follows:

[0015] 1. By identifying suspected problematic content, the table update time, and the table content, the quality of similar tables is determined and invalid tables are filtered out. This not only takes into account the differences in suspected problematic content in different tables, but also the differences in update time and table content size, thus achieving the filtering of invalid tables with low quality from multiple perspectives.

[0016] 2. By using valid fields and Jaccard coefficients to determine the correlation coefficients between the correlation evaluation table and other correlation evaluation tables, and by identifying the correlation tables themselves, the technical problem of inaccurate correlation evaluation caused by using all fields is avoided. This improves the efficiency of correlation evaluation and further enhances the accuracy of the evaluation.

[0017] 3. Based on the correlation coefficient between the correlation evaluation table and other correlation evaluation tables, as well as the number of correlation tables, the data reliability requirements of the correlation evaluation tables are determined. This fully considers the differences in data quality requirements of different correlation evaluation tables due to differences in their correlation with other tables, and realizes the assessment of data quality for tables with more complex correlation data, thus ensuring the reliability of table management.

[0018] A further technical solution is that the categories of the table include user table, metering point, electricity price table, line table and electricity address. Specifically, the title keywords of the table are determined based on the identification results of the table title, and the category of the table is determined through the title keywords.

[0019] A further technical solution involves determining similar tables based on the table category and the identification results of keywords in the titles of different tables, specifically including:

[0020] The comparison table is determined based on the category of the table, and the similar table is determined based on the number and similarity of the title keywords of the table and the comparison table.

[0021] A further technical solution is that the duplicate tables in the similar tables are determined based on the table content in the similar tables, specifically based on the duplicate identification results of the table content.

[0022] A further technical solution is that the suspected problematic content is determined based on the identification result of the table content, specifically based on the identification result that the content data in the table content does not conform to preset rules.

[0023] A further technical solution is that when the quality of the similar table does not meet the requirements, the similar table is determined to be an invalid table.

[0024] A further technical solution is that the association table of the association evaluation table is determined based on the association coefficient between the association evaluation table and the other association evaluation tables. Specifically, when the association coefficient between the other association evaluation tables and the association evaluation table meets the threshold requirement, the other association evaluation tables are determined to be association tables.

[0025] A further technical solution is that, after determining the associated tables of the associated evaluation table, the associated topology graph of the associated evaluation table is generated according to the type of the associated evaluation table and the type of the associated table.

[0026] On the other hand, the present invention provides a computer system comprising: a memory and a processor connected in communication, and a computer program stored in the memory and capable of running on the processor, characterized in that: when the processor runs the computer program, it executes the above-described method for identifying inter-table relationships based on an intelligent algorithm.

[0027] On the other hand, this application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed in a computer, it causes the computer to execute the above-mentioned method for identifying inter-table relationships based on an intelligent algorithm.

[0028] Other features and advantages will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained through the structures particularly pointed out in the description and the drawings.

[0029] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0030] The above and other features and advantages of the present invention will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0031] Figure 1 This is a flowchart of a method for identifying inter-table relationships based on intelligent algorithms;

[0032] Figure 2 This is a flowchart illustrating the method for determining the correlation coefficient between a correlation assessment table and other correlation assessment tables.

[0033] Figure 3 It is a framework diagram of a computer system. Detailed Implementation

[0034] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that the invention will be thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and therefore their detailed description will be omitted.

[0035] The terms “a,” “one,” “the,” and “the” are used to indicate the existence of one or more elements / components / etc.; the terms “including” and “having” are used to indicate an open-ended meaning of inclusion and that other elements / components / etc. may exist in addition to the listed elements / components / etc.

[0036] Example 1

[0037] To solve the above problems, according to one aspect of the present invention, such as Figure 1 As shown, according to one aspect of the present invention, a method for identifying inter-table relationships based on intelligent algorithms is provided, characterized in that it specifically includes:

[0038] S1 determines the category of the table based on the recognition results of the table's title, and determines similar tables based on the table's category and the recognition results of the title keywords of different tables;

[0039] It should be noted that the categories of the tables include user tables, metering points, electricity price tables, line tables, and electricity addresses. Specifically, the table title keywords are determined based on the identification results of the table titles, and the table category is determined through the title keywords.

[0040] Specifically, similar tables are determined based on the table category and the identification results of keywords in the titles of different tables. This includes:

[0041] The comparison table is determined based on the category of the table, and the similar table is determined based on the number and similarity of the title keywords of the table and the comparison table.

[0042] S2 identifies duplicate tables and suspected problematic content in the similar tables based on the table content of the similar tables, and determines the table quality of the similar tables and filters invalid tables based on the identification results of the suspected problematic content, the table update time, and the table content.

[0043] It should be noted that the duplicate tables in the similar tables are determined based on the table content in the similar tables, specifically based on the duplicate identification results of the table content.

[0044] In one possible embodiment, the method for determining the table quality of the similar tables is as follows:

[0045] S21 Based on the identification results of the suspected problem content of the similar table, determine the type of the suspected problem content of the similar table and the amount of data of the different types of suspected problem content, and determine whether the similar table is an invalid table based on the type of the suspected problem content of the similar table and the amount of data of the different types of suspected problem content. If yes, the similar table is determined to be an invalid table; otherwise, proceed to the next step.

[0046] S22 determines the association type of the suspected problem content in the similar table based on the type of the suspected problem content and the title keywords of the similar table, and determines whether the similar table is an invalid table based on the number of association types of the suspected problem content and the amount of data of the suspected problem content of different association types. If yes, the similar table is determined to be an invalid table; otherwise, proceed to the next step.

[0047] S23 determines the content quality assessment value of the similar table by the number of suspected problem content types in the similar table, the amount of data of suspected problem content of different types, the number of association types of suspected problem content, the amount of data of suspected problem content of different association types, and the amount of data of content in the similar table. Based on the content quality assessment value, it is determined whether the similar table is an invalid table. If yes, the similar table is determined to be an invalid table. If no, proceed to the next step.

[0048] S24 determines the table quality of the similar tables based on their table content, update time, and content quality assessment value.

[0049] It should be noted that the suspected problematic content is determined based on the identification results of the table content, specifically based on the identification results of the content data in the table content not conforming to preset rules.

[0050] Furthermore, if the quality of the similar table does not meet the requirements, the similar table is determined to be an invalid table.

[0051] In another possible embodiment, the method for determining the table quality of the similar tables is as follows:

[0052] When the number of similar tables is less than the preset number:

[0053] The quality of the similar tables is determined based on their table content and update time.

[0054] When the number of similar tables is not less than the preset number:

[0055] When it is determined that the similar table contains suspected problematic content based on the identification results of the suspected problematic content in the similar table:

[0056] By determining the type of suspected problematic content in the similar tables and the amount of data for different types of suspected problematic content, and based on the type of suspected problematic content in the similar tables and the amount of data for different types of suspected problematic content, it is determined whether the similar tables are invalid tables. If so, the similar tables are determined to be invalid tables; otherwise, proceed to the next step.

[0057] Based on the type of suspected problem content in the similar tables and the keywords of the titles in the similar tables, the association type of the suspected problem content in the similar tables is determined. Based on the number of association types of the suspected problem content and the amount of data of suspected problem content of different association types, it is determined whether the similar tables are invalid tables. If yes, the similar tables are determined to be invalid tables. If no, proceed to the next step.

[0058] The content quality assessment value of the similar table is determined by the number of suspected problem content types in the similar table, the amount of data of suspected problem content of different types, the number of association types of suspected problem content, the amount of data of suspected problem content of different association types, and the amount of data of content in the similar table. Based on the content quality assessment value, it is determined whether the similar table is an invalid table. If it is, the similar table is determined to be an invalid table. If not, proceed to the next step.

[0059] The table quality of the similar tables is determined based on their table content, update time, and content quality assessment value.

[0060] When it is determined, based on the identification results of suspected problematic content in the similar tables, that the similar tables do not contain suspected problematic content:

[0061] The quality of the similar tables is determined based on their table content and update time.

[0062] S3 uses the table that excludes invalid and duplicate tables as the correlation evaluation table, and excludes invalid fields according to the table title of the correlation evaluation table to obtain the valid fields of the correlation evaluation table. Based on the valid fields and the Jaccard coefficient, the correlation coefficient between the correlation evaluation table and other correlation evaluation tables is determined, and the correlation tables are identified.

[0063] In one possible embodiment, such as Figure 2 As shown, the method for determining the correlation coefficient between the correlation evaluation table and other correlation evaluation tables is as follows:

[0064] S31 determines the field correlation coefficient between the effective fields of the association evaluation table and the effective fields of other association evaluation tables based on the Jaccard coefficients of the effective fields of the association evaluation table and the effective fields of other association evaluation tables. If there are any effective fields in the other association evaluation tables with field correlation coefficients greater than the preset coefficient, proceed to the next step; otherwise, determine that the other association evaluation tables do not belong to the association tables.

[0065] S32 determines whether the other association evaluation table is an association table by the number of valid fields whose field association coefficient is greater than the preset coefficient and the number of valid fields in the association evaluation table. If yes, proceed to step S34; otherwise, proceed to step S33.

[0066] S33 Obtain the number of valid fields in the other association evaluation tables whose field association coefficients are not greater than a preset coefficient, and determine whether the other association evaluation tables belong to the association tables based on the number of valid fields in the association evaluation tables. If yes, proceed to step S34; otherwise, determine that the other association evaluation tables do not belong to the association tables.

[0067] S34 determines the correlation coefficient between the correlation evaluation table and the other correlation evaluation tables by the number of valid fields whose field correlation coefficients are not greater than the preset coefficient, the number of valid fields whose field correlation coefficients are greater than the preset coefficient, the average value of the field correlation coefficients, and the number of valid fields in the correlation evaluation table.

[0068] Furthermore, the association table of the association evaluation table is determined based on the association coefficient between the association evaluation table and the other association evaluation tables. Specifically, when the association coefficient between the other association evaluation tables and the association evaluation table meets the threshold requirement, the other association evaluation tables are determined to be association tables.

[0069] S4 determines the data reliability requirements of the association evaluation table based on the correlation coefficient between the association evaluation table and other association evaluation tables and the number of association tables, and determines whether table quality analysis of the association evaluation table is required based on the data reliability requirements.

[0070] In one possible embodiment, the method for determining the data reliability requirements is as follows:

[0071] The number of association tables that are inconsistent with the type of the association assessment table is determined based on the type of the association assessment table and the type of the association table.

[0072] When the number of association tables that are inconsistent with the type of the association evaluation table and the number of association tables of the association evaluation table are both within the preset number of tables:

[0073] The data reliability requirements of the association assessment table are determined by the number of association tables that are inconsistent with the type of the association assessment table and the number of association tables associated with the association assessment table.

[0074] When either the number of associated tables that are inconsistent with the type of the associated evaluation table or the number of associated tables of the associated evaluation table are outside the preset number of tables:

[0075] The number of associated tables that are related to the associated table is obtained, and the association complexity of the associated table and the number of complex tables are determined by the number of associated tables. The association complexity assessment quantity of the associated evaluation table is determined by the number of complex tables, their proportion in the associated table, and the association complexity of the associated table. The number of other associated evaluation tables with association coefficients greater than a preset association coefficient is determined by the association coefficient between the associated evaluation table and other associated evaluation tables. Furthermore, the data reliability requirements of the associated evaluation table are determined by combining the number of associated tables of the associated evaluation table, the number of associated tables of different types from the associated evaluation table, and the association complexity assessment quantity of the associated evaluation table.

[0076] It should be further explained that once the associated tables of the associated evaluation table are determined, the associated topology graph of the associated evaluation table is generated according to the type of the associated evaluation table and the type of the associated table.

[0077] In another possible embodiment, the method for determining the data reliability requirements is as follows:

[0078] S41 determines whether the number of associated tables in the association evaluation table is within a preset range by measuring the number of associated tables in the association evaluation table. If yes, proceed to step S44; otherwise, proceed to step S42.

[0079] S42 determines the number of association tables that are inconsistent with the type of the association evaluation table based on the type of the association evaluation table and the type of the association table, and determines whether the association evaluation table needs to be subjected to table quality analysis based on the number of association tables that are inconsistent with the type of the association evaluation table. If yes, proceed to step S44; otherwise, proceed to the next step.

[0080] S43 Obtain the number of related tables that are related to the related table, and determine the association complexity of the related table and the number of complex related tables by the number of related tables. Then, determine the association complexity assessment quantity of the related evaluation table by the number of complex related tables, their proportion in the related tables, and the association complexity of the related tables. Based on the association complexity assessment quantity of the related evaluation table, determine whether table quality analysis is required for the related evaluation table. If yes, proceed to step S44; otherwise, determine that table quality analysis is not required for the related evaluation table.

[0081] S44 determines the number of other association evaluation tables whose association coefficient is greater than a preset association coefficient by comparing the association coefficient between the association evaluation table and other association evaluation tables. Furthermore, it determines the data reliability requirements of the association evaluation table by combining the number of association tables of the association evaluation table, the number of association tables whose type is inconsistent with the association evaluation table, and the association complexity evaluation quantity of the association evaluation table.

[0082] Example 2

[0083] like Figure 3 As shown, the present invention provides a computer system, comprising: a memory and a processor connected in communication, and a computer program stored in the memory and capable of running on the processor, characterized in that: when the processor runs the computer program, it executes the above-described method for identifying inter-table relationships based on an intelligent algorithm.

[0084] Example 3

[0085] This application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed in a computer, it causes the computer to execute the above-described method for identifying inter-table relationships based on an intelligent algorithm.

[0086] Through the above embodiments, the present invention achieves the following beneficial effects:

[0087] 1. By identifying suspected problematic content, the table update time, and the table content, the quality of similar tables is determined and invalid tables are filtered out. This not only takes into account the differences in suspected problematic content in different tables, but also the differences in update time and table content size, thus achieving the filtering of invalid tables with low quality from multiple perspectives.

[0088] 2. By using valid fields and Jaccard coefficients to determine the correlation coefficients between the correlation evaluation table and other correlation evaluation tables, and by identifying the correlation tables themselves, the technical problem of inaccurate correlation evaluation caused by using all fields is avoided. This improves the efficiency of correlation evaluation and further enhances the accuracy of the evaluation.

[0089] 3. Based on the correlation coefficient between the correlation evaluation table and other correlation evaluation tables, as well as the number of correlation tables, the data reliability requirements of the correlation evaluation tables are determined. This fully considers the differences in data quality requirements of different correlation evaluation tables due to differences in their correlation with other tables, and realizes the assessment of data quality for tables with more complex correlation data, thus ensuring the reliability of table management.

[0090] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0091] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0092] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for identifying inter-table relationships based on intelligent algorithms, characterized in that, Specifically, it includes: The table category is determined by the recognition results of the table title, and similar tables are determined based on the table category and the recognition results of the title keywords of different tables. Based on the content of the similar tables, duplicate tables and suspected problematic content in the similar tables are identified, and the table quality of the similar tables and invalid tables are determined and screened by the identification results of the suspected problematic content, the table update time and the table content. The table that excludes invalid and duplicate tables is used as the association evaluation table. Invalid fields are excluded according to the table title of the association evaluation table to obtain the valid fields of the association evaluation table. The association coefficient between the association evaluation table and other association evaluation tables is determined based on the valid fields and the Jaccard coefficient. Based on the correlation coefficient between the correlation evaluation table and other correlation evaluation tables, as well as the number of correlation tables, the data reliability requirements of the correlation evaluation table are determined, and based on the data reliability requirements, it is determined whether table quality analysis is required for the correlation evaluation table. The suspected problematic content is determined based on the identification results of the table content, specifically based on the identification results of the content data in the table content not conforming to preset rules.

2. The method for identifying inter-table relationships based on intelligent algorithms as described in claim 1, characterized in that, The categories of the tables include user tables, metering points, electricity price tables, line tables, and electricity addresses. Specifically, the table title keywords are determined based on the identification results of the table titles, and the table category is determined through the title keywords.

3. The method for identifying inter-table relationships based on intelligent algorithms as described in claim 1, characterized in that, Based on the table category and the identification results of keywords in the titles of different tables, similar tables are determined, specifically including: The comparison table is determined based on the category of the table, and the similar table is determined based on the number and similarity of the title keywords of the table and the comparison table.

4. The method for identifying inter-table relationships based on intelligent algorithms as described in claim 1, characterized in that, The duplicate tables in the similar tables are determined based on the table content in the similar tables, specifically based on the duplicate identification results of the table content.

5. The method for identifying inter-table relationships based on intelligent algorithms as described in claim 1, characterized in that, If the quality of the similar table does not meet the requirements, the similar table is determined to be an invalid table.

6. The method for identifying inter-table relationships based on intelligent algorithms as described in claim 1, characterized in that, The method for determining the correlation coefficient between the correlation evaluation table and other correlation evaluation tables is as follows: S31 determines the field correlation coefficient between the effective fields of the association evaluation table and the effective fields of other association evaluation tables based on the Jaccard coefficients of the effective fields of the association evaluation table and the effective fields of other association evaluation tables. If there are any effective fields in the other association evaluation tables with field correlation coefficients greater than the preset coefficient, proceed to the next step; otherwise, determine that the other association evaluation tables do not belong to the association tables. S32 determines whether the other association evaluation table is an association table by the number of valid fields whose field association coefficient is greater than the preset coefficient and the number of valid fields in the association evaluation table. If yes, proceed to step S34; otherwise, proceed to step S33. S33 Obtain the number of valid fields in the other association evaluation tables whose field association coefficients are not greater than a preset coefficient, and determine whether the other association evaluation tables belong to the association tables based on the number of valid fields in the association evaluation tables. If yes, proceed to step S34; otherwise, determine that the other association evaluation tables do not belong to the association tables. S34 determines the correlation coefficient between the correlation evaluation table and the other correlation evaluation tables by the number of valid fields whose field correlation coefficients are not greater than the preset coefficient, the number of valid fields whose field correlation coefficients are greater than the preset coefficient, the average value of the field correlation coefficients, and the number of valid fields in the correlation evaluation table.

7. The method for identifying inter-table relationships based on intelligent algorithms as described in claim 6, characterized in that, The association table of the association evaluation table is determined based on the association coefficient between the association evaluation table and the other association evaluation tables. Specifically, when the association coefficient between the other association evaluation tables and the association evaluation table meets the threshold requirement, the other association evaluation tables are determined to be association tables.

8. A computer system, comprising: A memory and a processor connected in communication, and a computer program stored in the memory and capable of running on the processor, characterized in that: when the processor runs the computer program, it executes a table relationship identification method based on an intelligent algorithm as described in any one of claims 1-7.

Images