Redundant data table monitoring method and device, computer device and storage medium

Abstract

The application relates to an artificial intelligence technology and provides a redundant data table monitoring method and device, computer equipment and a storage medium, which comprises the following steps: performing partition processing on a target vector set, determining a first similarity threshold of a partition vector set, determining a first similarity value of a first partition vector and a second partition vector in the same partition vector set, detecting whether the first similarity value exceeds the first similarity threshold, when the answer is yes, dividing the first partition vector and the second partition vector into an initial same-group vector set, determining a second similarity value between a first initial same-group vector set and a second initial same-group vector set and a second similarity threshold in different partition vector sets, detecting whether the second similarity value exceeds the second similarity threshold, when the answer is yes, dividing the first initial same-group vector set and the second initial same-group vector set into a target same-group vector set, and determining a redundant data table in the target same-group vector set. The application can improve the monitoring accuracy of the redundant data table and promote the rapid development of a smart city.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer equipment, and storage medium for monitoring redundant data tables. Background Technology

[0002] With the popularization of the big data concept and the advancement of related technologies, large enterprises have deployed big data applications to varying degrees to meet diverse business needs. Many companies have implemented overall enterprise data strategies to guide various business departments and lines within the enterprise in data extraction, integration, and management, establishing relevant data usage standards, including unified data dictionaries and data analysis tools. These are beneficial for subsequent data integration and utilization. However, after a large amount of data is imported into the database, due to multiple teams and individuals collaborating on a single data warehouse, many redundant data tables with extremely similar structures are generated. These redundant data tables occupy a significant amount of disk space, hindering data source location during data processing and resulting in substantial resource waste.

[0003] In the process of implementing this application, the applicant discovered the following problems with the existing technology: when monitoring redundant data tables in the database, the redundant data tables are mainly detected by comparing the hash values ​​of the corresponding storage files of the tables. This is limited to the analysis and judgment of redundant data tables from the perspective of files, and cannot guarantee the accuracy of redundant data table monitoring.

[0004] Therefore, it is necessary to provide a method for monitoring redundant data tables that can improve the accuracy of redundant data table monitoring. Summary of the Invention

[0005] In view of the above, it is necessary to propose a method, device, computer equipment and storage medium for monitoring redundant data tables, which can improve the accuracy of redundant data table monitoring.

[0006] A first aspect of this application provides a method for monitoring redundant data tables, the method comprising:

[0007] Obtain a preset set of data tables in the target database, and determine the target information corresponding to each preset data table in the preset set of data tables to obtain a target information set;

[0008] Each of the target information is vectorized to obtain a target vector set;

[0009] The target vector set is partitioned to obtain multiple partition vector sets, and a first similarity threshold is determined for each partition vector set.

[0010] In the same set of partition vectors, a first similarity value is determined between the first partition vector and the second partition vector, and it is detected whether the first similarity value exceeds the first similarity threshold.

[0011] When the first similarity value exceeds the first similarity threshold, the first partition vector and the second partition vector are divided into an initial group vector set in the same partition vector set;

[0012] In different partitioned vector sets, a second similarity value and a second similarity threshold are determined between the first initial same-group vector set and the second initial same-group vector set, and it is detected whether the second similarity value exceeds the second similarity threshold;

[0013] When the second similarity value exceeds the second similarity threshold, the first initial same-group vector set and the second initial same-group vector set are divided into a target same-group vector set;

[0014] Identify the redundant data tables in the target vector set.

[0015] Furthermore, in the redundant data table monitoring method provided in the embodiments of this application, the step of determining the target information corresponding to each preset data table in the preset data table set to obtain the target information set includes:

[0016] Obtain the initial field description information of each preset data table in the preset data table set;

[0017] Detect whether there are preset descriptive keywords in the initial field description information;

[0018] When the preset description keyword exists in the initial field description information, the preset description keyword is replaced with the target description keyword to obtain the target field description information;

[0019] Obtain the field information and name information corresponding to the preset data table, and combine the field information, name information and target field description information according to a preset format to obtain target information;

[0020] The target information is combined to obtain a target information set.

[0021] Furthermore, in the redundant data table monitoring method provided in the embodiments of this application, after vectorizing each of the target information to obtain a target vector set, the method further includes:

[0022] Determine the intra-class scatter matrix corresponding to the target vector;

[0023] Determine the inter-class scatter matrix corresponding to the target vector;

[0024] The target matrix is ​​obtained by multiplying the inverse of the intra-class scatter matrix with the inter-class scatter matrix.

[0025] Determine multiple feature values ​​corresponding to the target matrix, and extract feature values ​​whose numerical ranking is below a preset ranking threshold from the feature values ​​as the target feature value set;

[0026] Based on the eigenvector corresponding to each eigenvalue in the target eigenvalue set, determine the projection matrix corresponding to the target vector;

[0027] Multiply the transpose of the projection matrix by the target vector to obtain the dimensionality-reduced target vector.

[0028] Furthermore, in the redundant data table monitoring method provided in the embodiments of this application, the step of partitioning the target vector set to obtain multiple partition vector sets includes:

[0029] Obtain the field information corresponding to each target vector in the target vector set;

[0030] The number of fields corresponding to each target vector is determined based on the field information;

[0031] The target vectors whose number of fields is within a preset range are divided into the same partition to obtain multiple partition vector sets.

[0032] Furthermore, in the redundant data table monitoring method provided in the embodiments of this application, determining the first similarity threshold for each partition vector set includes:

[0033] Obtain the target partition identifier for each partition vector set;

[0034] Obtain the mapping relationship between pre-set partition identifiers and similarity thresholds;

[0035] Based on the target partition identifier, the mapping relationship is traversed to obtain the first similarity threshold corresponding to each partition vector set.

[0036] Furthermore, in the redundant data table monitoring method provided in the embodiments of this application, determining the first similarity value between the first partition vector and the second partition vector includes:

[0037] Obtain the preset distance calculation model;

[0038] The preset distance calculation model is invoked to process the first partition vector and the second partition vector, and the vector distance between the first partition vector and the second partition vector is obtained as the first similarity value.

[0039] Furthermore, in the redundant data table monitoring method provided in the embodiments of this application, determining the redundant data tables in the target vector set includes:

[0040] Determine the target features of the source data table;

[0041] Determine the preset data table corresponding to each target vector in the target vector set and the data features of the preset data table;

[0042] Detect whether the data features satisfy the target features;

[0043] When the data feature satisfies the target feature, the preset data table corresponding to the data feature is determined as the source data table;

[0044] When the data feature does not meet the target feature, the preset data table corresponding to the data feature is determined to be a redundant data table.

[0045] A second aspect of this application also provides a redundant data table monitoring device, the redundant data table monitoring device comprising:

[0046] The information determination module is used to obtain a preset set of data tables in the target database and determine the target information corresponding to each preset data table in the preset set of data tables to obtain a target information set.

[0047] The vector processing module is used to perform vectorization processing on each of the target information to obtain a target vector set;

[0048] A partitioning module is used to partition the target vector set to obtain multiple partitioned vector sets, and to determine a first similarity threshold for each partitioned vector set.

[0049] The first similarity determination module is used to determine a first similarity value between a first partition vector and a second partition vector in the same set of partition vectors, and to detect whether the first similarity value exceeds the first similarity threshold.

[0050] The first vector grouping module is used to divide the first partition vector and the second partition vector into an initial group vector set in the same partition vector set when the first similarity value exceeds the first similarity threshold.

[0051] The second similarity determination module is used to determine a second similarity value and a second similarity threshold between the first initial same group vector set and the second initial same group vector set in different partition vector sets, and to detect whether the second similarity value exceeds the second similarity threshold.

[0052] The second vector grouping module is used to divide the first initial same-group vector set and the initial second group vector set into a target same-group vector set when the second similarity value exceeds the second similarity threshold.

[0053] The redundancy determination module is used to determine the redundant data tables in the target vector set.

[0054] A third aspect of this application also provides a computer device, the computer device including a processor, the processor being configured to execute a computer program stored in a memory to implement the redundant data table monitoring method as described in any of the preceding claims.

[0055] A fourth aspect of this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the redundant data table monitoring method described in any one of the above embodiments.

[0056] The redundant data table monitoring method, redundant data table monitoring device, computer equipment, and computer-readable storage medium provided in this application embodiment partition the target vector set to obtain multiple partition vector sets and determine a first similarity threshold for each partition vector set. By using partitioning and threshold-based similarity determination, the amount of data to be calculated can be reduced, the calculation speed can be improved, and the problem of different definitions of table similarity due to different numbers of fields can be avoided, thus improving the accuracy of similarity calculation and consequently improving the accuracy of redundant data table identification. Furthermore, this application embodiment also performs similarity calculations in different partition vector sets, which can avoid the problem of identical data tables being classified into different categories and the omission of classification edge data, further improving the accuracy of redundant data table identification. This application can be applied to various functional modules of smart cities such as smart government affairs and smart transportation, such as the redundant data table monitoring module of smart government affairs, and can promote the rapid development of smart cities. Attached Figure Description

[0057] Figure 1 This is a flowchart of the redundant data table monitoring method provided in Embodiment 1 of this application.

[0058] Figure 2 This is a flowchart provided in one embodiment of this application.

[0059] Figure 3 This is a flowchart provided in another embodiment of this application.

[0060] Figure 4 This is a structural diagram of the redundant data table monitoring device provided in Embodiment 2 of this application.

[0061] Figure 5 This is a schematic diagram of the structure of the computer device provided in Embodiment 3 of this application.

[0062] The following detailed description, in conjunction with the accompanying drawings, will further illustrate this application. Detailed Implementation

[0063] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0064] Numerous specific details are set forth in the following description in order to provide a full understanding of this application. The described embodiments are only some, not all, of the embodiments of this application.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0066] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0067] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0068] The redundant data table monitoring method provided in this embodiment of the invention is executed by a computer device, and correspondingly, the redundant data table monitoring device runs in the computer device. Figure 1 This is a flowchart of the redundant data table monitoring method provided in Embodiment 1 of this application. Figure 1 As shown, the redundant data table monitoring method may include the following steps. The order of these steps in the flowchart may be changed, and some may be omitted, depending on different requirements:

[0069] S11, obtain a preset data table set in the target database, and determine the target information corresponding to each preset data table in the preset data table set to obtain a target information set.

[0070] In at least one embodiment of this application, the target database can be used to store preset data tables, which can be divided into source data tables, redundant data tables, and non-redundant data tables. The preset data table set contains multiple preset data tables, and for each preset data table, there exists corresponding target information. The target information is used to uniquely identify the preset data table; by querying the target information, a unique corresponding preset data table can be obtained. Each preset data table corresponds to one piece of target information. The target information may include field information, name information, and field description information. In one embodiment, the target database can be an HDFS database.

[0071] Please see Figure 2 Optionally, determining the target information corresponding to each preset data table in the preset data table set to obtain the target information set includes:

[0072] S110, Obtain the initial field description information of each preset data table in the preset data table set;

[0073] S111, Detect whether there are preset descriptive keywords in the initial field description information;

[0074] S112, when the preset description keyword exists in the initial field description information, the preset description keyword is replaced with the target description keyword to obtain the target field description information;

[0075] S113, obtain the field information and name information corresponding to the preset data table, and combine the field information, the name information and the target field description information according to a preset format to obtain the target information;

[0076] S114, combine the target information to obtain a target information set.

[0077] The initial field description information refers to the information describing all fields in the preset data table. When the initial field description information contains several preset description keywords, by replacing the preset description keywords with target description keywords, similar description keywords can be uniformly named to obtain standardized target field description information. The target description keywords are stored in a preset database. Considering the reliability and privacy of data storage, the preset database can be a target node on the blockchain. The target description keywords are pre-set standardized field description keywords. For the initial field description information of each preset data table, semantic analysis can be performed on the initial field description information to detect whether there are initial description keywords that are semantically similar to the target description keywords. The initial description keywords are then replaced with the target description keywords to obtain the target field description information. In one embodiment, when the preset description keywords are not present in the initial field description information, it means that there is no need to uniformly name the initial field description information. In this case, the field information, the name information, and the initial field description information can be directly combined according to a preset format to obtain the target information. Each of the aforementioned preset data tables contains several fields, and each preset data table has a corresponding table name. The English names of the various fields in the preset data table are combined to obtain field information, and the English table name of the preset data table is used as name information. The preset format is a pre-set format used to sort the field information, the name information, and the target field description information. For example, the preset format can be [field information, target field description information, name information], and there is no limitation here.

[0078] In one embodiment, detecting whether a preset descriptive keyword exists in the initial field description information includes:

[0079] The initial field description information is segmented into words to obtain the initial field description words;

[0080] The cosine similarity between the target description keyword and the initial field description word segmentation is calculated using the distance function in word2vec.

[0081] When the cosine similarity exceeds a preset similarity threshold, it is determined that there is a preset description keyword in the initial field description information, and the initial field description with a cosine similarity exceeding the preset similarity threshold with the target description keyword is segmented as the preset description keyword;

[0082] When the cosine similarity does not exceed the preset similarity threshold, it is determined that the preset descriptive keyword does not exist in the initial field description information.

[0083] The initial field description information is segmented using a probabilistic statistical word segmentation model to obtain initial field description words. The `distance` function in `word2vec` determines whether the target description keyword and the initial field description words are semantically similar. When the cosine similarity exceeds a preset similarity threshold, the target description keyword and the initial field description words are considered semantically similar; when the cosine similarity does not exceed the preset similarity threshold, the target description keyword and the initial field description words are considered semantically dissimilar. The preset similarity threshold is a pre-set threshold used to evaluate whether there is semantic similarity between words.

[0084] S12, each of the target information is vectorized to obtain a target vector set.

[0085] In at least one embodiment of this application, the target information is segmented to obtain the segmentation result corresponding to the target information, the segmentation result is converted into the corresponding word vector, and finally the multiple word vectors are input into a pre-trained vector conversion model to obtain the target vector.

[0086] Optionally, the step of vectorizing each of the target information to obtain a target vector set includes:

[0087] Determine the word vector corresponding to each of the target information;

[0088] The word vectors are input into a pre-trained vector transformation model to obtain the target vector;

[0089] The target vectors are combined to obtain the target vector set.

[0090] The target information can be segmented using a probabilistic statistical word segmentation model to obtain multiple segmentation results. This probabilistic statistical word segmentation model is an existing technology and will not be elaborated upon here. After obtaining multiple segmentation results, the word vectors corresponding to each word in the segmentation results are obtained using the Word2Vec model. Word2Vec is a model that learns semantic knowledge from a large amount of text corpus in an unsupervised manner, and can transform each word in the segmentation results into a corresponding word vector. Specifically, each word in the segmentation results can be transformed into a k-dimensional row vector. The vector transformation model is a pre-trained model used to process the word vectors corresponding to each word in the segmentation results to obtain the target vector.

[0091] In one embodiment, after vectorizing each target information, since the target vector has a large dimension, it is necessary to reduce the dimensionality of the target vector to improve the efficiency of subsequent vector processing, thereby improving the efficiency of determining redundant data tables. In one embodiment, dimensionality reduction can be achieved by performing linear discriminant analysis on the target vector.

[0092] Optionally, after vectorizing each of the target information to obtain a target vector set, the method further includes:

[0093] Determine the intra-class scatter matrix corresponding to the target vector;

[0094] Determine the inter-class scatter matrix corresponding to the target vector;

[0095] The target matrix is ​​obtained by multiplying the inverse of the intra-class scatter matrix with the inter-class scatter matrix.

[0096] Determine multiple feature values ​​corresponding to the target matrix, and extract feature values ​​whose numerical ranking is below a preset ranking threshold from the feature values ​​as the target feature value set;

[0097] Based on the eigenvector corresponding to each eigenvalue in the target eigenvalue set, determine the projection matrix corresponding to the target vector;

[0098] Multiply the transpose of the projection matrix by the target vector to obtain the dimensionality-reduced target vector.

[0099] Linear discriminant analysis (LDA) is a feature extraction algorithm that improves computational efficiency in data analysis. The basic idea of ​​LDA is to project high-dimensional pattern samples onto an optimal discriminant vector space, i.e., projecting data points in the high-dimensional space onto a straight line, reducing the multidimensionality to a low-dimensionality space. It also aims to maximize the inter-class dispersion distance and minimize the intra-class dispersion distance after projection. The preset ranking threshold is a pre-set ranking threshold. In one embodiment, the preset ranking threshold can be the target dimension of the reduced target vector + 1, which is not limited here. Dimensionality reduction using LDA is existing technology and will not be elaborated upon here. For example, the target dimension of the target vector is d, and the target vector before dimensionality reduction is [x1, x2, ..., x...]. n The within-class scatter matrix of the target vector is S. w The inter-class scatter matrix of the target vector is S. b The target matrix is The target matrix is ​​subjected to eigenvalue decomposition, and the eigenvectors corresponding to the first d largest eigenvalues ​​are selected to form the projection matrix W = (w1, w2, ..., w...). d Then, the target vector is reduced in dimensionality to obtain y. i =w T x i Finally, we obtain the dimension-reduced vector Y = [y1, y2, ..., y]. d The calculation methods for the intra-class scatter matrix and the inter-class scatter matrix are existing technologies and will not be elaborated here.

[0100] S13, the target vector set is partitioned to obtain multiple partition vector sets, and a first similarity threshold is determined for each partition vector set.

[0101] In at least one embodiment of this application, the target vector set is partitioned according to the number of fields in a preset data table corresponding to each target vector, resulting in multiple partitioned vector sets. For each partitioned vector set, there exists a corresponding first similarity threshold, which can be obtained through multiple redundant data table monitoring results. For different partitioned vector sets, the corresponding first similarity threshold may be the same or different.

[0102] Please see Figure 3 Optionally, the partitioning of the target vector set to obtain multiple partitioned vector sets includes:

[0103] S130, Obtain the field information corresponding to each target vector in the target vector set;

[0104] S131, determine the number of fields corresponding to each target vector based on the field information;

[0105] S132, the target vectors whose number of fields is within a preset range are divided into the same partition to obtain multiple partition vector sets.

[0106] The field information may include the English names of each field in a preset data table, and the number of English names in the field information is the number of fields. The preset range is a pre-set range for partitioning the target vector. For example, target vectors with the number of fields in the preset range [0, 10] are divided into the same partition, target vectors with the number of fields in the preset range [10, 20] are divided into the same partition, and target vectors with the number of fields in the preset range [20, 40] are divided into the same partition, resulting in multiple partitioned vector sets.

[0107] Optionally, determining the first similarity threshold for each of the partition vector sets includes:

[0108] Obtain the target partition identifier for each partition vector set;

[0109] Obtain the mapping relationship between pre-set partition identifiers and similarity thresholds;

[0110] Based on the target partition identifier, the mapping relationship is traversed to obtain the first similarity threshold corresponding to each partition vector set.

[0111] For each interval of the partition vector set, there exists a corresponding target partition identifier, which uniquely identifies the partition vector set. In one embodiment, the target partition identifier can be a numeric identifier, a color identifier, or a letter identifier, without limitation. A mapping relationship exists between the partition identifier and a similarity threshold. By querying this mapping relationship, the first similarity threshold corresponding to the target partition identifier can be obtained. For example, the first similarity threshold corresponding to the partition vector set in the interval [0, 10] is 0.6, the first similarity threshold corresponding to the partition vector set in the interval [10, 20] is 0.78, and the first similarity threshold corresponding to the partition vector set in the interval [20, 40] is 0.8.

[0112] S14, in the same set of partition vectors, determine a first similarity value between the first partition vector and the second partition vector, and detect whether the first similarity value exceeds the first similarity threshold.

[0113] In at least one embodiment of this application, within each partition vector set, a first similarity value is calculated for any two partition vectors, and two partition vectors whose first similarity value exceeds a first similarity threshold are divided into an initial group vector set. The first partition vector and the second partition vector are any two different partition vectors in the same partition vector set.

[0114] Optionally, determining the first similarity value between the first partition vector and the second partition vector includes:

[0115] Obtain the preset distance calculation model;

[0116] The preset distance calculation model is invoked to process the first partition vector and the second partition vector, and the vector distance between the first partition vector and the second partition vector is obtained as the first similarity value.

[0117] The preset distance calculation model is a pre-set model for calculating the distance between vectors. In one embodiment, the preset distance calculation model can be an Euclidean distance calculation model, which determines the first similarity value between two vectors by calculating the Euclidean distance between the first partition vector and the second partition vector.

[0118] S15, when the first similarity value exceeds the first similarity threshold, the first partition vector and the second partition vector are divided into an initial group vector set in the same partition vector set.

[0119] In at least one embodiment of this application, in each of the partitioned vector sets, the number of the initial co-group vector sets can be one or more, and there is no limitation herein.

[0120] Optionally, after dividing the first partition vector and the second partition vector into an initial co-group vector set within the same partition vector set, the method further includes:

[0121] Determine the preset clustering algorithm;

[0122] The preset clustering algorithm is invoked to perform clustering processing on the initial set of vectors in the same group, thereby obtaining the cluster centers corresponding to the initial set of vectors in the same group.

[0123] The preset clustering algorithm can be the k-means algorithm, where k represents the number of clusters. The basic idea is as follows: after determining the value of k, k objects are randomly selected as initial cluster centers. Then, the Euclidean distance between each object and each cluster center is calculated, and each object is assigned to the nearest cluster center. Thus, each cluster center and the objects assigned to it represent a cluster. The cluster centers are then recalculated based on the existing objects in the cluster. This process is repeated until a termination condition is met. The termination condition could be that no (or a minimum number) objects are reassigned to different clusters, no (or a minimum number) cluster centers change, or the sum of squared errors reaches a local minimum, etc. Since the k-means algorithm is existing technology for clustering, it will not be elaborated upon here. Within the same partitioned vector set, each initial set of vectors has a corresponding cluster center, and the number of cluster centers can be one. The cluster centers are also in vector form.

[0124] In one embodiment, when the first similarity value does not exceed the first similarity threshold, the first partition vector and the second partition vector do not need to be divided into an initial set of vectors in the same group.

[0125] S16, in different partitioned vector sets, determine the second similarity value and the second similarity threshold between the first initial same-group vector set and the second initial same-group vector set, and detect whether the second similarity value exceeds the second similarity threshold.

[0126] In at least one embodiment of this application, the number of partition vector sets is multiple. This application example uses two partition vector sets, namely a first partition vector set and a second partition vector set. Both the first partition vector set and the second partition vector set contain multiple initial same-group vector sets, wherein the first initial same-group vector set is from the first partition vector set, and the second initial same-group vector set is from the second partition vector set.

[0127] Optionally, determining the second similarity value between the first initial set of vectors in the same group and the second initial set of vectors in the same group includes:

[0128] Determine the first cluster center corresponding to the first initial set of vectors in the same group; determine the second cluster center corresponding to the second initial set of vectors in the same group;

[0129] Calculate the second similarity value between the first cluster center and the second cluster center.

[0130] There is a correspondence between the first initial set of vectors in the same group and its corresponding first cluster center. By querying the correspondence, the first cluster center corresponding to the first initial set of vectors in the same group can be obtained. Similarly, there is a correspondence between the second initial set of vectors in the same group and its corresponding second cluster center. By querying the correspondence, the second cluster center corresponding to the second initial set of vectors in the same group can be obtained.

[0131] This application embodiment determines the cluster center corresponding to each initial group of vectors. By comparing the similarity of the cluster centers of each initial group of vectors, it avoids comparing the similarity of each vector in the initial group of vectors, thereby reducing the similarity calculation time and improving the monitoring efficiency of redundant data tables.

[0132] Optionally, determining the second similarity threshold between the first initial set of vectors in the same group and the second initial set of vectors in the same group includes:

[0133] Obtain the first partition vector set corresponding to the first initial group vector set and the first similarity threshold A corresponding to the first partition vector set;

[0134] Obtain the second partition vector set corresponding to the second initial group vector set and the first similarity threshold B corresponding to the second partition vector set;

[0135] The first similarity threshold A and the first similarity threshold B are processed by calling a preset mean calculation model to obtain the second similarity threshold.

[0136] The preset mean calculation model is a pre-set model used to calculate the mean of the first similarity threshold corresponding to two different partition vector sets.

[0137] S17, when the second similarity value exceeds the second similarity threshold, the first initial same-group vector set and the second initial same-group vector set are divided into a target same-group vector set.

[0138] In at least one embodiment of this application, when the second similarity value exceeds the second similarity threshold, the first initial same-group vector set and the second initial same-group vector set are divided into a target same-group vector set; when the second similarity value does not exceed the second similarity threshold, the first initial same-group vector set and the second initial same-group vector set do not need to be divided into a target same-group vector set.

[0139] S18, determine the redundant data table in the target vector set.

[0140] In at least one embodiment of this application, the preset data table corresponding to each target vector in the target vector set may include a source data table and a redundant data table.

[0141] Optionally, determining the redundant data table in the target vector set includes:

[0142] Determine the target features of the source data table;

[0143] Determine the preset data table corresponding to each target vector in the target vector set and the data features of the preset data table;

[0144] Detect whether the data features satisfy the target features;

[0145] When the data feature satisfies the target feature, the preset data table corresponding to the data feature is determined as the source data table;

[0146] When the data feature does not meet the target feature, the preset data table corresponding to the data feature is determined to be a redundant data table.

[0147] The target features refer to the pre-defined characteristics of the source data table. In one embodiment, the target features may include a name length shorter than a pre-defined length threshold and a number of fields less than a pre-defined number threshold. The pre-defined length threshold is the maximum length of the source data table name. The pre-defined number threshold is the maximum number of fields in the source data table.

[0148] In one embodiment, after determining the redundant data table in the target vector set, the method further includes:

[0149] Identify the redundant fields and the number of redundant fields in the redundant data table;

[0150] Determine the total number of fields in the redundant data table;

[0151] The redundancy probability corresponding to the redundant data table is calculated based on the number of redundant fields with the first preset weight and the total number of fields with the second preset weight.

[0152] When the redundancy probability exceeds a preset probability threshold, an early warning is issued for the redundant data table.

[0153] The redundant field refers to a field whose information is identical to that of a field in the source data table within the same vector set as the target. The first preset weight is a weight corresponding to the pre-set number of redundant fields, and the second preset weight is a weight corresponding to the pre-set total number of fields in the redundant data table. In one embodiment, the first preset weight can be 0.7, and the second preset weight can be 0.3. The preset probability threshold is a pre-set threshold used to assess whether an alert is needed; in one embodiment, the preset probability threshold can be 0.7. The alert method may include, but is not limited to, displaying the redundant data table on a page and issuing a red alert.

[0154] The redundant data table monitoring method provided in this application partitions the target vector set to obtain multiple partitioned vector sets and determines a first similarity threshold for each partitioned vector set. By using partitioning and threshold-based similarity determination, the amount of data to be calculated can be reduced, the calculation speed can be increased, and the problem of different definitions of table similarity due to different numbers of fields can be avoided, thus improving the accuracy of similarity calculation and consequently improving the accuracy of redundant data table identification. Furthermore, this application also performs similarity calculations in different partitioned vector sets, which can avoid the problem of identical data tables being assigned to different categories and the omission of classification edge data, further improving the accuracy of redundant data table identification. This application can be applied to various functional modules of smart cities, such as smart government affairs and smart transportation, including the redundant data table monitoring module in smart government affairs, and can promote the rapid development of smart cities.

[0155] In some embodiments, the redundancy table monitoring device 20 may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the redundancy table monitoring device 20 may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality for monitoring redundant data tables.

[0156] Please see Figure 4In this embodiment, the redundancy data table monitoring device 20 can be divided into multiple functional modules according to its functions. These functional modules may include: an information determination module 201, a vector processing module 202, a partitioning processing module 203, a first similarity determination module 204, a first vector grouping module 205, a second similarity determination module 206, a second vector grouping module 207, and a redundancy determination module 208. The term "module" in this application refers to a series of computer program segments that can be executed by at least one processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module will be detailed in subsequent embodiments.

[0157] The information determination module 201 can be used to obtain a preset data table set in the target database and determine the target information corresponding to each preset data table in the preset data table set to obtain a target information set.

[0158] The vector processing module 202 can be used to perform vectorization processing on each target information in the target information set to obtain a target vector set.

[0159] The partitioning module 203 can be used to partition the target vector set to obtain multiple partition vector sets, and determine a first similarity threshold for each partition vector set.

[0160] The first similarity determination module 204 can be used to determine a first similarity value between a first partition vector and a second partition vector in the same set of partition vectors, and detect whether the first similarity value exceeds the first similarity threshold.

[0161] The first vector grouping module 205 can be used to divide the first partition vector and the second partition vector into an initial group vector set in the same partition vector set when the first similarity value exceeds the first similarity threshold.

[0162] The second similarity determination module 206 is further configured to determine a second similarity value and a second similarity threshold between the first initial same group vector set and the second initial same group vector set in different partition vector sets, and detect whether the second similarity value exceeds the second similarity threshold.

[0163] The second vector grouping module 207 is further configured to divide the first initial same-group vector set and the initial second group vector set into a target same-group vector set when the second similarity value exceeds the second similarity threshold.

[0164] The redundancy determination module 208 can be used to determine the redundant data tables in the target vector set.

[0165] In one embodiment, the information determination module 201 can also be used to obtain initial field description information of each preset data table in the preset data table set; detect whether there is a preset description keyword in the initial field description information; when the preset description keyword exists in the initial field description information, replace the preset description keyword with a target description keyword to obtain target field description information; obtain the field information and name information corresponding to the preset data table, and combine the field information, the name information and the target field description information according to a preset format to obtain target information; combine the target information to obtain a target information set.

[0166] In one embodiment, the vector processing module 202 can also be used to determine the intra-class scatter matrix corresponding to the target vector; determine the inter-class scatter matrix corresponding to the target vector; multiply the inverse of the intra-class scatter matrix by the inter-class scatter matrix to obtain the target matrix; determine multiple eigenvalues ​​corresponding to the target matrix, and obtain eigenvalues ​​whose numerical ranking is before a preset ranking threshold from the eigenvalues ​​as a target eigenvalue set; determine the projection matrix corresponding to the target vector according to the eigenvector corresponding to each eigenvalue in the target eigenvalue set; and multiply the transpose of the projection matrix by the target vector to obtain the dimensionality-reduced target vector.

[0167] In one embodiment, the partitioning module 203 can also be used to obtain the field information corresponding to each target vector in the target vector set; determine the number of fields corresponding to each target vector according to the field information; and divide the target vectors with the number of fields within a preset range into the same partition to obtain multiple partition vector sets.

[0168] In one embodiment, the first similarity determination module 204 can also be used to obtain the target partition identifier of each partition vector set; obtain the mapping relationship between the pre-set partition identifier and the similarity threshold; and traverse the mapping relationship according to the target partition identifier to obtain the first similarity threshold corresponding to each partition vector set.

[0169] In one embodiment, the first similarity determination module 204 can also be used to obtain a preset distance calculation model; call the preset distance calculation model to process the first partition vector and the second partition vector, and obtain the vector distance between the first partition vector and the second partition vector as a first similarity value.

[0170] In one embodiment, the redundancy determination module 208 can also be used to determine the target features of the source data table; determine the preset data table corresponding to each target vector in the target vector set and the data features of the preset data table; detect whether the data features satisfy the target features; when the data features satisfy the target features, determine the preset data table corresponding to the data features as the source data table; when the data features do not satisfy the target features, determine the preset data table corresponding to the data features as a redundant data table.

[0171] See Figure 5 The diagram shown is a structural schematic of a computer device provided in Embodiment 3 of this application. In a preferred embodiment of this application, the computer device 3 includes a memory 31, at least one processor 32, at least one communication bus 33, and a transceiver 34.

[0172] Those skilled in the art should understand that Figure 5 The structure of the computer device shown does not constitute a limitation of the embodiments of this application. It can be a bus structure or a star structure. The computer device 3 may also include more or fewer other hardware or software than shown, or different component arrangements.

[0173] In some embodiments, the computer device 3 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The computer device 3 may also include client devices, which include, but are not limited to, any electronic product that can interact with a client via a keyboard, mouse, remote control, touchpad, or voice control device, such as personal computers, tablet computers, smartphones, and digital cameras.

[0174] It should be noted that the computer device 3 described is merely an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0175] In some embodiments, the memory 31 stores a computer program that, when executed by the at least one processor 32, implements all or part of the steps in the redundant data table monitoring method described above. The memory 31 includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0176] Furthermore, the computer-readable storage medium may primarily include a program storage area and a data storage area, wherein the program storage area may store the operating system, at least one application required for a function, etc.; and the data storage area may store data created based on the use of blockchain nodes, etc.

[0177] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0178] In some embodiments, the at least one processor 32 is the control unit of the computer device 3, connecting various components of the computer device 3 via various interfaces and lines. It executes programs or modules stored in the memory 31 and calls data stored in the memory 31 to perform various functions and process data of the computer device 3. For example, when the at least one processor 32 executes the computer program stored in the memory, it implements all or part of the steps of the redundant data table monitoring method described in this application embodiment; or it implements all or part of the functions of the redundant data table monitoring device. The at least one processor 32 may be composed of integrated circuits, such as a single-packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0179] In some embodiments, the at least one communication bus 33 is configured to enable communication between the memory 31 and the at least one processor 32, etc.

[0180] Although not shown, the computer device 3 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 32 via a power management device, thereby enabling functions such as charging, discharging, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 3 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0181] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes multiple instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the methods described in the various embodiments of this application.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0183] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0184] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0185] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that it can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements or, and the singular does not exclude the plural. Multiple elements or devices recited in the specification may also be implemented by a single element or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A method for monitoring redundant data tables, characterized in that, The method for monitoring redundant data tables includes: Obtain a preset set of data tables in the target database, and determine the target information corresponding to each preset data table in the preset data table set to obtain a target information set, including: obtaining initial field description information for each preset data table in the preset data table set; detecting whether a preset description keyword exists in the initial field description information; when the preset description keyword exists in the initial field description information, replacing the preset description keyword with a target description keyword to obtain target field description information; obtaining field information and name information corresponding to the preset data table, and combining the field information, the name information, and the target field description information according to a preset format to obtain target information; combining the target information to obtain a target information set; Each of the target information is vectorized to obtain a target vector set; The target vector set is partitioned to obtain multiple partition vector sets, and a first similarity threshold is determined for each partition vector set. In the same set of partition vectors, a first similarity value is determined between the first partition vector and the second partition vector, and it is detected whether the first similarity value exceeds the first similarity threshold. When the first similarity value exceeds the first similarity threshold, the first partition vector and the second partition vector are divided into an initial group vector set in the same partition vector set; In different partition vector sets, a second similarity value and a second similarity threshold are determined between a first initial same-group vector set and a second initial same-group vector set, and it is detected whether the second similarity value exceeds the second similarity threshold; wherein, the partition vector set includes a first partition vector set and a second partition vector set, the first initial same-group vector set is from the first partition vector set, and the second initial same-group vector set is from the second partition vector set; When the second similarity value exceeds the second similarity threshold, the first initial same-group vector set and the second initial same-group vector set are divided into a target same-group vector set; Identify the redundant data tables in the target vector set.

2. The method for monitoring redundant data tables according to claim 1, characterized in that, After vectorizing each of the target information to obtain a target vector set, the method further includes: Determine the intra-class scatter matrix corresponding to the target vector; Determine the inter-class scatter matrix corresponding to the target vector; The target matrix is ​​obtained by multiplying the inverse of the intra-class scatter matrix with the inter-class scatter matrix. Determine multiple feature values ​​corresponding to the target matrix, and extract feature values ​​whose numerical ranking is below a preset ranking threshold from the feature values ​​as the target feature value set; Based on the eigenvector corresponding to each eigenvalue in the target eigenvalue set, determine the projection matrix corresponding to the target vector; Multiply the transpose of the projection matrix by the target vector to obtain the dimensionality-reduced target vector.

3. The method for monitoring redundant data tables according to claim 1, characterized in that, The partitioning of the target vector set yields multiple partitioned vector sets, including: Obtain the field information corresponding to each target vector in the target vector set; The number of fields corresponding to each target vector is determined based on the field information; The target vectors whose number of fields is within a preset range are divided into the same partition to obtain multiple partition vector sets.

4. The method for monitoring redundant data tables according to claim 1, characterized in that, Determining the first similarity threshold for each partition vector set includes: Obtain the target partition identifier for each partition vector set; Obtain the mapping relationship between pre-set partition identifiers and similarity thresholds; Based on the target partition identifier, the mapping relationship is traversed to obtain the first similarity threshold corresponding to each partition vector set.

5. The method for monitoring redundant data tables according to claim 1, characterized in that, Determining the first similarity value between the first partition vector and the second partition vector includes: Obtain the preset distance calculation model; The preset distance calculation model is invoked to process the first partition vector and the second partition vector, and the vector distance between the first partition vector and the second partition vector is obtained as the first similarity value.

6. The method for monitoring redundant data tables according to claim 1, characterized in that, The step of determining the redundant data table in the target vector set includes: Determine the target features of the source data table; Determine the preset data table corresponding to each target vector in the target vector set and the data features of the preset data table; Detect whether the data features satisfy the target features; When the data feature satisfies the target feature, the preset data table corresponding to the data feature is determined as the source data table; When the data feature does not meet the target feature, the preset data table corresponding to the data feature is determined to be a redundant data table.

7. A redundant data table monitoring device, characterized in that, For performing the redundant data table monitoring method as described in any one of claims 1 to 6, the redundant data table monitoring device comprises: The information determination module is used to obtain a preset set of data tables in the target database and determine the target information corresponding to each preset data table in the preset set of data tables to obtain a target information set. The vector processing module is used to perform vectorization processing on each of the target information to obtain a target vector set; A partitioning module is used to partition the target vector set to obtain multiple partitioned vector sets, and to determine a first similarity threshold for each partitioned vector set. The first similarity determination module is used to determine a first similarity value between a first partition vector and a second partition vector in the same set of partition vectors, and to detect whether the first similarity value exceeds the first similarity threshold. The first vector grouping module is used to divide the first partition vector and the second partition vector into an initial group vector set in the same partition vector set when the first similarity value exceeds the first similarity threshold. The second similarity determination module is used to determine a second similarity value and a second similarity threshold between the first initial same group vector set and the second initial same group vector set in different partition vector sets, and to detect whether the second similarity value exceeds the second similarity threshold. The second vector grouping module is used to divide the first initial same-group vector set and the second initial same-group vector set into a target same-group vector set when the second similarity value exceeds the second similarity threshold. The redundancy determination module is used to determine the redundant data tables in the target vector set.

8. A computer device, characterized in that, The computer device includes a processor that executes a computer program stored in a memory to implement the redundant data table monitoring method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the redundant data table monitoring method as described in any one of claims 1 to 6.

Images