An online real-time duplicate checking analysis method and system suitable for libraries

Abstract

The application relates to the technical field of book duplication checking, in particular to an online real-time duplication checking analysis method and system suitable for libraries, the method comprising the following steps: acquiring a multi-source heterogeneous data set of a book management system, and performing preliminary screening on the acquired data set; performing data deep cleaning and feature extraction based on the screened data set, performing real-time duplication checking according to the extracted feature vectors, performing association rule mining on the duplication checking results, performing book resource classification and evaluation by using the association rules, including classifying the library resources by using a clustering algorithm; and performing visual output according to the results of the association rule mining and the clustering analysis. The application determines the optimal k value by using a contour coefficient, evaluates the clustering results in combination with the association rules, can comprehensively analyze the duplication and utilization rate of the book resources in each cluster, proposes reasonable resource cleaning, optimization or expansion suggestions according to different situations, and improves the configuration efficiency and management level of the book resources.

Description

Technical Field

[0001] This invention relates to the field of book plagiarism detection technology, and in particular to a method and system for online real-time plagiarism analysis in libraries. Background Technology

[0002] With the rapid development of information technology, the scale of library collections has grown exponentially, and their types have become increasingly diverse, covering traditional paper books, electronic documents, audio-visual materials, etc. At the same time, the business exchanges between libraries and external institutions and readers are becoming more and more frequent, and cooperation in resource procurement, shared lending, and academic research support is constantly deepening. Against this background, efficient management, accurate deduplication, and in-depth analysis of library data have become crucial.

[0003] In terms of library management, existing technologies have many shortcomings. On the one hand, many libraries rely on manual recording and scattered spreadsheet management for data maintenance, lacking unified and efficient system support. This approach not only leads to delayed data updates, affecting the timely availability of new resources and the implementation of services, but also easily results in data entry errors, making it difficult to ensure data consistency and integrity. On the other hand, existing plagiarism detection tools have obvious limitations. Most tools only support single-data plagiarism checks, which cannot meet the needs of large-scale data processing. When libraries need to compare a large number of collections, or when authorized clients need to process a large number of data for plagiarism checks at once, the operation is cumbersome and time-consuming, significantly reducing work efficiency. At the same time, the records during the plagiarism detection process are not detailed enough, making it difficult to trace specific operational details. The results display and export functions are also extremely limited, which is not conducive to the in-depth analysis and utilization of the plagiarism results and cannot provide strong support for library resource integration and procurement decision adjustments. At present, there is a need for a method and system for online real-time plagiarism analysis suitable for libraries. Summary of the Invention

[0004] To address the issues of low efficiency in book plagiarism detection and incomplete analysis of plagiarism results, this invention provides a method and system for online real-time plagiarism detection and analysis suitable for libraries.

[0005] Firstly, the present invention provides a method for online real-time deduplication analysis in libraries, employing the following technical solution:

[0006] A method for online real-time plagiarism detection and analysis suitable for libraries includes:

[0007] Obtain multi-source heterogeneous datasets from the library management system and perform preliminary screening on the obtained datasets;

[0008] Deep cleaning and feature extraction were performed on the filtered dataset, including missing value imputation based on Bayesian network and feature vectorization using TF-IDF algorithm.

[0009] Real-time deduplication is performed based on the extracted feature vectors, including building a hash index and calculating the similarity between records, and finally marking the deduplication results.

[0010] The plagiarism detection results are used to mine association rules, including mining frequent itemsets and association rules by calculating the support and confidence of itemsets;

[0011] Using association rules for the classification and evaluation of library resources, including using the K-Means clustering algorithm to classify library resources;

[0012] The results of association rule mining and cluster analysis are visualized.

[0013] Furthermore, the preliminary screening of the acquired dataset includes applying information entropy theory to the collected data. For each record, the information entropy of each field is calculated, and fields with information entropy below a set threshold are considered low-value information and are initially removed. The formula for calculating information entropy is:

[0014] ,

[0015] in, Represented as a field, Represented as field values The probability of.

[0016] Furthermore, the missing value imputation method based on Bayesian networks handles missing values ​​by constructing a Bayesian network to describe the dependencies between fields in the library dataset, where nodes represent fields in the dataset, directed edges between nodes represent dependencies between fields, the structure and parameters of the Bayesian network are learned using the maximum likelihood estimation method, the conditional probability of fields with missing values ​​in the dataset is calculated according to Bayes' theorem, a dynamic Bayesian network is introduced, and time slices are divided, with a Bayesian network structure constructed for each time slice, and nodes between adjacent time slices are connected to each other through transition probabilities.

[0017] Furthermore, the real-time deduplication based on the extracted feature vectors includes constructing a hash index for the extracted features using a Bloom filter, mapping records to binary vectors using multiple hash functions, and calculating the similarity between records using a deep learning-based Siamese network model for different records. The Siamese network consists of two sub-networks with shared weights. Two records are input into the two sub-networks respectively, and feature vectors are output. The similarity between feature vectors is calculated using Euclidean distance, and records with similarity higher than a set threshold are marked as duplicate records.

[0018] Furthermore, the process of mining frequent itemsets and association rules by calculating the support and confidence of itemsets includes employing a dynamic threshold adjustment strategy to perform cluster analysis on historical data, dividing the data into multiple clusters based on different characteristics, calculating the standard deviation and coefficient of variation of each cluster to assess the data dispersion, determining the minimum support threshold required for mining frequent itemsets based on the data dispersion, using the weighted Apriori algorithm to assign weights to each item, calculating the support based on the assigned weights, and filtering out frequent itemsets based on the support and the minimum support threshold.

[0019] Furthermore, the step of mining frequent itemsets and association rules by calculating the support and confidence of itemsets also includes calculating the confidence based on frequent itemsets, generating association rules when the confidence is greater than a set confidence threshold, and updating the association rules using an incremental association rule update method. The confidence calculation formula is as follows:

[0020] ,

[0021] in, This is represented as an association rule from itemset X to itemset Y. Let be the weighted support of the union of itemsets X and Y. It is represented as the weighted support of itemset X.

[0022] Furthermore, the K-Means clustering algorithm is used to classify library resources, including determining the optimal k value using the silhouette coefficient, selecting the feature vectors of k book resources as initial cluster centers, calculating the distance between the feature vector of each book and each cluster center according to the Euclidean distance, assigning the feature vector to the cluster containing the nearest cluster center, calculating the mean of the feature vectors of all books in the cluster and using it as the new cluster center, evaluating the book resources based on the clustering results, and analyzing the duplication and utilization rate of book resources in each cluster.

[0023] Secondly, a system for online real-time plagiarism detection and analysis suitable for libraries includes:

[0024] The data acquisition module is configured to: acquire multi-source heterogeneous datasets from the library management system and perform preliminary screening of the acquired datasets;

[0025] The feature extraction module is configured to perform deep data cleaning and feature extraction based on the filtered dataset, including handling missing values ​​using a Bayesian network-based missing value imputation method and using the TF-IDF algorithm for feature vectorization.

[0026] The plagiarism detection module is configured to perform real-time plagiarism detection based on the extracted feature vectors, including building a hash index and calculating the similarity between records, and finally marking the plagiarism detection results.

[0027] The association module is configured to perform association rule mining on the deduplication results, including mining frequent itemsets and association rules by calculating the support and confidence of itemsets.

[0028] The resource classification module is configured to classify and evaluate book resources using association rules, including classifying library resources using the K-Means clustering algorithm.

[0029] The output module is configured to visualize the results of association rule mining and cluster analysis.

[0030] Thirdly, the present invention provides a computer-readable storage medium storing a plurality of instructions adapted to be loaded and executed by a processor of a terminal device, the aforementioned method for online real-time plagiarism detection and analysis in libraries.

[0031] Fourthly, the present invention provides a terminal device, including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor to provide a method for online real-time plagiarism analysis applicable to libraries.

[0032] In summary, the present invention has the following beneficial technical effects:

[0033] 1. This invention uses information entropy theory for preliminary screening, which can quickly eliminate low-value information, reduce data redundancy, and improve the efficiency of subsequent processing. The missing value imputation method based on Bayesian network, especially the introduction of dynamic Bayesian network, can fully consider the dependencies and dynamic changes between data fields, and more accurately fill missing values. Compared with traditional methods, it greatly improves the integrity and accuracy of data, and provides a reliable data foundation for subsequent analysis.

[0034] 2. This invention utilizes a Bloom filter to construct a hash index, which can quickly determine whether records are likely to be duplicated in massive amounts of book data, greatly improving the speed of deduplication. Based on the Siamese network model of deep learning, it calculates similarity, which can more accurately identify the similarity between book records compared with traditional similarity calculation methods. It accurately marks records with similarity higher than a set threshold as duplicate records, reduces the false judgment rate, and effectively avoids duplicate procurement of resources and chaotic management.

[0035] 3. When using the Apriori algorithm to mine association rules, this invention employs a dynamic threshold adjustment strategy and a weighted Apriori algorithm. Based on data characteristics and business needs, it can more accurately uncover potential and valuable associations between book resources, such as the close relationship between specific authors and publishers, and between Chinese Library Classification numbers, providing in-depth data support for book management decisions.

[0036] 4. When classifying book resources using the K-Means clustering algorithm, this invention uses the silhouette coefficient to determine the optimal k value, making the clustering results more consistent with the actual distribution of book resources. Combined with association rules, the clustering results are evaluated, which can comprehensively analyze the duplication and utilization rate of book resources within each cluster. Reasonable suggestions for resource cleaning, optimization or expansion are proposed for different situations, thereby improving the allocation efficiency and management level of book resources.

[0037] 5. This invention visualizes the results of association rule mining and cluster analysis, presenting information such as the relationships, classifications, and utilization rates of book resources in intuitive charts. This facilitates library management personnel in quickly understanding the information behind the data, providing a powerful visual decision-making tool for procurement decisions, bookshelf layout adjustments, and resource promotion, thereby enhancing the scientific and timely nature of library management decisions. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the overall process of an online real-time plagiarism detection and analysis method applicable to libraries, according to an embodiment of the present invention. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the accompanying drawings.

[0040] Example 1

[0041] Reference Figure 1 This embodiment of a method for online real-time plagiarism detection and analysis applicable to libraries includes:

[0042] Obtain multi-source heterogeneous datasets from the library management system and perform preliminary screening on the obtained datasets;

[0043] Deep cleaning and feature extraction were performed on the filtered dataset, including missing value imputation based on Bayesian network and feature vectorization using TF-IDF algorithm.

[0044] Real-time deduplication is performed based on the extracted feature vectors, including building a hash index and calculating the similarity between records, and finally marking the deduplication results.

[0045] The plagiarism detection results are used to mine association rules, including mining frequent itemsets and association rules by calculating the support and confidence of itemsets;

[0046] Using association rules for the classification and evaluation of library resources, including using the K-Means clustering algorithm to classify library resources;

[0047] The results of association rule mining and cluster analysis are visualized.

[0048] Specifically, a method for online real-time plagiarism detection and analysis suitable for libraries includes the following steps:

[0049] like Figure 1 As shown, S1, obtain the multi-source heterogeneous dataset of the library management system and perform preliminary screening on the obtained dataset;

[0050] Data was acquired from multiple sources, including the library management system, collection database, purchase order records, and electronic resource platforms, covering various resource information such as print books, electronic journals, and audio-visual materials. The collected data included structured data (such as database tables) and unstructured data (such as text descriptions and image captions). Preliminary data screening was then conducted using information entropy theory. For each record, the information entropy of each field was calculated. Fields with information entropy below a set threshold (e.g., 0.2) were considered low-value information and initially removed. The formula for calculating information entropy is:

[0051] ,

[0052] Where X is a field. Get values ​​for a field The probability of.

[0053] S2. Based on the filtered dataset, perform deep data cleaning and feature extraction, including handling missing values ​​using a Bayesian network-based missing value imputation method and using the TF-IDF algorithm for feature vectorization.

[0054] Even after initial screening, the data may still contain issues that affect the accuracy and efficiency of subsequent analysis. Missing values ​​are a common problem that can impact the results of subsequent data analysis and mining. Traditional methods for handling missing values, such as simple mean or median imputation, do not take into account the dependencies between fields in the dataset. However, missing value imputation methods based on Bayesian networks can effectively solve this problem.

[0055] Bayesian networks are essentially probabilistic graphical models that represent probabilistic dependencies between variables using directed acyclic graphs. In traditional Bayesian networks, the relationships between variables are modeled based on static data, neglecting the impact of time. However, in library management scenarios, data exhibits significant dynamic characteristics. For example, library acquisition strategies differ from year to year, and readers' borrowing preferences change over time. This leads to book-related data (such as borrowing volume and book category distribution) displaying different characteristics at different times. This application employs Dynamic Bayesian Networks (DBNs) to incorporate the time dimension into traditional Bayesian networks. In library management data processing, time slices are divided according to chronological order. Assuming each time slice is... In each time slice Constructing Bayesian network structures ,in, It is a time film The set of nodes corresponds to the various fields of the book data, such as... ; It is the set of directed edges between nodes within time slice t, representing the dependencies between fields within that time slice.

[0056] For a Bayesian network at each time slice t, the conditional probability distribution between nodes needs to be learned. Let the nodes... Its conditional probability distribution is ,in, It is a node exist The set of parent nodes in the data is used to determine these conditional probability distributions using the maximum likelihood estimation method, and adjacent time slices. and The nodes are connected by transition probabilities. Taking node X as an example, its values ​​in different time slices... and The transition probability is expressed as Suppose X represents the number of times a book is borrowed, and its value is a set of discrete values. The transition probability matrix is ​​determined based on the discrete values ​​and transition probabilities. The elements in this matrix represent the transition from time slice to time slice. Transitioning to a certain value of X in time slices The probability of X taking another value is used to connect nodes between adjacent time slices via a transition probability matrix. Specifically, the connection method is as follows: assuming in time slice... Node X is in state The probability is The elements in the transition probability matrix A Indicates from state Transition to state The probability, i.e. So in time slice At that time, node X is in state probability Calculated using the following formula: ,

[0057] This formula means that, for a time slice Each possible state of node X Its probability is determined by the time slice. All possible states of node X Transferred to The sum of probabilities, through this formula, can be used to determine the time slice. The state probability distribution and transition probability matrix of the middle node are used to calculate the time slice. The state probability distribution of the nodes in the middle enables nodes to be interconnected through transition probabilities between adjacent time slices.

[0058] Collect time slices after connection Excluding missing value fields Other observational data besides those mentioned above are denoted as Collect the past k time slices ( arrive The observation data of ) is denoted as ,

[0059] First, calculate the numerator of the conditional probability distribution. Using time slices Bayesian network structure Conditional probability distribution between nodes And the transition probabilities of adjacent time slices. Based on the dependencies between nodes in the Bayesian network, combined with known observation data. Calculate in a given The joint probability of these observations occurring under different values.

[0060] Next, calculate the denominator of the conditional probability distribution. It is a normalization constant used to ensure that the final calculated The sum of the conditional probability distributions is 1, which is determined by the conditional probability distributions of the molecule. exist We obtain the summation over all possible values, where, for Based on the prior probability obtained, and using Bayes' theorem formula, the numerator and denominator are used to calculate the probability under the given conditions. and In this case, The probability distribution of different values ​​is obtained from the calculation. Given a conditional probability distribution, select the value with the highest probability from the conditional probability distribution as the filler value.

[0061] Key features, such as book title, author, ISBN, publication year, and Chinese Library Classification number, are extracted from the filled data. For textual features (such as book title and author), the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm is used to vectorize the features, converting the text into numerical feature vectors, which facilitates subsequent deduplication and analysis operations.

[0062] S3. Perform real-time deduplication based on the extracted feature vectors, including building a hash index and calculating the similarity between records, and finally marking the deduplication results;

[0063] A hash index is constructed using the extracted features through a Bloom filter. A Bloom filter is a space-efficient probabilistic data structure that maps records to a binary vector using multiple hash functions. For book data, the feature vector of each book (e.g., a vector obtained by vectorizing features such as title, author, publisher, and publication year) is processed by multiple hash functions. Each hash function corresponds to a position in the binary vector, setting the value of that position to 1 (initially, all bits in the binary vector are 0). In this way, a large number of book records can be efficiently compressed and stored in the Bloom filter, allowing for quick determination of whether a record exists in the existing record set (although there is a certain false positive rate, it greatly improves search efficiency in large-scale data processing).

[0064] A deep learning-based Siamese network model is used to calculate the similarity between different records. The Siamese network has a unique structure consisting of two sub-networks sharing weights. When processing book records, the two records to be compared are input into these two sub-networks respectively. The sub-networks perform a series of complex feature extraction and transformation operations on the input records, ultimately outputting feature vectors. Because the two sub-networks share weights, they can extract features from different input records in the same way, making the output feature vectors comparable. Then, the distance between these two feature vectors is calculated using Euclidean distance. Euclidean distance is a commonly used method to measure the distance between two vectors in a spatial space, intuitively reflecting how close the two vectors are in space. In calculating the similarity of book records, the smaller the Euclidean distance between the feature vectors, the more similar the two book records are.

[0065] Based on the calculated similarity, records with similarity scores higher than a set threshold are marked as duplicate records. This threshold is pre-set based on actual business needs and data analysis. For example, if the threshold is set to 0.8 (similarity values ​​are assumed to range from 0 to 1, with higher values ​​indicating greater similarity), when the similarity between two book records calculated using the Siamese network and Euclidean distance is 0.85, these two records will be marked as duplicate records.

[0066] S4. Perform association rule mining on the plagiarism detection results, including mining frequent itemsets and association rules by calculating the support and confidence of itemsets;

[0067] Traditional Apriori algorithms use fixed minimum support and minimum confidence thresholds, which are difficult to adapt to the complex and ever-changing characteristics of library data. This solution introduces a dynamic threshold adjustment strategy, dynamically adjusting the thresholds based on data distribution and business needs.

[0068] By performing cluster analysis on historical data, the data is divided into multiple clusters according to different characteristics (such as time, resource type, etc.). For each cluster, the standard deviation and coefficient of variation are calculated to assess the dispersion of the data. For clusters with high dispersion, the minimum support threshold is adjusted to the lower limit of the initial range. For example, if the coefficient of variation (CV) of a cluster is greater than a certain set threshold... Then the minimum support threshold of the cluster is... Set as For clusters with low dispersion, the minimum support threshold is adjusted towards the upper limit of the initial range. If the CV is less than... ,but Simultaneously, in conjunction with the library's business objectives, a minimum confidence threshold is determined. Let there be m key indicators related to the library's business objectives. For each key indicator... Define a weight ( ),and This indicates the importance of the indicator in the business objectives, for a relationship rule. Define a correlation index This is used to measure the association rule R with key metrics. The degree of closeness of the connection, The larger the value, the closer the association rule R is to the key metric. The closer the correlation, the better. Let the initial minimum confidence threshold be... (This embodiment uses 0.6), and the final minimum confidence threshold minConf is calculated using the following formula:

[0069] ,

[0070] in, It is an adjustment range parameter This indicates the increase in the minimum confidence threshold when the association rule is closely related to the key metric.

[0071] In library data, different resource attributes have varying importance to association rules. When evaluating book associations, the "Chinese Library Classification Number" may reflect the intrinsic connections between books better than the "Publisher's Location." Therefore, this embodiment proposes a weighted Apriori algorithm, assigning a weight to each item.

[0072] The support calculation formula has been adjusted to:

[0073]

[0074] in, Given the weight of item i, calculate the support of the itemset, and filter out itemsets with a support greater than the minimum support threshold. When using the Apriori algorithm to mine frequent itemsets, if a subset of a frequent itemset does not meet the minimum support threshold, the subset is removed from the frequent itemset search space.

[0075] When calculating the confidence score, the weighting factor is also considered. Based on the mined frequent itemsets, the adjusted confidence score formula is as follows:

[0076] ,

[0077] in, This is represented as an association rule from itemset X to itemset Y. Let be the weighted support of the union of itemsets X and Y. It is represented as the weighted support of itemset X. When the confidence is greater than the set confidence threshold, an association rule is generated. If new data causes the confidence of some association rules to fall below the set minimum confidence threshold, these association rules are deleted from the rule base.

[0078] Finally, an incremental association rule update method is adopted to update association rules. As book data continues to increase and change, existing association rules may no longer accurately reflect the relationships between data. The incremental update method can efficiently update association rules when new data is added. When new data is added, the impact of the new data on existing frequent itemsets is first determined. For newly added transactions, the intersection of the itemsets contained therein and existing frequent itemsets is calculated to determine which frequent itemsets may be affected. Then, the weighted support and weighted confidence of the potentially affected frequent itemsets are recalculated. If the new data causes the confidence of some association rules to fall below the set minimum confidence threshold, these association rules are deleted from the rule base. At the same time, new frequent itemsets and association rules that may be generated by the new data are added to the rule base after calculation and screening, thereby ensuring that the association rules can always accurately reflect the relationships between book data.

[0079] S5. Classify and evaluate library resources using association rules, including classifying library resources using the K-Means clustering algorithm;

[0080] The K-Means clustering algorithm is used to classify library resources, including determining the optimal k value using the silhouette coefficient. The silhouette coefficient measures the tightness and separation of clusters. For each sample point in the dataset, the silhouette coefficient is calculated based on two key distances: first, the average distance between the sample point and other sample points within the same cluster (denoted as 'a'), the smaller this distance, the tighter the sample point is within its cluster; second, the average distance between the sample point and the nearest other sample point in its cluster (denoted as 'b'). The formula for calculating the silhouette coefficient 's' is... The value of s ranges from [-1, 1]. When s is close to 1, it indicates that the sample point is closely connected to other points in its cluster and has a high degree of separation from other clusters, indicating a good clustering effect. When s is close to -1, it indicates that the sample point may have been incorrectly assigned to an unsuitable cluster. The k value with the largest average silhouette coefficient is selected as the optimal number of clusters. The feature vectors of k book resources are used as the initial cluster centers. The distance between the feature vector of each book and each cluster center is calculated according to the Euclidean distance. The feature vectors are assigned to the cluster containing the nearest cluster center. The mean of the feature vectors of all books in the cluster is calculated and used as the new cluster center. The book resources are evaluated based on the clustering results, and the duplication and utilization rate of book resources in each cluster are analyzed.

[0081] S6. Visualize the results of association rule mining and cluster analysis.

[0082] The results of association rules and cluster analysis are presented in a comprehensive manner. Based on the cluster diagram, important association rules related to certain clusters are marked to provide a more comprehensive understanding of the relationships and clustering of book resources. Finally, pie charts or bar charts are created to show the duplication rate of different types of books. Books are divided into different categories according to the Chinese Library Classification number, and the proportion of books marked as duplicates in each category is counted to show which categories have more serious duplication issues. This provides a reference for libraries to make reasonable adjustments to procurement strategies and reduce unnecessary duplicate purchases when purchasing and managing resources.

[0083] Example 2

[0084] The difference between this embodiment and Embodiment 1 is that this embodiment provides a library online real-time plagiarism detection and analysis system, including:

[0085] The data acquisition module is configured to: acquire multi-source heterogeneous datasets from the library management system and perform preliminary screening of the acquired datasets;

[0086] The feature extraction module is configured to perform deep data cleaning and feature extraction based on the filtered dataset, including handling missing values ​​using a Bayesian network-based missing value imputation method and using the TF-IDF algorithm for feature vectorization.

[0087] The plagiarism detection module is configured to perform real-time plagiarism detection based on the extracted feature vectors, including building a hash index and calculating the similarity between records, and finally marking the plagiarism detection results.

[0088] The association module is configured to perform association rule mining on the deduplication results, including mining frequent itemsets and association rules by calculating the support and confidence of itemsets.

[0089] The resource classification module is configured to classify and evaluate book resources using association rules, including classifying library resources using the K-Means clustering algorithm.

[0090] The output module is configured to visualize the results of association rule mining and cluster analysis.

[0091] A computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device, the aforementioned method for online real-time plagiarism detection and analysis in libraries.

[0092] A terminal device includes a processor and a computer-readable storage medium, the processor being used to implement various instructions; the computer-readable storage medium being used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor, and the aforementioned method for online real-time plagiarism detection and analysis in libraries.

[0093] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. An online real-time duplicate analysis method suitable for libraries, characterized in that, The method comprises the following steps: acquiring a multi-source heterogeneous data set of a library management system and performing preliminary screening on the acquired data set; Based on the screened data set, data deep cleaning and feature extraction are carried out, including missing value filling method based on Bayesian network to process missing values, and using algorithm for feature vectorization; performing real-time duplicate checking according to the extracted feature vectors, including constructing a hash index and calculating the similarity between records, and finally marking the duplicate checking results; Correlation rule mining is performed on the duplicate checking result, and support and confidence of item sets are calculated to mine frequent item sets and correlation rules, including: a dynamic threshold adjustment strategy is adopted to perform cluster analysis on historical data, the data is divided into multiple clusters according to different characteristics of the data, data standard deviation and coefficient of variation of each cluster are calculated to evaluate data dispersion, the initial threshold of the coefficient of variation of the cluster is set as , if the coefficient of variation CV of the cluster is greater than the threshold , the minimum support threshold of the cluster is set as , if the coefficient of variation CV of the cluster is less than , the minimum support threshold of the cluster is set as . Using a weighted algorithm, assigning a weight to each item, calculating the support according to the assigned weight, screening the frequent item set through the support and a minimum support threshold, calculating the confidence based on the frequent item set, and determining a minimum confidence threshold; The determining the minimum confidence threshold comprises: acquiring key indicators related to the library business target, assigning a weight to each key indicator, and calculating a final minimum confidence threshold by a formula: wherein minConf is the final minimum confidence threshold, is an initial minimum confidence threshold, is the ith key indicator corresponding weight, is the number of key indicators, is a correlation degree indicator, used to measure the correlation between the correlation rule and the key indicator , is an adjustment range parameter, when the confidence is greater than the set confidence threshold, the correlation rule is generated, and an incremental correlation rule updating method is used for correlation rule updating. classifying and evaluating the library resources by using the association rules, including classifying the library resources by using a K-Means clustering algorithm; outputting the results of the association rule mining and the clustering analysis in a visualized manner.

2. The method for online real-time duplicate analysis for library according to claim 1, characterized in that, The preliminary screening on the acquired data set comprises the following steps: applying information entropy theory to perform preliminary screening on the collected data; for each record, calculating the information entropy of each field; and regarding the fields with information entropy lower than a set threshold as low-value information and performing preliminary elimination. ， wherein, is represented as a field, is represented as a field value probability.

3. The method for online real-time duplicate analysis for library according to claim 1, characterized in that, The method for filling in the missing values based on the Bayesian network comprises the following steps: constructing a Bayesian network to describe the dependency relationship between the fields in the library data set, wherein the nodes represent the fields in the data set, and the directed edges between the nodes represent the dependency relationship between the fields; learning the structure and parameters of the Bayesian network by using a maximum likelihood estimation method; calculating the conditional probability of the fields with missing values in the data set according to the Bayesian theorem; introducing a dynamic Bayesian network and dividing time slices; and constructing a Bayesian network structure for each time slice, and connecting the nodes between adjacent time slices through transition probabilities.

4. The method for online real-time duplicate analysis for libraries according to claim 1, wherein, The real-time duplicate checking according to the extracted feature vectors comprises the following steps: constructing a hash index for the extracted features by using a Bloom filter; mapping the records into binary vectors through multiple hash functions; for different records, calculating the similarity between the records by using a Siamese network model based on deep learning, wherein the Siamese network is composed of two sub-networks sharing weights, two records are input into the two sub-networks respectively, and the feature vectors are output; the similarity between the feature vectors is calculated through the Euclidean distance; and the records with a similarity higher than a set threshold are marked as duplicate records.

5. The method for online real-time duplicate analysis for library according to claim 1, characterized in that, The classification of the library resources by using the K-Means clustering algorithm comprises the following steps: determining the optimal k value by using a silhouette coefficient; selecting the feature vectors of k library resources as initial clustering centers; calculating the distance between the feature vector of each book and each clustering center according to the Euclidean distance; assigning the feature vector to the cluster where the nearest clustering center is located; calculating the mean value of all book feature vectors in the cluster as a new clustering center; evaluating the library resources according to the clustering results; and analyzing the duplication and utilization rate of the library resources in each cluster.

6. A library online real-time duplicate analysis system suitable for, The method comprises the following steps: a data acquisition module configured to acquire a multi-source heterogeneous data set of a library management system and perform preliminary screening on the acquired data set; a feature extraction module configured to perform data deep cleaning and feature extraction based on the screened data set, including a method for filling in missing values based on a Bayesian network to process the missing values, and performing feature vectorization by using a TF-IDF algorithm; The duplicate checking module is configured to perform real-time duplicate checking according to the extracted feature vectors, including constructing a hash index and calculating the similarity between records, and finally marking the duplicate checking result; The association module is configured to perform association rule mining on the duplicate checking result, including mining frequent item sets and association rules by calculating the support and confidence of the item sets; The resource classification module is configured to classify and evaluate the library resources using the association rules, including classifying the library resources using a K-Means clustering algorithm; The output module is configured to visually output the results of the association rule mining and clustering analysis.

7. A computer readable storage medium having stored therein a plurality of instructions, wherein the instructions, when executed by a processor, cause the processor to perform the method of any one of claims 1-6. The instructions are adapted to be loaded and executed by the processor of the terminal device to implement the library online real-time duplicate checking analysis method according to any one of claims 1-5. 8.A terminal device, comprising a processor and a computer readable storage medium, the processor is configured to implement instructions; the computer readable storage medium is configured to store a plurality of instructions, characterized in that, The instructions are adapted to be loaded and executed by the processor to implement the library online real-time duplicate checking analysis method according to any one of claims 1-5.

Images