Data storage method, system, device and storage medium based on multi-source heterogeneity

Abstract

The present invention provides a multi-source heterogeneous data storage method, system, device, and storage medium. The method includes: obtaining multiple document contents based on multiple metadata in a database, and obtaining multiple feature values ​​corresponding to each document content; calculating each feature value based on a text similarity algorithm and a word frequency-inverse file frequency weighted algorithm to obtain a fingerprint value corresponding to each document content; performing a Hamming distance comparison on the fingerprint value corresponding to each document content with each fingerprint value in a set of fingerprint values ​​of existing documents to obtain a comparison result; when the comparison result meets a preset condition, determining that the document content corresponding to the fingerprint value is non-duplicate, and storing the corresponding metadata in a storage system. The method calculates the corresponding fingerprint value of each document based on its feature value, and compares the fingerprint value of each document with the set of fingerprint values ​​of existing documents to obtain non-duplicate documents, thereby enabling rapid batch deduplication and storage of multi-source heterogeneous document content.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a data storage method, system, device and storage medium based on multi-source heterogeneity. Background Art

[0002] With the continuous development of big data processing technology, the data generated by various information systems is becoming increasingly interconnected, forming information networks such as social networks, mobile internet, biomolecular networks, digital resources, and knowledge graphs. Within the library, information, and archives management industry, research focuses on heterogeneous information networks composed of digital resources, encompassing their organization, management, disclosure, use, and analysis. The goal of this research focus is to improve service quality and efficiency, specifically, to provide readers with faster, better, and more convenient service throughout the entire document service process, from user search requests to resource / knowledge acquisition.

[0003] However, the current digital resource information network presents the following characteristics: (1) The amount of data is extremely large; with the explosive growth of global information, resource metadata is controlled by different digital resource providers, with a total amount exceeding tens of billions, and the storage cost is high. (2) The data duplication rate is high; due to the commercial or technical barriers between digital resource providers, the resources of different providers have cross-cutting, overlapping, and complementary characteristics, resulting in low resource access quality. (3) Metadata is multi-source and heterogeneous; almost all digital resource providers have their own specific standard open access methods, with uneven fields, different meanings, and different formats, making knowledge conversion difficult. Therefore, how to achieve non-duplicated storage of multi-source heterogeneous data is an urgent problem to be solved. Summary of the Invention

[0004] In view of the technical problems existing in the prior art, the present invention provides a data storage method, system, device and storage medium based on multi-source heterogeneity to solve the problem of how to achieve non-duplicated storage of multi-source heterogeneous data.

[0005] According to a first aspect of the present invention, a multi-source heterogeneous data storage method is provided, comprising:

[0006] Obtain multiple document contents in different formats from multiple data sources based on multiple metadata in a PgSQL database, and obtain multiple feature values ​​corresponding to each document content;

[0007] Each feature value is calculated using the Simhash text similarity algorithm and the TF-IDF term frequency-inverse document frequency weighted algorithm to obtain the fingerprint value corresponding to each document content;

[0008] Obtain a set of fingerprint values ​​of existing documents, perform Hamming distance comparison between the fingerprint value corresponding to each document content and each fingerprint value in the fingerprint value set, and obtain a comparison result;

[0009] When the comparison result meets the preset conditions, it is determined that the document content corresponding to the fingerprint value is not repeated, and the corresponding metadata is stored in the storage system.

[0010] On the basis of the above technical solution, the present invention can also make the following improvements.

[0011] Optionally, the step of obtaining multiple document contents in different formats from multiple data sources based on multiple metadata in the PgSQL database, and obtaining multiple feature values ​​corresponding to each document content, includes:

[0012] Get multiple metadata in the PgSQL database;

[0013] Acquire multiple document contents in different formats from multiple data sources according to the multiple metadata;

[0014] The key fields in each document content are extracted as its corresponding multiple feature values.

[0015] Optionally, the key fields include at least: title, author, and signing organization.

[0016] Optionally, the step of calculating each feature value according to the Simhash text similarity algorithm and the TF-IDF term frequency-inverse document frequency weighted algorithm to obtain a fingerprint value corresponding to each document content includes:

[0017] According to the word frequency and inverse document frequency corresponding to each eigenvalue, the TF-IDF word frequency-inverse document frequency weighted algorithm is used to obtain the weight value corresponding to each eigenvalue;

[0018] Each feature value of each document content is calculated according to the Simhash text similarity algorithm and the weight value to obtain a fingerprint value corresponding to each document content.

[0019] Optionally, the step of obtaining a weight value corresponding to each eigenvalue using a TF-IDF word frequency-inverse document frequency weighted algorithm based on the word frequency and inverse document frequency corresponding to each eigenvalue further includes:

[0020] Obtain weight values ​​corresponding to multiple feature values ​​corresponding to each document content;

[0021] Performing linear fitting on the weight values ​​corresponding to the multiple eigenvalues ​​according to Zipf's law to obtain a fitting function;

[0022] The weight values ​​corresponding to the multiple eigenvalues ​​are optimized according to the fitting function to obtain the optimized weight values ​​corresponding to the multiple eigenvalues.

[0023] Optionally, the step of performing Hamming distance comparison on the fingerprint value corresponding to each document content with each fingerprint value in the fingerprint value set to obtain a comparison result includes:

[0024] The sliding window algorithm is used to perform Hamming distance comparison between the fingerprint value corresponding to each document content and each fingerprint value in the fingerprint value set, and a comparison fingerprint value set in the fingerprint value set that meets the difference threshold is obtained.

[0025] Optionally, when the comparison result satisfies a preset condition, the step of determining whether the document content corresponding to the fingerprint value is non-repetitive includes:

[0026] When the number of elements in the comparison fingerprint value set is equal to 0, it is determined that the fingerprint values ​​of the document contents corresponding to the comparison fingerprint value set are not repeated.

[0027] According to a second aspect of the present invention, there is provided a multi-source heterogeneous data storage system, comprising:

[0028] A feature acquisition module is used to acquire multiple document contents in different formats from multiple data sources based on multiple metadata in a PgSQL database, and to acquire multiple feature values ​​corresponding to each document content;

[0029] Fingerprint calculation module, used to calculate each feature value based on Simhash text similarity algorithm and TF-IDF word frequency-inverse document frequency weighted algorithm to obtain the fingerprint value corresponding to each document content;

[0030] The fingerprint comparison module is used to obtain a fingerprint value set of existing documents, and compare the fingerprint value corresponding to each document content with each fingerprint value in the fingerprint value set by Hamming distance to obtain a comparison result;

[0031] The data storage module is used to determine that the document content corresponding to the fingerprint value is not repeated when the comparison result meets the preset conditions, and store the corresponding metadata in the storage system.

[0032] According to a third aspect of the present invention, a device is provided, comprising a memory and a processor, wherein the processor is configured to implement any step of the multi-source heterogeneous data storage method described in the first aspect when executing a computer management program stored in the memory.

[0033] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer management program is stored. When the computer management program is executed by a processor, the steps of any multi-source heterogeneous data storage method in the first aspect are implemented.

[0034] The present invention provides a multi-source heterogeneous data storage method, system, device and storage medium. The method includes: obtaining multiple document contents in different formats from multiple data sources based on multiple metadata in a PgSQL database, and obtaining multiple feature values ​​corresponding to each document content; calculating each feature value based on a Simhash text similarity algorithm and a TF-IDF term frequency-inverse document frequency weighted algorithm to obtain a fingerprint value corresponding to each document content; obtaining a fingerprint value set of existing documents, performing a Hamming distance comparison on the fingerprint value corresponding to each document content with each fingerprint value in the fingerprint value set to obtain a comparison result; when the comparison result meets a preset condition, determining that the document content corresponding to the fingerprint value is not repeated, and storing the corresponding metadata in a storage system. The method of the present invention extracts feature values ​​from document contents of different formats in multiple data sources, so that the multi-source heterogeneous document contents can be processed according to a unified feature processing method. The fingerprint value corresponding to each document is obtained by utilizing the Simhash text similarity algorithm and the TF-IDF word frequency-inverse document frequency weighted algorithm, so that the fingerprint value of each document obtained through its multiple feature values ​​is more consistent with expectations, thereby reducing the complexity and accuracy of fingerprint value calculation, and comparing the fingerprint value of each document with the fingerprint values ​​of existing documents to obtain non-repeated document contents, and then storing the non-repeated document contents, so that the multi-source heterogeneous document contents can be quickly deduplicated and stored in batches, which greatly reduces the complexity of document deduplication and the computing power requirements of the server, and improves the storage efficiency of multi-source heterogeneous documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flow chart of a multi-source heterogeneous data storage method provided by the present invention;

[0036] Figure 2 This is a comparison chart of the accuracy of the improved Simhash algorithm of the present invention and the original Simhash algorithm;

[0037] Figure 3 This is a comparison chart of the recall rates of the improved Simhash algorithm of the present invention and the original Simhash algorithm;

[0038] Figure 4 This is a comparison chart of the execution time of the improved Simhash algorithm of the present invention and the original Simhash algorithm;

[0039] Figure 5A schematic diagram of the structure of a multi-source heterogeneous data storage system provided by the present invention;

[0040] Figure 6 A schematic diagram of the hardware structure of a possible device provided by the present invention;

[0041] Figure 7 A schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0042] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0043] Figure 1 The present invention provides a flow chart of a data storage method based on multi-source heterogeneity, such as Figure 1 As shown, the method includes:

[0044] Step S100: obtaining multiple document contents in different formats from multiple data sources according to multiple metadata in the PgSQL database, and obtaining multiple feature values ​​corresponding to each document content;

[0045] It should be noted that the execution subject of the method of this embodiment can be a computer terminal device with data processing, network communication, and program execution functions, such as a computer, tablet computer, etc.; it can also be a server device with the same or similar functions, or a cloud server with similar functions, and this embodiment does not limit this. For ease of understanding, this embodiment and the following embodiments will be described using a server device as an example.

[0046] It is understandable that the above-mentioned PgSQL database is an object-relational database system that can be used to store metadata. In the method of this embodiment, other object-relational databases can also be used to replace the PgSQL database, and this embodiment does not limit this.

[0047] It should be understood that the metadata may be data used to describe the basic information of the document content and the corresponding storage information, such as document name, document number, document source, document type, document storage address, etc.

[0048] It should also be noted that each of the above document contents has corresponding multiple characteristic values, and the characteristic values ​​can be key fields of each document, such as: title, author, signing organization and other fields.

[0049] It can also be understood that the above-mentioned multiple document contents are documents to be stored that need to be deduplicated.

[0050] It should also be understood that the multiple document contents in different formats in the above-mentioned multiple data sources may refer to the different data sources and formats of the above-mentioned multiple documents, such as Chinese journal articles, foreign language conference papers, Chinese degree theses and other documents.

[0051] In a specific implementation, multiple metadata are obtained from a PgSQL database, corresponding document contents are obtained through the multiple metadata, and multiple feature values ​​corresponding to each document content are extracted.

[0052] Step S200: Calculate each feature value according to the Simhash text similarity algorithm and the TF-IDF term frequency-inverse document frequency weighted algorithm to obtain the fingerprint value corresponding to each document content;

[0053] In the specific implementation, each eigenvalue of each document content is calculated using the Simhash text similarity algorithm and the TF-IDF term frequency-inverse document frequency weighted algorithm to obtain the feature fingerprint value corresponding to each eigenvalue. The feature fingerprint values ​​of each document content are then merged, accumulated, and reduced in dimension to obtain the fingerprint value corresponding to each document content.

[0054] Step S300: obtaining a fingerprint value set of existing documents, performing a Hamming distance comparison between the fingerprint value corresponding to each document content and each fingerprint value in the fingerprint value set, and obtaining a comparison result;

[0055] It should be noted that the fingerprint value set of the existing documents can be the fingerprint values ​​of all documents stored in a designated location. The designated location can be local to the server or in a distributed storage system, which is not limited in this embodiment. To further improve data reading efficiency, the fingerprint values ​​of the existing documents can be stored in a distributed memory storage called Redis.

[0056] It can be understood that the Hamming distance mentioned above refers to the number of bits that differ in the corresponding bits of two legal codes in information encoding, which is called the code distance, also known as the Hamming distance. The number of bits that differ in the corresponding values ​​of two codewords is called the Hamming distance between the two codewords. For example, starting from the first bit, the first, fourth, and fifth bits of 10101 and 00110 differ, respectively, so the Hamming distance is 3.

[0057] It should be understood that the above Hamming distance comparison specifically involves setting the fingerprint value of the document content to be stored to newhash and the fingerprint value of the existing document to redishash; wherein:

[0058] newhash=x1、x2……x i , Simhash=y1、y2……y i ;

[0059] The calculation formula of Hamming distance is:

[0060]

[0061] In a specific implementation, the fingerprint value corresponding to each document content is compared with each fingerprint value in the fingerprint value set of existing documents by Hamming distance, and the Hamming distance between each document content in the above multiple document contents and the fingerprint value of each document in the existing document set is obtained.

[0062] Step S400: When the comparison result meets a preset condition, it is determined that the document content corresponding to the fingerprint value is not repeated, and the corresponding metadata is stored in a storage system.

[0063] It should be noted that the above preset conditions can be set according to actual needs, and the preset conditions can be: when the fingerprint value of each document in the above multiple document contents and the fingerprint value of each document in the existing document set are both less than 7, it is determined that the document contents corresponding to the above fingerprint values ​​are not repeated.

[0064] It is understandable that, based on the deficiencies in the background technology, an embodiment of the present invention proposes a data storage method based on multi-source heterogeneity. The method includes: obtaining multiple document contents in different formats from multiple data sources based on multiple metadata in a PgSQL database, and obtaining multiple feature values ​​corresponding to each document content; calculating each feature value based on the Simhash text similarity algorithm and the TF-IDF term frequency-inverse document frequency weighted algorithm to obtain a fingerprint value corresponding to each document content; obtaining a set of fingerprint values ​​of existing documents, performing a Hamming distance comparison on the fingerprint value corresponding to each document content with each fingerprint value in the above fingerprint value set, and obtaining a comparison result; when the above comparison result meets a preset condition, determining that the document content corresponding to the above fingerprint value is not repeated, and storing the corresponding metadata in a storage system. The method of the present invention extracts feature values ​​from document contents of different formats in multiple data sources, so that the multi-source heterogeneous document contents can be processed according to a unified feature processing method. The fingerprint value corresponding to each document is obtained by utilizing the Simhash text similarity algorithm and the TF-IDF word frequency-inverse document frequency weighted algorithm, so that the fingerprint value of each document obtained through its multiple feature values ​​is more consistent with expectations, thereby reducing the complexity and accuracy of fingerprint value calculation, and comparing the fingerprint value of each document with the fingerprint values ​​of existing documents to obtain non-repeated document contents, and then storing the non-repeated document contents, so that the multi-source heterogeneous document contents can be quickly deduplicated and stored in batches, which greatly reduces the complexity of document deduplication and the computing power requirements of the server, and improves the storage efficiency of multi-source heterogeneous documents.

[0065] In a possible embodiment, the step of obtaining multiple document contents in different formats from multiple data sources based on multiple metadata in a PgSQL database, and obtaining multiple feature values ​​corresponding to each document content, includes:

[0066] Step S101: Acquire multiple metadata in a PgSQL database;

[0067] Step S102: Acquire multiple document contents in different formats from multiple data sources according to the multiple metadata;

[0068] Step S103: extract key fields in each document content as its corresponding multiple feature values.

[0069] In an embodiment of the present invention, by extracting key fields from multiple document contents with different formats in multiple data sources, the multi-source heterogeneous document contents can be uniformly matched, which greatly reduces the complexity of data processing and improves the accuracy of data processing.

[0070] In one possible embodiment, the step of calculating each feature value according to the Simhash text similarity algorithm and the TF-IDF term frequency-inverse document frequency weighted algorithm to obtain a fingerprint value corresponding to each document content includes:

[0071] Step S201: according to the word frequency and inverse document frequency corresponding to each eigenvalue, a TF-IDF word frequency-inverse document frequency weighted algorithm is used to obtain a weight value corresponding to each eigenvalue;

[0072] Step S202: Calculate each feature value of each document content according to the Simhash text similarity algorithm and the weight value to obtain a fingerprint value corresponding to each document content.

[0073] In the embodiment of the present invention, the weight value of each eigenvalue is optimized by using the TF-IDF term frequency-inverse document frequency weighted algorithm, and the fingerprint value of each eigenvalue of each document content is calculated according to the Simhash text similarity algorithm and the optimized weight value, so that the calculated fingerprint value can more accurately reflect the characteristics of each eigenvalue, greatly improving the accuracy of subsequent duplicate data identification.

[0074] In a possible embodiment, the step of obtaining a weight value corresponding to each eigenvalue using a TF-IDF word frequency-inverse document frequency weighted algorithm based on the word frequency and inverse document frequency corresponding to each eigenvalue further includes:

[0075] Step S2011: Obtain weight values ​​corresponding to multiple feature values ​​corresponding to each document content;

[0076] Step S2012: performing linear fitting on the weight values ​​corresponding to the plurality of eigenvalues ​​according to Zipf's law to obtain a fitting function;

[0077] Step S2013: Optimizing the weight values ​​corresponding to the plurality of eigenvalues ​​according to the fitting function to obtain optimized weight values ​​corresponding to the plurality of eigenvalues.

[0078] In an embodiment of the present invention, by further performing linear fitting on the weight values ​​of multiple eigenvalues ​​of each document content, the weight value corresponding to each eigenvalue is further optimized, so that the weight value of each eigenvalue can more accurately reflect each feature, thereby further improving the accuracy of subsequent duplicate data identification.

[0079] In a possible embodiment, the step of performing a Hamming distance comparison on the fingerprint value corresponding to each document content with each fingerprint value in the fingerprint value set to obtain a comparison result includes:

[0080] Step S301: using a sliding window algorithm, perform a Hamming distance comparison between the fingerprint value corresponding to each document content and each fingerprint value in the fingerprint value set, and obtain a set of compared fingerprint values ​​in the fingerprint value set that meets a difference threshold.

[0081] It should be noted that the above sliding window algorithm can obtain the platform and specific time where the document first appears based on the document type. The document types include monographs, journals, books, newspapers, conference documents, scientific reports, standard documents, patent documents, dissertations and government publications. The current document platforms usually do not cover all documents. For example, patent documents usually use the State Intellectual Property Office as the main publication platform. Therefore, the documents belonging to each document platform are distinguished by file blocking in the distributed memory, which facilitates the sliding window algorithm to identify and read.

[0082] It can be understood that the algorithm of the above sliding window is specifically as follows: according to the document type, obtain the platform and time where the document first appears, and use the time as a node or the platform and time as a node. Specifically, according to the document type, traverse the platform and time where the document first appears in the distributed memory. There is a situation where the document cannot find the platform where it first appears or appears on multiple platforms at the same time. If this is the case, use the time as a node and advance the preset time period forward, and select the fingerprint value within the time period for matching. In this embodiment, the preset time period is 5 years. For example, if the time when a document first appears is 20 For example, if the platform and time of the first appearance of a document can be found in the distributed memory, the platform and time are combined as a node, and the platform and time of the first appearance of the document are advanced a certain number of years. The fingerprint values ​​of the document metadata of other platforms of the same type are read for matching. For example, if the platform on which a document first appears is EI and the time of the first appearance is 2010, all the document fingerprint values ​​between 2005 and 2010 in the distributed memory are traversed except for the EI platform.

[0083] It should be understood that the difference threshold may be a threshold obtained from experiments or a threshold range, and the compliance difference threshold may be a fingerprint value whose Hamming distance is less than the threshold or whose Hamming distance is within the threshold range.

[0084] The embodiment of the present invention uses a sliding window algorithm to match the fingerprint values ​​of documents, combines the time dimension and the space dimension, and thus greatly improves the matching efficiency of the fingerprint values, while also reducing the computing pressure of the server.

[0085] In a possible embodiment, when the comparison result satisfies a preset condition, the step of determining that the document content corresponding to the fingerprint value is non-repetitive includes:

[0086] Step S302: When the number of elements in the comparison fingerprint value set is equal to 0, it is determined that the fingerprint values ​​of the document contents corresponding to the comparison fingerprint value set are not repeated.

[0087] It should be noted that the number of elements in the above-mentioned comparison fingerprint value set can be greater than 1, equal to 1 or equal to 0; when the number of the above-mentioned elements is greater than 1, manual intervention can be performed to determine whether the fingerprint value of the document content corresponding to the comparison fingerprint value set is repeated with the fingerprint value of the existing document; when the above-mentioned element is equal to 1, it can be determined that the fingerprint value of the document content corresponding to the comparison fingerprint value set is repeated with the document in the document to be stored; when the number of the above-mentioned elements is equal to 0, it can be determined that the fingerprint value of the document content corresponding to the comparison fingerprint value set is not repeated.

[0088] In a specific implementation, if the fingerprint values ​​of the document content corresponding to the compared fingerprint value set are determined to be non-duplicate, the metadata corresponding to the document content and the document itself are stored in a distributed storage system. Each node in this distributed storage system consists of two layers of storage: internal storage and disk. For metadata operations, the operation type and operation time are also stored in the distributed storage system for each update (CURD) operation. After a metadata entry is updated and stored, the metadata entry is cleared from PgSQL, freeing up storage space and reducing server computing pressure.

[0089] In a possible application scenario, to further illustrate the performance and effect improvements of the embodiments of the present invention, this embodiment also provides a test environment and test effect analysis. In the above application scenario, the operating environment of the system corresponding to the method of the present invention is:

[0090] 1. Physical environment

[0091] CPU: i7 16 cores, memory: 64G, the IP addresses of the three test machines are: 192.168.21.106, 192.168.21.107, 192.168.21.108.

[0092] 2. Network environment

[0093] Clickhouse (Click Stream Data WareHouse column storage database) is deployed on 192.168.21.106, and Redis is deployed on 192.168.21.107.

[0094] 3. Storage method and address of original data set

[0095] The original dataset is stored in the avro file format in the 192.168.21.108 / data / base_data directory. After processing, the dataset is currently stored in the ztdb_base database of ClickHouse.

[0096] 4. Result dataset storage method and address

[0097] The resulting dataset is stored in the Redis db1 table. Duplicate datasets found by the algorithm are backed up in the Dulp_Data table in the ClickHouse data_2020 database. The Hamming distance calculation results are stored in the HMD_data table in the ClickHouse data_2020 database.

[0098] The above application scenarios also include indicator values, where the numerical indicator terms are explained as follows: TP: True Postive, that is, the judgment is correct and this record is a duplicate; TN: True Negative, that is, the judgment is correct and this record is not a duplicate; FP: False Postive, that is, the judgment is incorrect, and the record corresponding to the target simhash value is not a duplicate, but it is judged to be a duplicate; this record may actually be a non-duplicate or a duplicate of another record; FN: False Negative, that is, the judgment is incorrect, and the record corresponding to the target simhash value is a duplicate, but it is judged to be a non-duplicate.

[0099] The above application scenario also includes three test indicators: deduplication rate, precision rate, and recall rate, among which:

[0100] Accuracy: Accuracy refers to the ratio of the number of correctly classified samples to the total number of samples. In this experiment, it is predicted to be the ratio of the number of duplicate documents to the total number of documents, also known as the deduplication rate.

[0101] Precision: Precision refers to the ratio of the number of correctly classified samples to the total number of classified samples. In this experiment, it is the ratio of the number of correctly predicted duplicate documents to the number of predicted duplicate documents, also known as the precision rate.

[0102] Recall: Recall refers to the ratio of the number of correctly classified samples to the number of known samples. In this experiment, it is the ratio of the number of correctly predicted duplicate documents to the number of known duplicate documents, also known as the recall rate.

[0103] The formulas for the accuracy, precision, and recall of the results are defined as follows:

[0104]

[0105]

[0106]

[0107] The test of the above application scenario also includes a description of the dataset: the dataset year is 2020. The test results are as follows: the total number of documents predicted by the algorithm of the embodiment of the present invention is 225,277. Among the documents predicted to be duplicates, a total of 142,950 documents with a Hamming distance less than or equal to 2 are selected. After checking the preset rules, the number of correctly predicted documents is 142,950, that is, the TP is 142,926, and the calculated FP is 24. Among the documents predicted to be non-duplicates, a total of 82,327 documents with a Hamming distance greater than 2 are selected. After checking the rules, 7,924 documents are duplicated, that is, the FN is 7,924, and the calculated TN is 74,403.

[0108] According to the above results, the results of this experiment are: accuracy (deduplication rate) is 96.47%, precision (check rate) is 99.98%, and recall (check rate) is 94.75%.

[0109] In the test of the above application scenario, we can also test Chinese documents to obtain the deduplication effect of Chinese document data. The total amount of Chinese document data in the experiment is 2347285.

[0110] In the above-mentioned Chinese literature test experiment, the test results obtained are as follows: the total number of duplicate documents predicted by the algorithm of the embodiment of the present invention is 2347285, and the number of documents with a Hamming distance less than or equal to 2 selected from the documents predicted to be duplicate is 297898. Among them, the number of documents correctly predicted through the preset rule check is 295716, that is, the TP is 295716, and the FP is calculated to be 2182. The total number of documents with a Hamming distance greater than 2 selected from the documents predicted to be non-duplicate is 2049387. According to the preset rule check, 46037 documents are duplicated, that is, the FN is 46307, and the TN is calculated to be 2003350.

[0111] In the Chinese literature test in the above application scenario, the results of this experiment calculated based on the above results are: accuracy (deduplication rate) is 97.95%, precision (checking rate) is 99.27%, and recall (checking rate) is 86.46%.

[0112] The above-mentioned preset rule check may be performed manually.

[0113] In order to further illustrate the improvement of performance and effect in the test results of this embodiment, the present invention also provides a comparison between the improved simhash algorithm of this application and the original simhash algorithm in the five keyword fields of Internet, education, AI, medical care, and housing, including the comparison of accuracy, recall rate and execution time. Figure 2 、 Figure 3 and Figure 4 ;according to Figure 2 、 Figure 3 and Figure 4 It can be clearly seen that the multi-source heterogeneous data storage method provided by the present application has a significant improvement in accuracy and recall rate compared to the existing Simhash algorithm, and has a significant decrease in execution time. Therefore, the multi-source heterogeneous data storage method provided in the embodiment of the present application can quickly realize batch deduplication and storage of multi-source heterogeneous document content, greatly reducing the complexity of document deduplication and the computing power requirements of the server, and improving the storage efficiency of multi-source heterogeneous documents.

[0114] See also Figure 5 , Figure 5 A schematic diagram of a multi-source heterogeneous data storage system structure provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, a multi-source heterogeneous data storage system includes a feature acquisition module 100, a fingerprint calculation module 200, a fingerprint comparison module 300 and a data storage module 400, wherein:

[0115] The feature acquisition module 100 is used to obtain multiple document contents in different formats from multiple data sources based on multiple metadata in the PgSQL database, and obtain multiple feature values ​​corresponding to each document content; the fingerprint calculation module 200 is used to calculate each feature value based on the Simhash text similarity algorithm and the TF-IDF term frequency-inverse document frequency weighted algorithm to obtain a fingerprint value corresponding to each document content; the fingerprint comparison module 300 is used to obtain a fingerprint value set of existing documents, perform a Hamming distance comparison on the fingerprint value corresponding to each document content with each fingerprint value in the fingerprint value set, and obtain a comparison result; the data storage module 400 is used to determine that the document content corresponding to the fingerprint value is not repeated when the comparison result meets the preset conditions, and store the corresponding metadata in the storage system.

[0116] It can be understood that the data storage system based on multi-source heterogeneity provided by the present invention corresponds to the data storage method based on multi-source heterogeneity provided in the aforementioned embodiments. The relevant technical features of the data storage system based on multi-source heterogeneity can refer to the relevant technical features of the data storage method based on multi-source heterogeneity, which will not be repeated here.

[0117] See also Figure 6 , Figure 6 Schematic diagram of an embodiment of the device provided by the embodiment of the present invention. Figure 6 As shown, an embodiment of the present invention provides a device, including a memory 1310, a processor 1320, and a computer program 1311 stored in the memory 1310 and executable on the processor 1320. When the processor 1320 executes the computer program 1311, the following steps are implemented:

[0118] Based on multiple metadata in a PgSQL database, multiple document contents in different formats are obtained from multiple data sources, and multiple feature values ​​corresponding to each document content are obtained; each feature value is calculated based on the Simhash text similarity algorithm and the TF-IDF term frequency-inverse document frequency weighted algorithm to obtain a fingerprint value corresponding to each document content; a fingerprint value set of existing documents is obtained, and the fingerprint value corresponding to each document content is compared with each fingerprint value in the above fingerprint value set by Hamming distance to obtain a comparison result; when the above comparison result meets the preset conditions, it is determined that the document content corresponding to the above fingerprint value is not repeated, and its corresponding metadata is stored in a storage system.

[0119] See also Figure 7 , Figure 7 Schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. Figure 7 As shown, this embodiment provides a computer-readable storage medium 1400 on which a computer program 1411 is stored. When the computer program 1411 is executed by a processor, the following steps are implemented:

[0120] Based on multiple metadata in a PgSQL database, multiple document contents in different formats are obtained from multiple data sources, and multiple feature values ​​corresponding to each document content are obtained; each feature value is calculated based on the Simhash text similarity algorithm and the TF-IDF term frequency-inverse document frequency weighted algorithm to obtain a fingerprint value corresponding to each document content; a fingerprint value set of existing documents is obtained, and the fingerprint value corresponding to each document content is compared with each fingerprint value in the above fingerprint value set by Hamming distance to obtain a comparison result; when the above comparison result meets the preset conditions, it is determined that the document content corresponding to the above fingerprint value is not repeated, and its corresponding metadata is stored in a storage system.

[0121] The present invention provides a multi-source heterogeneous data storage method, system, device and storage medium. The method includes: obtaining multiple document contents in different formats from multiple data sources based on multiple metadata in a PgSQL database, and obtaining multiple feature values ​​corresponding to each document content; calculating each feature value based on a Simhash text similarity algorithm and a TF-IDF term frequency-inverse document frequency weighted algorithm to obtain a fingerprint value corresponding to each document content; obtaining a fingerprint value set of existing documents, performing a Hamming distance comparison on the fingerprint value corresponding to each document content with each fingerprint value in the fingerprint value set to obtain a comparison result; when the comparison result meets a preset condition, determining that the document content corresponding to the fingerprint value is not repeated, and storing the corresponding metadata in a storage system. The method of the present invention extracts feature values ​​from document contents of different formats in multiple data sources, so that the multi-source heterogeneous document contents can be processed according to a unified feature processing method. The fingerprint value corresponding to each document is obtained by utilizing the Simhash text similarity algorithm and the TF-IDF word frequency-inverse document frequency weighted algorithm, so that the fingerprint value of each document obtained through its multiple feature values ​​is more consistent with expectations, thereby reducing the complexity and accuracy of fingerprint value calculation, and comparing the fingerprint value of each document with the fingerprint values ​​of existing documents to obtain non-repeated document contents, and then storing the non-repeated document contents, so that the multi-source heterogeneous document contents can be quickly deduplicated and stored in batches, which greatly reduces the complexity of document deduplication and the computing power requirements of the server, and improves the storage efficiency of multi-source heterogeneous documents.

[0122] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0123] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0127] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0128] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A data storage method based on multi-source heterogeneity, characterized in that: The method comprises: Obtain multiple document contents in different formats from multiple data sources based on multiple metadata in a PgSQL database, and obtain multiple feature values ​​corresponding to each document content; Each feature value is calculated using the Simhash text similarity algorithm and the TF-IDF term frequency-inverse document frequency weighted algorithm to obtain the fingerprint value corresponding to each document content; Obtain a set of fingerprint values ​​of existing documents, perform Hamming distance comparison between the fingerprint value corresponding to each document content and each fingerprint value in the fingerprint value set, and obtain a comparison result; When the comparison result satisfies a preset condition, determining that the document content corresponding to the fingerprint value is not repeated, and storing the corresponding metadata in a storage system; The step of calculating each feature value according to the Simhash text similarity algorithm and the TF-IDF term frequency-inverse document frequency weighted algorithm to obtain the fingerprint value corresponding to each document content includes: According to the word frequency and inverse document frequency corresponding to each eigenvalue, the TF-IDF word frequency-inverse document frequency weighted algorithm is used to obtain the weight value corresponding to each eigenvalue; Calculate each feature value of each document content according to the Simhash text similarity algorithm and the weight value to obtain a fingerprint value corresponding to each document content; The step of obtaining a weight value corresponding to each eigenvalue using a TF-IDF word frequency-inverse document frequency weighted algorithm based on the word frequency and inverse document frequency corresponding to each eigenvalue further includes: Obtain weight values ​​corresponding to multiple feature values ​​corresponding to each document content; Performing linear fitting on the weight values ​​corresponding to the multiple eigenvalues ​​according to Zipf's law to obtain a fitting function; The weight values ​​corresponding to the multiple eigenvalues ​​are optimized according to the fitting function to obtain the optimized weight values ​​corresponding to the multiple eigenvalues.

2. The multi-source heterogeneous data storage method according to claim 1, characterized in that: The step of obtaining multiple document contents in different formats from multiple data sources based on multiple metadata in the PgSQL database, and obtaining multiple feature values ​​corresponding to each document content, includes: Get multiple metadata in the PgSQL database; Acquire multiple document contents in different formats from multiple data sources according to the multiple metadata; The key fields in each document content are extracted as its corresponding multiple feature values.

3. The multi-source heterogeneous data storage method according to claim 2, characterized in that: The key fields include at least: title, author, and signing organization.

4. The multi-source heterogeneous data storage method according to claim 1, characterized in that: The step of performing Hamming distance comparison on the fingerprint value corresponding to each document content and each fingerprint value in the fingerprint value set to obtain a comparison result includes: The sliding window algorithm is used to perform Hamming distance comparison between the fingerprint value corresponding to each document content and each fingerprint value in the fingerprint value set, and a comparison fingerprint value set in the fingerprint value set that meets the difference threshold is obtained.

5. The multi-source heterogeneous data storage method according to claim 4, characterized in that: When the comparison result satisfies a preset condition, the step of determining that the document content corresponding to the fingerprint value is not repeated includes: When the number of elements in the comparison fingerprint value set is equal to 0, it is determined that the fingerprint values ​​of the document contents corresponding to the comparison fingerprint value set are not repeated.

6. A data storage system based on multi-source heterogeneity, characterized in that: include A feature acquisition module is used to acquire multiple document contents in different formats from multiple data sources based on multiple metadata in a PgSQL database, and to acquire multiple feature values ​​corresponding to each document content; Fingerprint calculation module, used to calculate each feature value based on Simhash text similarity algorithm and TF-IDF word frequency-inverse document frequency weighted algorithm to obtain the fingerprint value corresponding to each document content; The fingerprint comparison module is used to obtain a fingerprint value set of existing documents, and compare the fingerprint value corresponding to each document content with each fingerprint value in the fingerprint value set by Hamming distance to obtain a comparison result; A data storage module is used to determine that the document content corresponding to the fingerprint value is not repeated when the comparison result meets the preset conditions, and store the corresponding metadata in the storage system; The fingerprint calculation module calculates each feature value according to the Simhash text similarity algorithm and the TF-IDF term frequency-inverse document frequency weighted algorithm to obtain the fingerprint value corresponding to each document content, including: According to the word frequency and inverse document frequency corresponding to each eigenvalue, the TF-IDF word frequency-inverse document frequency weighted algorithm is used to obtain the weight value corresponding to each eigenvalue; Calculate each feature value of each document content according to the Simhash text similarity algorithm and the weight value to obtain a fingerprint value corresponding to each document content; The step of obtaining a weight value corresponding to each eigenvalue using a TF-IDF word frequency-inverse document frequency weighted algorithm based on the word frequency and inverse document frequency corresponding to each eigenvalue further includes: Obtain weight values ​​corresponding to multiple feature values ​​corresponding to each document content; Performing linear fitting on the weight values ​​corresponding to the multiple eigenvalues ​​according to Zipf's law to obtain a fitting function; The weight values ​​corresponding to the multiple eigenvalues ​​are optimized according to the fitting function to obtain the optimized weight values ​​corresponding to the multiple eigenvalues.

7. A device, characterized in that It includes a memory and a processor, and the processor is used to implement the steps of the multi-source heterogeneous data storage method according to any one of claims 1 to 5 when executing the computer management program stored in the memory.

8. A computer-readable storage medium, characterized in that A computer management program is stored thereon, and when the computer management program is executed by a processor, the steps of the multi-source heterogeneous data storage method according to any one of claims 1 to 5 are implemented.

Images