Data duplication checking method, device and electronic equipment
By calculating the correlation between the metadata of scientific data and entity data, duplicate data is screened out, which improves the efficiency and accuracy of scientific data to check the diversion efficiency and solves the problem of low-quality diversion efficiency of scientific data in the prior art.
Patent Information
- Application Number
- CN202210050995.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-17
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-01-17
AI Technical Summary
Scientific data is prone to repeated submissions when submitting, resulting in database redundancy, reducing data retrieval efficiency, and low efficiency and accuracy of existing plagiarism checking methods.
By obtaining the metadata and entity data of the data set to be re-checked, the correlation between metadata is calculated using the topic content similarity algorithm, the candidate data sets that need to be re-checked are selected, and the binary code or summary encoding of the entity data is used. Calculate the correlation between entity data and determine the sludge check result.
It improves the efficiency and accuracy of scientific data to check severity, reduces the amount of data that needs to be checked again, reduces the amount of calculation, and ensures the accuracy of checking severity.
Smart Images

Figure CN114547233B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data processing technology, and in particular to a data duplication checking method, device and electronic equipment. Background Art
[0002] With the rapid development of information technologies such as the Internet, big data, and artificial intelligence, a large amount of scientific data has been generated in the process of scientific research. Scientific data is the original data generated by researchers through observation, experiment, computational analysis, and other scientific research processes.
[0003] These scientific data containing scientific theories are often buried in scattered laboratories. In order to deeply explore the value of scientific data, researchers will package the data into scientific data sets and submit them to the data center for use by all researchers. When submitting data sets, there may be repeated submissions of data sets, which increases the redundancy of the database and reduces the efficiency of data retrieval. Since the amount of scientific data sets is very large, it is impossible to check for duplicates through manual review methods, so a more efficient method is needed. Summary of the invention
[0004] The present invention provides a data duplication checking method, device and electronic equipment, which are used to solve the defect of low duplication checking efficiency for scientific data in the prior art and improve the duplication checking efficiency and accuracy of scientific data.
[0005] In a first aspect, the present invention provides a data duplication checking method, comprising:
[0006] Obtaining metadata and entity data of the data set to be checked for duplicates, wherein the metadata is used to describe the scientific data contained in the data set to be checked for duplicates, and the entity data is the scientific data contained in the data set to be checked for duplicates;
[0007] Calculating a first correlation between the to-be-checked duplicate data set and each data set in the database based on the metadata, the first correlation being a correlation between the metadata of the to-be-checked duplicate data set and the metadata of the data sets in the database;
[0008] Screening out candidate data sets in the database that need to be checked for duplicates again according to the first relevance;
[0009] Based on the entity data of the to-be-checked duplicate data set, calculating a second correlation between the to-be-checked duplicate data set and the candidate data set, the second correlation being a correlation between the entity data of the to-be-checked duplicate data set and the entity data of the candidate data set;
[0010] The duplicate checking result of the to-be-checked duplicate data set is determined according to the first relevance and the second relevance.
[0011] According to an exemplary embodiment provided by the present invention, the step of screening out the candidate data set in the database that needs to be checked for duplicates again according to the first relevance includes:
[0012] determining whether the first correlation is within a preset range, and taking the data sets in the database whose first correlation is within the preset range as the candidate data sets;
[0013] The data set in the database whose first correlation is greater than the preset range is used as the duplicate checking result of the data set to be checked for duplicates.
[0014] According to an exemplary embodiment provided by the present invention, the calculating, based on the metadata, a first correlation between the to-be-checked duplicate data set and each data set in a database includes:
[0015] Construct a topic content similarity algorithm;
[0016] The first relevance between the metadata and the metadata of each data set in the database is calculated by the subject content similarity algorithm.
[0017] According to an exemplary embodiment provided by the present invention, the subject content similarity algorithm includes:
[0018] The metadata of the to-be-checked duplicate data set is divided into a first semantic group and a first statistical group by using a language knowledge base, and
[0019] dividing the metadata of the data set in the database into a second semantic group and a second statistical group by using a language knowledge base;
[0020] Calculating the similarity between the first semantic group and the second semantic group;
[0021] Calculating the matching probability of the words in the first statistical group and the words in the second statistical group that match each other;
[0022] A first correlation between the to-be-checked duplicate data set and a data set in the database is determined according to the sum of the similarities and the matching probability.
[0023] According to an exemplary embodiment provided by the present invention, the metadata of the to-be-checked duplicate data set includes one or more types of data including title, keywords, and abstract.
[0024] According to an exemplary embodiment provided by the present invention, the value of the preset range is greater than 0.6 and less than 1.
[0025] According to an exemplary embodiment provided by the present invention, the calculating, based on the entity data of the to-be-checked duplicate data set, the second correlation between the to-be-checked duplicate data set and the candidate data set includes:
[0026] Obtaining a first summary code of the entity data of the to-be-checked duplicate data set and a second summary code of the entity data of the candidate data set;
[0027] The first summary code is matched with the second summary code to determine a second correlation between the to-be-checked duplicate data set and the candidate data set.
[0028] According to an exemplary embodiment provided by the present invention, the calculating, based on the entity data of the to-be-checked duplicate data set, the second correlation between the to-be-checked duplicate data set and the candidate data set includes:
[0029] Obtaining a first binary code of the entity data of the to-be-checked duplicate data set and a second binary code of the entity data of the candidate data set;
[0030] The first binary code is matched with the second binary code to determine a second correlation between the to-be-checked duplicate data set and the candidate data set.
[0031] According to an exemplary embodiment provided by the present invention, the calculating, based on the entity data of the to-be-checked duplicate data set, the second correlation between the to-be-checked duplicate data set and the candidate data set includes:
[0032] Determine the data format of the entity data of the to-be-checked duplicate data set and the entity data of the candidate data set;
[0033] Reading the entity data of the to-be-checked duplicate data set according to the data format to obtain first reading information, and
[0034] Read the entity data of the candidate data set according to the data format to obtain second read information;
[0035] The first read information and the second read information corresponding to the same data format are matched to determine a second correlation between the to-be-checked duplicate data set and the candidate data set.
[0036] In a second aspect, the present invention further provides a data duplication checking device, comprising:
[0037] A data acquisition module, used to acquire metadata and entity data of the data set to be checked for duplicates, wherein the metadata is used to describe the scientific data contained in the data set to be checked for duplicates, and the entity data is the scientific data contained in the data set to be checked for duplicates;
[0038] A metadata comparison module, used to calculate a first correlation between the to-be-checked duplicate data set and each data set in the database based on the metadata, wherein the first correlation is a correlation between the metadata of the to-be-checked duplicate data set and the metadata of the data sets in the database;
[0039] A data screening module, used for screening out candidate data sets in the database that need to be checked for duplicates again according to the first relevance;
[0040] An entity data comparison module, configured to calculate a second correlation between the to-be-checked duplicate data set and the candidate data set based on the entity data of the to-be-checked duplicate data set, wherein the second correlation is a correlation between the entity data of the to-be-checked duplicate data set and the entity data of the candidate data set;
[0041] A result determination module is used to determine the duplication checking result of the to-be-checked duplication data set according to the first relevance and the second relevance.
[0042] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described data duplication checking methods are implemented.
[0043] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described data duplication checking methods.
[0044] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned data duplication checking methods.
[0045] The data duplication checking method, device and electronic device provided by the present invention first check for duplicates based on the metadata of the scientific data when checking for duplicates on scientific data. Compared with the scientific data itself, the metadata has a greatly reduced amount of data, which can improve the speed of checking for duplicates, thereby quickly screening out a portion of duplicate data. Then, the entity data of the scientific data, that is, the scientific data itself, is checked for duplicates. Using the scientific data itself to check for duplicates can ensure the accuracy of the check for duplicates. In addition, by using the metadata to screen out a portion of the data, the amount of scientific data that needs to be repeated can be reduced, thereby reducing the amount of calculation, and further improving the efficiency of checking for duplicates on the basis of ensuring the accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0047] Figure 1This is one of the flow charts of the data duplication checking method provided by the present invention;
[0048] Figure 2 This is the second flow chart of the data duplication checking method provided by the present invention;
[0049] Figure 3 This is the third flow chart of the data duplication checking method provided by the present invention
[0050] Figure 4 This is the fourth flow chart of the data duplication checking method provided by the present invention;
[0051] Figure 5 This is the fifth flow chart of the data duplication checking method provided by the present invention;
[0052] Figure 6 This is the sixth flow chart of the data duplication checking method provided by the present invention;
[0053] Figure 7 This is the seventh flow chart of the data duplication checking method provided by the present invention;
[0054] Figure 8 It is a structural schematic diagram of the data duplication checking device provided by the present invention;
[0055] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0057] Scientific data usually consists of two parts: one is the metadata information describing the scientific data, such as data representation information, data classification information, instrument platform information, etc. The other part is the entity data file, which contains the available entity data and data description documents.
[0058] Related technologies mainly check data for duplicate content by calculating the similarity of text. Since scientific data sets contain not only text data, but also data resources in various formats, such as tables, graphics, etc., the above method cannot take into account the entire content of the scientific data set when checking for duplicate content, resulting in low accuracy of checking for duplicate content.
[0059] Another way to check for duplicate content is to use statistical methods to calculate relevance. However, this method does not take into account the semantics between words, phrases, and sentences, resulting in low accuracy in checking for duplicate content in scientific data.
[0060] Based on this, the present invention first provides a data duplication checking method. The method can be executed by electronic devices such as personal computers, tablet computers, mobile phones, or servers, and the present invention does not specifically limit the form of the electronic device.
[0061] Combine the following Figure 1-Figure 7 The technical solution of the data duplication checking method of the present invention is described.
[0062] like Figure 1 As shown, the data duplication checking method may include the following steps:
[0063] Step S10: Obtain metadata and entity data of the data set to be checked for duplicates, wherein the metadata is used to describe the scientific data contained in the data set to be checked for duplicates, and the entity data is the scientific data contained in the data set to be checked for duplicates.
[0064] In this embodiment, a data set refers to a collection of scientific data. After obtaining scientific data during research, researchers can package the same type of scientific data into a scientific data set. The types of scientific data in different fields are different, such as geographic data, meteorological data, biological data, etc. This embodiment does not specifically limit the type of scientific data. The data set to be checked for duplicates can be multiple data sets. The scientific data in the same data set belong to the same type.
[0065] The metadata of the dataset to be checked for duplicates is data that describes the scientific dataset, such as the data type, data size, data meaning, etc. of the dataset. The more detailed the description of the dataset, the more metadata the dataset has. Exemplarily, the metadata of the dataset to be checked for duplicates may include representation information, classification information, instrument information, platform information, etc. of the scientific data. In order to accurately and efficiently reflect the content of the scientific data, the title, keywords, and abstract in the metadata of the dataset to be checked for duplicates can be obtained for duplicate checking. The entity data of the dataset to be checked for duplicates refers to the scientific data itself in the dataset, such as geographic data, meteorological data, etc. The same dataset to be checked for duplicates may include entity data in multiple data formats, such as graphics, text, tables, etc.
[0066] For example, a scientific data set that a researcher is preparing to submit to a data center can be used as a data set to be checked for duplicates. This embodiment can provide a graphical user interface, which may include an interface for uploading data. Through this interface, a researcher can upload a data set to be checked for duplicates that needs to be checked for duplicates. After obtaining the data set to be checked for duplicates uploaded by the researcher through this interface, the metadata and entity data of the data set to be checked for duplicates can be obtained.
[0067] Step S20: Calculate a first correlation between the to-be-checked duplicate data set and a data set in a database based on the metadata, wherein the first correlation is a correlation between the metadata of the to-be-checked duplicate data set and the metadata of the data set in the database.
[0068] The database refers to a system for storing scientific data. A large number of data sets can be stored in the database. The metadata of each data set in the database is obtained, and the first correlation between the metadata of the data set to be checked for duplicates and the metadata of each data set in the database is calculated in turn.
[0069] In an exemplary embodiment, the metadata includes a title, an abstract, and keywords. A title can be a sentence; an abstract can be a combination of multiple sentences; and keywords are a set of words or phrases. Exemplarily, the similarity between two words can be determined through a language knowledge base. A language knowledge base is a knowledge system that describes the relationship between words, such as HowNet, WordNet, etc. The language knowledge base defines the relationship between words, such as synonym relationships, antonym relationships, hyponymy relationships, and so on. Each relationship can correspond to a distance value, such as a distance of 1 for synonyms and a distance of 3 for antonyms. The farther the distance between two words, the lower the similarity. Exemplarily, a statistical method can also determine the similarity between two words. The statistical method determines the similarity between two words by counting the probability of the same words appearing in two texts.
[0070] For example, when calculating the first relevance between the to-be-checked duplicate data set and a data set (e.g., the first data set) in the database, the title, abstract, and keywords in the metadata of the two data sets can be obtained. Keywords contain one or more words; the corresponding words in the title and abstract can be obtained by word segmentation. Then the method based on the language knowledge base can calculate the keyword semantic relevance, title semantic relevance, and abstract semantic relevance between the keywords of the to-be-checked duplicate data set and the keywords of the first data set. Combining these three parts can determine the first relevance between the to-be-checked duplicate data set and the metadata of the first data set. For example, the result of adding the keyword semantic relevance, title semantic relevance, and abstract semantic relevance is used as the first relevance; for another example, the result of multiplying the keyword semantic relevance, title semantic relevance, and abstract semantic relevance is used as the first relevance, and so on.
[0071] In an exemplary embodiment, in order to calculate the first correlation between metadata of a data set, an algorithm may be constructed first, which is recorded as a subject content similarity algorithm. The subject content similarity algorithm may be used to calculate the correlation between metadata of scientific data. Scientific data metadata may include some professional terms. In order to prevent these professional terms from not being included in the language knowledge base, the subject content similarity algorithm may also be constructed in combination with a similarity calculation method based on a statistical method. Specifically, the subject content similarity algorithm includes the following steps:
[0072] Step S21: dividing the metadata of the to-be-checked duplicate data set into a first semantic group and a first statistical group through a language knowledge base, and dividing the metadata of the data set in the database into a second semantic group and a second statistical group through a language knowledge base.
[0073] The metadata of the data set to be checked for duplicates, such as the title, is segmented by a word segmentation algorithm to obtain multiple words corresponding to the metadata of the data set to be checked for duplicates. For the convenience of distinction, the metadata of the data set to be checked for duplicates is recorded as metadata A, and the metadata of any data set in the database is recorded as metadata B. Each word after the segmentation of metadata A is matched with the language database respectively. If a certain word, such as word a, can match the word defined in the language database, then the word a is divided into the first semantic group. Similarly, if the words in metadata B match the words in the language knowledge base, the words in metadata B that match the language database are divided into the second semantic group. If the words contained in metadata A cannot match the words defined in the language database, the words in metadata A that cannot match the language database are divided into the first statistical group. Similarly, the words in metadata B that cannot match the language database are divided into the second statistical group.
[0074] Step S22: Calculate the similarity between the words in the first semantic group and the words in the second semantic group.
[0075] The words in the first semantic group and the second semantic group can all be queried through the language knowledge base, so the similarity between the first semantic group and the second semantic group can be calculated by a text similarity algorithm based on the language knowledge base. Specifically, all the words in the first semantic group are regarded as a text, and the words in the second semantic group are also regarded as a text. The text similarity algorithm can segment and encode the text, convert the text into a vector, and then calculate the similarity between the two vectors. Exemplarily, the text similarity algorithm may include a similarity algorithm based on cosine distance, a Jaccard similarity algorithm, and the like, which is not particularly limited in this embodiment.
[0076] Step S23: Calculate the matching probability of the matching words in the first statistical group and the second statistical group.
[0077] The matching probability of the first statistical group matching the second statistical group refers to the proportion of the matching words in the first statistical group. For example, the first statistical group contains 10 words, and 4 words match the words in the second statistical group, then the matching probability is 0.4. That is to say, the matching probability can be calculated by dividing the number of matching words in the first statistical group and the second statistical group by the total number of words contained in the first statistical group. Exemplarily, a word b in the first statistical group is compared with each word in the second statistical group to determine whether it matches. If the word b is the same as a word in the second statistical group, the number of matching words in the first statistical group and the second statistical group is increased by 1. Each word in the first statistical group is compared with the words in the second statistical group respectively. After completion, the number of matching words in the first statistical group and the second statistical group can be obtained, and then divided by the total number of words in the first statistical group to obtain the matching probability.
[0078] Step S24: determining a first relevance of the to-be-checked duplicate data set according to the similarity and the matching probability.
[0079] Exemplarily, the similarity and matching probability of the to-be-checked duplicate data set and the candidate data set are added, and the result obtained can be used as the correlation between the to-be-checked duplicate data set and the metadata of a data set in the database, that is, the first correlation. Alternatively, a weight is determined for each of the similarity and the matching probability, and the weighted sum is calculated as the first correlation; the similarity and the matching probability are multiplied as the first correlation, etc.
[0080] In summary, the relevance between the metadata of the data sets can be calculated according to the subject content similarity algorithm in steps S21 to S24. In the subject content similarity algorithm of the present invention, for the words in the metadata of the data set to be checked for duplicates that exist in the semantic knowledge base, the similarity can be calculated based on the knowledge base similarity algorithm. For some professional terms in the metadata that are not included in the knowledge base, the similarity is calculated based on a statistical method. It can be seen that the subject content similarity algorithm of this embodiment can more perfectly determine the correlation between scientific data, thereby improving the accuracy of the correlation between scientific data and the accuracy of scientific data duplication checking.
[0081] Furthermore, metadata may include titles, abstracts, and keywords. The above-mentioned subject content similarity algorithm can be used to calculate the title relevance between the titles of the two data sets, the abstract relevance between the abstracts, and the keyword relevance between the keywords. Then, the overall relevance between the metadata of the two data sets, i.e., the first relevance, is obtained by combining the title relevance, abstract relevance, and keyword relevance. In this way, by measuring the similarity between data sets from the three aspects of the title, abstract, and keywords of the scientific data set, it is possible to more comprehensively determine whether the data sets are repeated, thereby improving the accuracy of duplicate checking.
[0082] In an exemplary embodiment, Figure 3 A method for calculating the first correlation between a dataset to be checked for duplicates and a dataset in a database is shown. Figure 3 As shown, first obtain a metadata information table, which refers to a list of metadata. The metadata information table includes a metadata information table of the data set to be checked for duplicates and a metadata information table of the data set in the database. In step S31, extract the title, keyword, and summary information. Extract the title, keyword, and summary information of the data set to be checked for duplicates from the metadata information table of the data set to be checked for duplicates; and extract the title, keyword, and summary information from the metadata information table of the data set in the database. In step S32, calculate the similarity (i.e., the first correlation) between the metadata of the data set to be checked for duplicates and the data set in the database. Exemplarily, a subject content similarity algorithm can be constructed in advance, and the subject content similarity algorithm is such as steps S21 to S24, which will not be repeated here. In step S33, obtain the sorted similarity result. Specifically, after calculating the similarity between the metadata by the subject content similarity algorithm, it can be sorted according to the similarity size to obtain the sorted similarity result. Alternatively, the sorted similarity result can be displayed as a duplicate checking result for reference by users who need to check for duplicates.
[0083] Next, continue to refer to Figure 1 , step S30: filtering out candidate data sets in the database that need to be checked for duplicates again according to the first relevance.
[0084] Exemplarily, the minimum and maximum values of the first correlation are first determined. For example, the minimum value of the first correlation is 0, and the maximum value is 1. If the first correlation between the data set to be checked for duplicates and a data set X in the database is 1, or is greater than a specific threshold value such as 0.7, 0.8, etc., it can be determined that the data set to be checked for duplicates and the data in the data set X are duplicates. Then the data set X and the first correlation between the data set X and the data set to be checked for duplicates can be saved in the duplicate checking result. If the first correlation between the data set to be checked for duplicates and the data set X in the database is 0, or is less than a certain threshold value, such as 0.1, 0.2, etc., it can be determined that there is no duplicate data between the data set to be checked for duplicates and the data set X. Then the data set X can be excluded.
[0085] In an exemplary embodiment, according to the maximum and minimum values of the value of the first correlation, a preset range between the maximum and minimum values can be determined. For example, the interval of values of the first correlation greater than 0.6 and less than 1 is used as the preset range, or the interval of values of the first correlation greater than 0.5 and less than 0.9 is used as the preset range, and so on. As can be seen from the above, when the first correlation is greater than the preset range, it can be determined that the corresponding data set is repeated with the data set to be checked for duplicates. For example, the preset range is greater than 0.6 and less than 1. If the first correlation of a data set is 1, the data set is repeated with the data in the data set to be checked for duplicates. Therefore, a data set greater than the above preset range can be used as the duplicate checking result of the data set to be checked for duplicates. If the value of the first correlation between a data set in the database and the data set to be checked for duplicates is within the preset range, the data set is used as a candidate data set for the data to be checked for duplicates.
[0086] Step S40: Based on the entity data of the to-be-checked duplicate data set, a second correlation between the to-be-checked duplicate data set and the candidate data set is calculated, where the second correlation is the correlation between the entity data of the to-be-checked duplicate data set and the entity data of the candidate data set.
[0087] Exemplarily, the similarity between the entity data of the to-be-checked duplicate data set and the entity data of the candidate data set can also be calculated by a text similarity algorithm as the second correlation between the to-be-checked duplicate data set and the candidate data set. Since the amount of entity data is large, in order to improve the calculation efficiency, the second correlation of the entity data can be determined by summary coding. Specifically, a first summary coding of the entity data of the to-be-checked duplicate data set and a second summary coding of the entity data of the candidate data set are obtained; the first summary coding is matched with the second summary coding to determine the second correlation between the to-be-checked duplicate data set and the candidate data set.
[0088] The summary code refers to the result obtained by encoding the entity data through a summary algorithm such as a hash algorithm. The hash algorithm can calculate the entity data of each data set to obtain the corresponding summary code (MD5 code). Exemplarily, the entity data of the data to be checked for duplicates is first encoded by the hash algorithm to obtain the corresponding first summary code; then the entity data of each candidate data set is encoded by the hash algorithm to obtain the second summary code of each candidate data set. Compare the first summary code of the data to be checked for duplicates with the second summary code of each candidate data set to determine whether the first summary code is the same as the second summary code. The MD5 codes obtained for exactly the same entity data are the same. Therefore, when the first summary code is the same as the second summary code, the second correlation between the data set to be checked for duplicates and the corresponding candidate data set can be 1.
[0089] Figure 4 A method for calculating the second correlation is shown. Figure 4 As shown, in step S41, a data set folder is obtained. The folder may include multiple files, which are the entity data of the data set. The folder of the data set to be checked for duplicates and the folder of the data set in the database are obtained respectively. In step S42, the MD5 code of the entity data is obtained. The MD5 code of each file of the data set to be checked for duplicates and the MD5 code of each file of the data set in the database are calculated respectively by a hash algorithm, so as to obtain the MD5 code of the entity data. In step S43, the MD5 code of the entity data is stored in the form of a key-value pair. The key-value pair contains two values: the file name of the file and the MD5 code corresponding to the file, such as [file name: MD5 code]. The entity data of the data set to be checked for duplicates and the entity data of the data set in the database can be stored separately, for example, in two different files. Alternatively, each data set can be distinguished by the file name, that is, the file of the data set to be checked for duplicates and the file of the data set in the database have different file names. In step S44, each key-value pair is traversed to compare whether there is a key-value pair with the same MD5 code. In step S45, the number of files and file names with the same MD5 code are obtained. The number of files and file names with the same MD5 code finally obtained can be used as the result of the duplicate checking, and the result is displayed to the user so that the user can check the data set according to the result.
[0090] In an exemplary embodiment, the second correlation between entity data can also be determined by comparing the binary codes of the entity data. Specifically, the first binary code of the entity data of the to-be-checked duplicate data set and the second binary code of the entity data of the candidate data set are obtained; the first binary code is matched with the second binary code to determine the second correlation between the to-be-checked duplicate data set and the candidate data set.
[0091] Specifically, Figure 5As shown. In step S51, a data set folder is obtained. In step S52, a binary code of the entity data is obtained. The binary code is a code composed of two basic characters "0" and "1". In a computer, data is represented in only two forms, 0 and 1. By encoding all data, a binary code corresponding to each entity data can be obtained. Exemplarily, encoding the entity data of the duplicate data set to be checked can obtain a first binary code, and encoding the entity data of the candidate data set can obtain a second binary code. In step S53, the binary code is stored in the form of a key-value pair. For example, the key-value pair is [file name: binary code]. Then in step S54, all key-value pairs are traversed to determine whether there are repeated binary codes. The binary codes in each key-value pair are compared with each other to determine whether there are repeated binary codes. Exemplarily, the first binary code in the key-value pair corresponding to the duplicate data set to be checked is matched with the second binary code corresponding to the database. The first binary code and the second binary code can both be regarded as characters, and can be matched by a string matching algorithm to determine whether the first binary code and the second binary code are the same. If the first binary code is the same as the second binary code, the second correlation between the to-be-checked duplicate data set and the candidate data set is 1. In step S55, the file name and the number of files with the same binary code are obtained. And the corresponding data set can be determined according to the file name, so as to obtain the name of the duplicate data set and the number of data sets. Exemplarily, the file name and the number of files corresponding to the duplicate binary code, or the name and the number of duplicate data sets can be displayed for user reference.
[0092] In an exemplary embodiment, the data format of the entity data of the data set to be checked for duplicates may be different from the data format of the candidate data set. The correlation between the entity data of the data set to be checked for duplicates and the entity data of the candidate data may be determined, and the entity data of the same data format may be matched. Specifically, firstly, the data formats of the entity data of the data set to be checked for duplicates and the entity data of the candidate data set are determined; the entity data of the data set to be checked for duplicates is read according to the data format to obtain first read information, and the entity data of the candidate data set is read according to the data format to obtain second read information; then the first read information corresponding to the same data format is matched with the second read information to determine the second correlation between the data set to be checked for duplicates and the candidate data set.
[0093] like Figure 6As shown, in step S61, format judgment is performed. Data in different data formats need to be read in different ways. For example, text data needs to be obtained in TXT and XLS formats; image data needs to be read in TIF format, and image file header information, directory information, directory item information, etc. can be read; graphics can be read in shp format, and its point, line, surface and other attribute information can be read. First, the data format of the entity data of the data set to be checked for duplicates can be determined, and each data format can be matched with the entity data in turn until the correct data format is matched. The correct data format can read the correct information of the entity data. In step S62, the entity data is read according to the data format. Specifically, after determining the data format of the entity data of the data set to be checked for duplicates, the corresponding entity data is read according to the data format, and the obtained read information is recorded as the first read information. Similarly, the data format of the entity data of the candidate data set is determined, and then the entity data of the candidate data set is read according to the correct data format, and the information obtained by reading is recorded as the second read information. In step S63, the read information is stored in the form of a key-value pair, such as [file name: read information]. The file name in the key-value pair is the name of the file corresponding to the entity data.
[0094] Then, in step S64, the key-value pairs of the same data format are compared. According to all the stored key-value pairs, the data format corresponding to each key-value pair is determined, and then the key-value pairs of the same data format are compared to determine whether the read information in the key-value pairs is the same. Exemplarily, for the first read information of the data to be checked for duplicates, the second read information with the same data format as the first read information is determined, and then the first read information with the same data format is matched with the second read information, and the matching result is the second correlation. For example, if the entity data of the data to be checked for duplicates is text information, the text information is compared with the candidate data set whose entity data is also text information, and it is determined that 50% of the information in the two text information is the same, then the second correlation can be 50%. In step S65, the file name and the number of files with the same read information are obtained. If the read information in two key-value pairs is the same, the two files corresponding to the two key-value pairs are repeated. After comparing all the key-value pairs, all repeated file names and file numbers can be obtained.
[0095] Step S50: determining a duplicate checking result of the to-be-checked duplicate data set according to the first relevance and the second relevance.
[0096] The duplicate checking result may include the name of the data set that is repeated with the data set to be checked for duplicates, the number of data sets, or may also include repeated file names, the number of files, etc. Exemplarily, the name of the data set whose first correlation with the data set to be checked for duplicates is 1, and the number of data sets are used as the duplicate checking result. Alternatively, the name of the file (entity data) and the number of files that meet the conditions with the second correlation with the data set to be checked for duplicates are used as the duplicate checking result. For example, if the second correlation is 1, the second correlation meets the conditions, or if the second correlation is greater than 0.6, etc., the second correlation meets the conditions. In addition, the above condition can also be that the second correlation is greater than 0.7, 0.8, etc., and this embodiment does not make any special limitation to this.
[0097] For example, the duplicate checking results can be displayed for researchers' reference. When researchers obtain duplicate files and the number of files, they can browse the entity data of the corresponding files to provide reference for scientific research. In addition, after browsing the duplicate files, researchers can choose not to upload the data set to be checked for duplicates, thereby avoiding the redundancy of scientific data in the database, which is conducive to improving the retrieval efficiency of scientific data.
[0098] In an exemplary embodiment, the data duplication checking method of the present invention may include: Figure 7 The steps shown are as follows. Figure 7, in step S71, extract the title, keywords, and summary information from the metadata of the data set. In this embodiment, the data set includes the data set to be checked for duplicate content and all data sets in the database. In step S72, the similarity between the metadata is calculated by the subject content similarity algorithm. The title, keywords, and summary information of the data set are taken as a text, and the similarity (i.e., the first correlation) between the text of the data set to be checked for duplicate content and each other data set is calculated. In step S73, the similarity is determined. Exemplarily, the value interval of the similarity is [0, 1]. If the similarity corresponding to the data set to be checked for duplicate content and the other data sets is within a preset range, the corresponding data set is a candidate data set with a higher similarity to the data set to be checked for duplicate content. The preset range can be (0.6, 1). If the similarity is [0, 0.6], the data set to be checked for duplicate content is not similar to the data set, that is, there is no data set that is repeated with the data set to be checked for duplicate content. If the similarity is 1, the data sets are similar, that is, there is a data set that is repeated with the data set to be checked for duplicate content. The data set with a similarity of 1 to the data set to be checked for duplicate content is used as a repeated data set. In step S74, the entity data of the candidate data set is obtained. In step S75, the similarity between the candidate data set and the data set to be checked for duplicates is determined by the binary code of the entity data. Alternatively, in step S75, the similarity between the data set to be checked for duplicates and the candidate data set can also be determined by the read information of the entity data. Alternatively, the similarity between the data set to be checked for duplicates and the candidate data set is determined by the MD5 code of the entity data. In step S76, the number of duplicate files and the file names are output. The files that are duplicated with the data set to be checked for duplicates include the files whose similarity meets the conditions in step S73, and the files whose similarity meets the conditions in step S75. These duplicate files can be displayed to users or researchers for review by users, which is conducive to reducing data redundancy.
[0099] In the above implementation, the metadata of the data set is first compared, and some duplicate data sets and data sets with a certain similarity are queried according to the interval with a large first correlation credibility. Then, a deep query is performed on the candidate data sets with a certain similarity, and the duplicate data sets are determined by further comparison through the entity data, which can improve the accuracy of the duplicate check.
[0100] In addition, the similarity between the entity data of the data set to be checked for duplicates and the candidate data set can also be determined by other methods. For example, the similarity is first determined according to the MD5 code, and a part of the duplicate data set is screened out from the candidate data set according to the similarity. For example, the data set with the same MD5 code can be used as the duplicate data set of the data set to be checked for duplicates and recorded in the duplicate checking result. Then the remaining data sets in the candidate data set are re-determined in the form of binary code, and the data set with the same binary code as the entity data of the data set to be checked for duplicates is screened out again and added to the duplicate checking result. Finally, the data format of the remaining data set after screening is used to determine the similarity, and the data set with the same data format as the entity data of the data set to be checked for duplicates and the same reading information is screened out, added to the duplicate checking result, and the final duplicate checking result is obtained. In this embodiment, each time the similarity of the entity data is calculated once by a similarity calculation method, the candidate data set is screened once, and the candidate data set that needs to be calculated is less and less, and the calculation speed is faster and faster, which is conducive to improving the efficiency of duplicate checking. Alternatively, the above three methods of calculating the similarity of entity data are adopted, and the similarity is calculated three times for the candidate data set, and the results of the three calculations are used to determine whether the candidate data set is a duplicate data set. For example, if the entity data of the candidate data set L and the data set to be checked for duplicates have the same MD5 code, the same binary code, and the same read information, then the candidate data set L is treated as a duplicate data set and added to the duplicate checking results.
[0101] The present invention also provides a data duplication checking device, which can be used to execute the above data duplication checking method. The data duplication checking device provided by the present invention is described below.
[0102] like Figure 8 As shown, the data duplication checking device 80 may include a data acquisition module 81 , a metadata comparison module 82 , a data screening module 83 , an entity data comparison module 84 and a result determination module 85 .
[0103] Specifically, the data acquisition module 81 is used to acquire metadata and entity data of the data set to be checked for duplicates, wherein the metadata is used to describe the scientific data contained in the data set to be checked for duplicates, and the entity data is the scientific data contained in the data set to be checked for duplicates.
[0104] The metadata comparison module 82 is used to calculate the first correlation between the to-be-checked duplicate data set and each data set in the database based on the metadata, wherein the first correlation is the correlation between the metadata of the to-be-checked duplicate data set and the metadata of the data sets in the database.
[0105] The data screening module 83 is used to screen out candidate data sets in the database that need to be checked for duplicates again according to the first relevance.
[0106] The entity data comparison module 84 is used to calculate the second correlation between the to-be-checked duplicate data set and the candidate data set based on the entity data of the to-be-checked duplicate data set, wherein the second correlation is the correlation between the entity data of the to-be-checked duplicate data set and the entity data of the candidate data set.
[0107] The result determination module 85 is used to determine the duplication checking result of the to-be-checked duplication data set according to the first relevance and the second relevance.
[0108] In an exemplary embodiment of the present invention, the data duplication checking device 80 further includes a judgment module and a result output module. The judgment module is used to determine whether the first relevance is within a preset range, and use the data set in the database whose first relevance is within the preset range as the candidate data set. The result output module is used to use the data set in the database whose first relevance is greater than the preset range as the duplication checking result of the data set to be checked for duplication.
[0109] In an exemplary embodiment of the present invention, the metadata comparison module 82 includes an algorithm construction module and an algorithm application module. The algorithm construction module is used to construct a subject content similarity algorithm based on the language knowledge base. The algorithm application module is used to calculate the first relevance between the metadata and the metadata of each data set in the database through the subject content similarity algorithm.
[0110] In an exemplary embodiment of the present invention, the algorithm construction module specifically includes a grouping module, a similarity calculation module, a statistics module and a relevance determination module.
[0111] Among them, the grouping module can be used to divide the metadata of the to-be-checked duplicate data set into a first semantic group and a first statistical group through a language knowledge base, and divide the metadata of the data set in the database into a second semantic group and a second statistical group through a language knowledge base. The similarity calculation module can be used to calculate the similarity between the first semantic group and the second semantic group. The statistical module can be used to calculate the matching probability of the matching words in the first statistical group and the second statistical group. The relevance determination module can be used to determine the first relevance of the to-be-checked duplicate data set and a data set in the database based on the sum of the similarities and the matching probability.
[0112] In an exemplary embodiment of the present invention, the metadata of the to-be-checked duplicate data set includes one or more types of data including title, keywords, and abstract.
[0113] In an exemplary embodiment of the present invention, the value of the preset range is greater than 0.6 and less than 1.
[0114] In an exemplary embodiment of the present invention, the entity data comparison module 84 may include a summary code comparison module, which is used to obtain a first summary code of the entity data of the data set to be checked for duplicates, and a second summary code of the entity data of the candidate data set; match the first summary code with the second summary code to determine the second correlation between the data set to be checked for duplicates and the candidate data set.
[0115] In an exemplary embodiment of the present invention, the entity data comparison module 84 may include a binary code comparison module. The binary code comparison module is used to obtain a first binary code of the entity data of the to-be-checked duplicate data set and a second binary code of the entity data of the candidate data set; match the first binary code with the second binary code to determine a second correlation between the to-be-checked duplicate data set and the candidate data set.
[0116] In an exemplary embodiment of the present invention, the entity data comparison module 84 may include a data format determination module, a data reading module, and a read information comparison module. Among them, the data format determination module is used to determine the data format of the entity data of the to-be-checked duplicate data set and the entity data of the candidate data set. The data reading module is used to read the entity data of the to-be-checked duplicate data set according to the data format to obtain the first read information, and read the entity data of the candidate data set according to the data format to obtain the second read information. The read information comparison module can be used to match the first read information corresponding to the same data format with the second read information to determine the second correlation between the to-be-checked duplicate data set and the candidate data set.
[0117] Since the various functional modules of the data duplication checking device of the exemplary embodiment of the present invention correspond to the steps of the exemplary implementation method of the above-mentioned data duplication checking device, for details not disclosed in the device embodiment of the present invention, please refer to the implementation method of the above-mentioned data duplication checking device of the present invention.
[0118] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9As shown, the electronic device may include: a processor (processor) 910, a communication interface (Communications Interface) 920, a memory (memory) 930 and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call the logic instructions in the memory 930 to execute the above-mentioned data duplication checking device method, which includes: step S10, obtaining metadata and entity data of the data set to be checked for duplication, wherein the metadata is used to describe the scientific data contained in the data set to be checked for duplication, and the entity data is the scientific data contained in the data set to be checked for duplication; step S20, calculating a first correlation between the data set to be checked for duplication and each data set in the database based on the metadata, wherein the first correlation is the correlation between the metadata of the data set to be checked for duplication and the metadata of the data set in the database; step S30, screening out candidate data sets in the database that need to be checked for duplication again according to the first correlation; step S40, calculating a second correlation between the data set to be checked for duplication and the candidate data set based on the entity data of the data set to be checked for duplication, wherein the second correlation is the correlation between the entity data of the data set to be checked for duplication and the entity data of the candidate data set; step S50, determining the duplication checking result of the data set to be checked for duplication according to the first correlation and the second correlation.
[0119] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0120] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data duplication checking device method provided by the above methods, and the method includes: step S10, obtaining metadata and entity data of the data set to be checked for duplication, wherein the metadata is used to describe the scientific data contained in the data set to be checked for duplication, and the entity data is the scientific data contained in the data set to be checked for duplication; step S20, calculating a first correlation between the data set to be checked for duplication and each data set in a database based on the metadata, wherein the first correlation is the correlation between the metadata of the data set to be checked for duplication and the metadata of the data set in the database; step S30, screening out candidate data sets in the database that need to be checked for duplication again according to the first correlation; step S40, calculating a second correlation between the data set to be checked for duplication and the candidate data set based on the entity data of the data set to be checked for duplication, wherein the second correlation is the correlation between the entity data of the data set to be checked for duplication and the entity data of the candidate data set; step S50, determining the duplication checking result of the data set to be checked for duplication according to the first correlation and the second correlation.
[0121] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented by a processor to execute the data duplication checking device method provided by the above methods when the computer program is executed, the method comprising: step S10, obtaining metadata and entity data of the data set to be checked for duplication, wherein the metadata is used to describe the scientific data contained in the data set to be checked for duplication, and the entity data is the scientific data contained in the data set to be checked for duplication; step S20, calculating a first correlation between the data set to be checked for duplication and each data set in a database based on the metadata, the first correlation being the correlation between the metadata of the data set to be checked for duplication and the metadata of the data set in the database; step S30, screening out candidate data sets in the database that need to be checked for duplication again according to the first correlation; step S40, calculating a second correlation between the data set to be checked for duplication and the candidate data set based on the entity data of the data set to be checked for duplication, the second correlation being the correlation between the entity data of the data set to be checked for duplication and the entity data of the candidate data set; step S50, determining the duplication checking result of the data set to be checked for duplication according to the first correlation and the second correlation.
[0122] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0123] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A data duplication checking method, characterized in that: include: Obtaining metadata and entity data of the data set to be checked for duplicates, wherein the metadata is used to describe the scientific data contained in the data set to be checked for duplicates, and the entity data is the scientific data contained in the data set to be checked for duplicates; Calculating a first correlation between the to-be-checked duplicate data set and each data set in the database based on the metadata, the first correlation being a correlation between the metadata of the to-be-checked duplicate data set and the metadata of the data sets in the database; Screening out candidate data sets in the database that need to be checked for duplicates again according to the first relevance; Based on the entity data of the to-be-checked duplicate data set, calculating a second correlation between the to-be-checked duplicate data set and the candidate data set, the second correlation being a correlation between the entity data of the to-be-checked duplicate data set and the entity data of the candidate data set; Determine a duplicate checking result of the to-be-checked duplicate data set according to the first relevance and the second relevance; The calculating, based on the metadata, a first correlation between the to-be-checked duplicate data set and each data set in the database includes: Construct a topic content similarity algorithm; Calculating a first correlation between the metadata and the metadata of each data set in the database by using the subject content similarity algorithm; The topic content similarity algorithm includes: The metadata of the to-be-checked duplicate data set is divided into a first semantic group and a first statistical group by using a language knowledge base, and dividing the metadata of the data set in the database into a second semantic group and a second statistical group by using a language knowledge base; Calculating the similarity between the first semantic group and the second semantic group; Calculating the matching probability of the words in the first statistical group and the words in the second statistical group that match each other; A first correlation between the to-be-checked duplicate data set and a data set in the database is determined according to the sum of the similarities and the matching probability.
2. The data duplication checking method according to claim 1, characterized in that: The step of screening out the candidate data set in the database that needs to be checked for duplicates again according to the first relevance includes: determining whether the first correlation is within a preset range, and taking the data sets in the database whose first correlation is within the preset range as the candidate data sets; The data set in the database whose first correlation is greater than the preset range is used as the duplicate checking result of the data set to be checked for duplicates.
3. The data duplication checking method according to any one of claims 1 to 2, characterized in that: The metadata of the to-be-checked duplicate data set includes one or more types of data including title, keywords, and abstract.
4. The data duplication checking method according to claim 2, characterized in that: The value of the preset range is greater than 0.6 and less than 1.
5. The data duplication checking method according to claim 1, characterized in that: The calculating, based on the entity data of the to-be-checked duplicate data set, a second correlation between the to-be-checked duplicate data set and the candidate data set comprises: Obtaining a first summary code of the entity data of the to-be-checked duplicate data set and a second summary code of the entity data of the candidate data set; The first summary code is matched with the second summary code to determine a second correlation between the to-be-checked duplicate data set and the candidate data set.
6. The data duplication checking method according to claim 1, characterized in that: The calculating, based on the entity data of the to-be-checked duplicate data set, a second correlation between the to-be-checked duplicate data set and the candidate data set comprises: Obtaining a first binary code of the entity data of the to-be-checked duplicate data set and a second binary code of the entity data of the candidate data set; The first binary code is matched with the second binary code to determine a second correlation between the to-be-checked duplicate data set and the candidate data set.
7. The data duplication checking method according to claim 1, characterized in that: The calculating, based on the entity data of the to-be-checked duplicate data set, a second correlation between the to-be-checked duplicate data set and the candidate data set comprises: Determine the data format of the entity data of the to-be-checked duplicate data set and the entity data of the candidate data set; Reading the entity data of the to-be-checked duplicate data set according to the data format to obtain first reading information, and Read the entity data of the candidate data set according to the data format to obtain second read information; The first read information and the second read information corresponding to the same data format are matched to determine a second correlation between the to-be-checked duplicate data set and the candidate data set.
8. A data duplication checking device for implementing the data duplication checking method according to claim 1, characterized in that: include: A data acquisition module, used to acquire metadata and entity data of the data set to be checked for duplicates, wherein the metadata is used to describe the scientific data contained in the data set to be checked for duplicates, and the entity data is the scientific data contained in the data set to be checked for duplicates; A metadata comparison module, used to calculate a first correlation between the to-be-checked duplicate data set and each data set in the database based on the metadata, wherein the first correlation is a correlation between the metadata of the to-be-checked duplicate data set and the metadata of the data sets in the database; A data screening module, used for screening out candidate data sets in the database that need to be checked for duplicates again according to the first relevance; An entity data comparison module, configured to calculate a second correlation between the to-be-checked duplicate data set and the candidate data set based on the entity data of the to-be-checked duplicate data set, wherein the second correlation is a correlation between the entity data of the to-be-checked duplicate data set and the entity data of the candidate data set; A result determination module is used to determine the duplication checking result of the to-be-checked duplication data set according to the first relevance and the second relevance.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the data duplication checking method as described in any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data duplication checking method as described in any one of claims 1 to 7 is implemented.
11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data duplication checking method as described in any one of claims 1 to 7 is implemented.
