Structured evidence-based data duplication detection method and device and electronic equipment

Abstract

The present disclosure provides a structured evidence-based data duplication detection method and device and electronic equipment, and relates to the technical field of blockchains. Unlike traditional blockchains, which only focus on data authenticity during evidence storage, the core idea of the present disclosure is to introduce structured data information, a preset data structure, a bidirectional sliding window, and a dynamic step mechanism for data processing to achieve evidence storage and verification. This makes the evidence storage process not only ensure the consistency and integrity of the data, but also accurately reflect the duplication of the verification data and the evidence file. In the electronic evidence mode based on the blockchain, the present disclosure can detect the duplication of large-scale data sets while ensuring the effectiveness and accuracy of data storage, taking into account a lower false positive rate and a higher storage space efficiency, thereby reducing storage costs and significantly expanding the functionality and application range of evidence storage.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of blockchains, and in particular to a data duplication degree detection method and device based on structured evidence and an electronic device. BACKGROUND

[0002] With the rapid development of the information age, a large amount of data is recorded and saved in the form of electronic evidence. At the same time, the blockchain technology, with its unique characteristics of non-tamperability, non-repudiation and multi-party participation, has become an ideal choice in the field of electronic data evidence, making electronic data evidence an important application of blockchain technology.

[0003] A blockchain is a distributed database, and its operation mode requires synchronization and storage of all data on each network node. If a large amount of data is directly chained, it will consume a large amount of storage space and network resources, thereby affecting the performance and efficiency of the blockchain. In addition, considering the openness of the blockchain ledger, directly storing the original text of electronic evidence on the chain may pose a risk of privacy leakage. Therefore, in order to reduce the storage overhead of the blockchain and ensure the privacy of the original data, the common practice in practice is to only store the hash value (hash value) of the electronic evidence on the chain. When verifying the authenticity of the electronic evidence, only the hash value of the electronic evidence to be verified needs to be compared with the hash value stored on the chain, and it can be confirmed whether the electronic evidence has been tampered with.

[0004] However, the traditional hash function is a one-to-one mapping relationship, which means that even a slight change in data will cause a huge change in the hash value. For some application scenarios that need to detect the degree of content change (such as detecting the version difference of a document, the degree of duplication), this feature makes the hash function unable to accurately reflect the degree of data change. SUMMARY

[0005] The present disclosure is proposed in view of the above problems. The present disclosure provides a data duplication degree detection method and device based on structured evidence, and an electronic device, and accordingly provides a computer readable storage medium and a computer program product.

[0006] According to one aspect of the present disclosure, a data duplication degree detection method based on structured evidence is provided, comprising:

[0007] Pre-use bidirectional sliding window to obtain structured data information of the file to be stored;

[0008] Store the structured data information in a preset data structure and complete the chaining;

[0009] When the duplication detection is performed, the structured data information of the file to be detected is obtained by using the bidirectional sliding window, and is matched with the information in the data structure extracted from the blockchain and parsed; and in the matching process, the moving step of the sliding window is dynamically adjusted to match the remaining file content when the non-matching condition occurs.

[0010] After the matching ends, the duplication of the file to be detected is obtained by all matching results.

[0011] In addition, according to the data duplication detection method based on structured evidence according to one aspect of the present disclosure, the matching of the remaining file content when the non-matching condition occurs by dynamically adjusting the moving step of the sliding window comprises:

[0012] When the structured data information of the file to be detected does not match the information in the data structure, the sliding window stops moving;

[0013] After both sliding windows stop, the remaining content between the two sliding windows is continued to be window-slid and matched by using the dynamically decreasing step.

[0014] In addition, according to the data duplication detection method based on structured evidence according to one aspect of the present disclosure, the condition for confirming the end of the matching comprises:

[0015] All available steps that are dynamically adjusted have been used, or the information content of the evidence in the data structure has been compared.

[0016] In addition, according to the data duplication detection method based on structured evidence according to one aspect of the present disclosure, the storing of the structured data information into the preset data structure comprises:

[0017] An array corresponding to the structured data information is constructed by using the data structure, and the array is used as the storage fingerprint of the file to be stored in the data structure.

[0018] In addition, according to the data duplication detection method based on structured evidence according to one aspect of the present disclosure, the obtaining of the duplication of the file to be detected by all matching results comprises:

[0019] The similarity between the file to be detected and the corresponding evidence file is obtained according to the ratio of the successfully matched structured data to the total number of structured data of the corresponding evidence file; or the weights of different structured data of the evidence file are set, and the similarity between the file to be detected and the corresponding evidence file is calculated according to the successful matching result and in combination with the weights.

[0020] In addition, according to the data duplication detection method based on structured evidence according to one aspect of the present disclosure, the way of obtaining the structured data information by using the bidirectional sliding window comprises:

[0021] A sliding window is set at the beginning and the end of the file respectively, the two sliding windows move towards each other based on a preset step length from the start position and the end position of the file, and during the movement of each sliding window, the corresponding hash value is calculated for the file region within the sliding window.

[0022] According to another aspect of the present disclosure, a data duplication detection device based on structured evidence storage is provided, the detection device comprising:

[0023] An evidence storage fragmentation module is configured to obtain structured data information of a file to be stored as evidence by using a bidirectional sliding window in advance.

[0024] A fragmentation information processing module is configured to store the structured data information in a preset data structure and complete the chain-up.

[0025] A file verification module is configured to obtain structured data information of a file to be detected by using a bidirectional sliding window when performing duplication detection, and match the information in the data structure extracted from the blockchain and parsed; and in the matching process, the remaining file content when the matching fails is matched by dynamically adjusting the moving step length of the sliding window.

[0026] A duplication calculation module is configured to obtain the duplication of the file to be detected by all matching results after the matching ends.

[0027] In addition, according to the data duplication detection device based on structured evidence storage according to an aspect of the present disclosure, the file verification module comprises:

[0028] A matching pause unit is configured to stop the movement of the sliding window when the structured data information of the file to be detected does not match the information in the data structure.

[0029] A dynamic step length adjustment unit is configured to continue to execute window sliding and matching on the remaining content between the two sliding windows by using a dynamically decreasing step length after the two sliding windows are stopped.

[0030] In addition, according to the data duplication detection device based on structured evidence storage according to an aspect of the present disclosure, the condition for confirming the end of matching comprises that all available step lengths after dynamic adjustment have been used, or the comparison of the information content stored in the data structure has been completed.

[0031] In addition, according to the data duplication detection device based on structured evidence storage according to an aspect of the present disclosure, the fragmentation information processing module is configured to construct an array corresponding to the structured data information by using the data structure, and use the array as the storage fingerprint of the file to be stored as evidence in the data structure.

[0032] In addition, the structured evidence-based data duplication detection device according to one aspect of the present disclosure, the duplication calculation module comprises:

[0033] The first similarity calculation unit is configured to obtain the similarity between the to-be-detected file and the corresponding evidence file according to the ratio of the successfully matched structured data to the total number of structured data of the corresponding evidence file; or the second similarity calculation unit is configured to set a weight for different structured data of the evidence file, and calculate the similarity between the to-be-detected file and the corresponding evidence file according to the successful matching result and in combination with the weight.

[0034] In addition, the structured evidence-based data duplication detection device according to one aspect of the present disclosure, the structured data information obtaining method using the bidirectional sliding window comprises: setting a sliding window at the beginning and the end of the file respectively, and moving the two sliding windows from the start position and the end position of the file towards each other based on a preset step size; and during the movement of each sliding window, calculating the corresponding hash value for the file region in the sliding window.

[0035] According to still another aspect of the present disclosure, an electronic device is provided, comprising: a memory configured to store computer readable instructions; and a processor configured to execute the computer readable instructions, so that the electronic device performs the structured evidence-based data duplication detection method as described above.

[0036] According to still another aspect of the present disclosure, a non-transitory computer readable storage medium is provided, configured to store computer readable instructions, when the computer readable instructions are executed by a processor, the processor performs the structured evidence-based data duplication detection method as described above.

[0037] According to still another aspect of the present disclosure, a computer program product is provided, when the computer program product is executed by a computer, for executing the method in the first aspect or any possible implementation manner of the first aspect. In one possible design of the fifth aspect, the related programs involved in the product can be stored on the memory packaged with the processor in whole or in part, or on the storage medium not packaged with the processor in part or in whole.

[0038] As will be described in detail below, the structured storage-based data duplication detection method, device, electronic device, computer-readable storage medium and computer program product according to embodiments of the present disclosure differ from the traditional block chain storage in that only data authenticity is concerned. The core idea of the present disclosure is to introduce structured data information, a preset data structure, a bidirectional sliding window and a dynamic step length mechanism for data processing to realize storage and verification, so that the storage process can not only ensure the consistency and integrity of the data, but also accurately reflect the duplication of the verification data and the storage file. In the electronic evidence mode based on the block chain, the present disclosure can detect the duplication of a large-scale data set while ensuring the effectiveness and accuracy of data storage, taking into account a lower false positive rate and a higher storage space efficiency, thereby reducing storage costs and significantly expanding the functionality and application range of storage.

[0039] The present disclosure scheme broadens the functionality of the block chain technology in data storage, so that the block chain storage mechanism has data consistency and integrity, similarity detection function, and can almost cover all fields that need data storage, such as copyright protection, malicious code detection, SQL injection prevention, judiciary, medical treatment, education, supply chain management and other fields, which have potential value and broad market prospects.

[0040] It is to be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further explanation of the subject technology claimed. BRIEF DESCRIPTION OF DRAWINGS

[0041] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:

[0042] Figure 1 FIG. 1 is a schematic diagram illustrating a main process of a structured storage-based data duplication detection method according to an embodiment of the present disclosure.

[0043] Figure 2 FIG. 2 is a schematic diagram illustrating a sliding window matching method based on a dynamic step length according to an embodiment of the present disclosure.

[0044] Figure 3 FIG. 3 is a schematic diagram illustrating a file to be stored according to an embodiment of the present disclosure.

[0045] Figure 4 FIG. 4 is a schematic diagram illustrating a sliding window (sliding block) matching of a file to be detected according to an embodiment of the present disclosure.

[0046] Figure 5 is a functional block diagram illustrating a data duplication detection apparatus based on structured storage evidence according to an embodiment of the present disclosure.

[0047] Figure 6 is a hardware block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0048] Figure 7 is a schematic diagram illustrating a computer-readable storage medium according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present disclosure more obvious, the following will describe the example embodiments according to the present disclosure in detail with reference to the drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the example embodiments described herein.

[0050] First, referring to Figures 1 to 4 a data duplication detection method based on structured storage evidence according to an embodiment of the present disclosure is described. Figure 1 is a main flow schematic diagram illustrating a data duplication detection method based on structured storage evidence according to an embodiment of the present disclosure. Figure 2 is a flow schematic diagram further illustrating a sliding window matching method based on dynamic step length according to an embodiment of the present disclosure. Figure 3 is a schematic diagram further illustrating a file to be stored as evidence according to an embodiment of the present disclosure. Figure 4 is a schematic diagram further illustrating a sliding window matching of a file to be detected according to an embodiment of the present disclosure.

[0051] Referring to Figures 1 to 4 a data duplication detection method based on structured storage evidence according to an embodiment of the present disclosure will be described in detail, including:

[0052] Step S1, pre-use bidirectional sliding window to obtain structured data information of the file to be stored as evidence;

[0053] For example, the file to be stored as evidence (which can be understood as a standard article as a basis for subsequent comparison) can be split and the hash value of each fragment can be calculated by means of a sliding window.

[0054] Specifically, two sliding windows can be set, one of which slides backward from the starting position of the file to be stored as evidence based on a preset step length, and the other also slides forward from the ending position of the file to be stored as evidence based on a preset step length, and during the sliding of each sliding window, the corresponding hash value is calculated for the file region within the sliding window, and when the two sliding windows meet the predetermined condition, the sliding is ended, and finally the hash values corresponding to the multiple text segments of the file to be stored as evidence are obtained.

[0055] Step S2, store the structured data information into the preset data structure and complete the uplink;

[0056] In combination with the application scenario of the text duplication detection task under the premise of low storage overhead concerned in the present disclosure, the data structure can consider adopting a specific structural form for checking whether a certain element is in a set, for example, various filter structures for quickly checking whether a certain element is in a set through a high space efficiency data structure: Bloom filter, Cuckoo filter, quotient filter, etc. For ease of illustration, the Bloom filter will be used as an example hereinafter.

[0057] Specifically, for each shard hash in the foregoing example, the k values corresponding to the hash value can be calculated by using the Bloom filter function to form a Bloom storage array corresponding to the current shard, and the Bloom filter corresponding to each value is set to 1. Thus, the entire file to be stored as evidence is stored in the Bloom filter in a fuzzy hash manner, that is, the storage fingerprints of each shard in the Bloom filter are obtained.

[0058] Then, the Bloom filter itself and its related information (such as the Bloom filter function, false positive rate, etc.) can be stored in the blockchain. It can be supplemented here that in some preferred embodiments, multiple Bloom filters can be used to construct a Bloom filter group or multiple layers of Bloom filters can be used to further increase the accuracy of the duplication detection.

[0059] Step S3, when performing duplication detection, the structured data information of the file to be detected is obtained by using the bidirectional sliding window, and is matched with the information in the data structure extracted from the blockchain and obtained by analysis; and in the matching process, the moving step of the sliding window is dynamically adjusted to match the remaining file content when the matching fails.

[0060] In combination with the foregoing example, the following can be referred to: when it is necessary to verify the measured file, in order to calculate the file duplication (the reverse meaning of which represents the degree of change), the Bloom filter and its related information can be obtained from the blockchain and analyzed, and two opposite sliding windows can be set at the beginning and end of the file to be detected, and the sliding windows start to slide from both ends of the file to be detected. Similarly to the storage stage, during the sliding of each sliding window, the hash value of the file region in the sliding window is calculated; then, the calculated hash value is matched with the storage information parsed from the Bloom filter, and the number of matched shards is accumulated in this process, that is, the degree of change of the file is determined in this way. Figure 2

[0061] ​Step S31, when the structured data information of the file to be detected does not match the information in the data structure, stop the sliding window movement;

[0062] Step S32, after both sliding windows stop, for the remaining content between the two sliding windows, continue to perform window sliding and continue to match in a dynamic decreasing step manner.

[0063] That is, when the hash value of the measured file does not match the data of the Bloom filter, stop the sliding window movement, and after both sliding windows stop, for the remaining text between the two sliding windows (i.e., the content suspected to have been modified), continue to perform sliding on these remaining texts in a dynamic step adjustment manner (preferably decreasing step), and continue to accumulate the number of matching shards.

[0064] This stage of continuing to match with dynamic step has higher flexibility and adaptability, especially it can find a balance between detection accuracy and storage overhead, which is difficult to achieve in general fuzzy hashing.

[0065] Step S4, after the matching ends, obtain the duplication degree of the file to be detected through all matching results.

[0066] That is, under the condition that it is not possible to continue to match with dynamic step, the final quantitative result of the duplication degree of the file to be detected and the original stored file is obtained by using the matching results counted in the entire matching stage. Overall, the implementation concept of this step is to calculate and measure the duplication degree between the whole files according to the similarity between the structured data, and different duplication degree calculation methods will be provided for reference later.

[0067] In combination with the above embodiments, the following will introduce the complete scheme implementation through specific examples.

[0068] I. Preprocessing of the file to be stored, to unify the character encoding:

[0069] Mainly convert each character of the file to be processed (i.e., the file to be stored) to the same length; specifically, encode all characters in the file to be processed to the same length, for example but not limited to using UTF-16 encoding, this step can ensure that the encoding length of each character is consistent, avoiding the "cutting" of characters in the subsequent fragmentation link.

[0070] II. Constructing the shard hash of the file to be stored:

[0071] (1) Fragmentation and initial step setting: corresponding to the aforementioned preprocessing of the file to be processed, the fragmentation length and the evidence storage sliding step are set. For example, in the example described above, if UTF-16 encoding is used, the fragmentation length and the step can be set to a multiple of 2, so that the movement of the sliding window (referred to as sliding window or sliding block) will not cut the characters.

[0072] (2) Fragmentation hash initial value calculation: in actual operation, from another point of view, that is, the first L bytes of the file to be stored are regarded as a sliding window (i.e. the fragmentation length is L), and the hash value of all bytes in the sliding window is calculated. Specifically, assuming that the decimal code of the character is b[i], where i is from 0 to L-1, the hash value h can be calculated by the following formula:

[0073]

[0074] Wherein, a is the sliding factor, and q is a predetermined large prime number.

[0075] The calculated hash value is taken as the hash value (i.e. the initial hash value) of the first fragment of the file, and then the sliding window is moved S bytes backward, and the new hash value is calculated. Specifically, the hash value h' of the new hash value can be efficiently calculated by using the hash value h of the previous step and the characteristics of the rolling hash by the following formula: h(b') = (h(b) - old_char·a L / 2-i )·a+new_char mod q;

[0076] Wherein, old_char is the first character code value to be removed, and new_char is the first character code value to be added. The hash value is updated by iteration until The final hash value is taken as the hash value of the second fragment of the file.

[0077] Three, bidirectional sliding window evidence storage:

[0078] On the basis of the foregoing setting, specifically, the disclosure proposes to use double sliding windows for opposite storage (which can also be referred to as opposite storage).

[0079] (1) File fragmentation:

[0080] First, two sliding windows are initialized, one at the beginning of the file to be stored and the other at the end of the file to be stored. Then, the hash values of the contents of the two sliding windows are calculated, and the sliding window is moved backward (for the beginning window) or forward (for the end window) until the two sliding windows meet or cross, and then the sliding can be stopped, which represents that the entire file to be stored has been traversed, and finally u fragments are obtained, as shown in Figure 3The article to be stored as evidence is illustrated, wherein the initial length of the fragment is L=10 and the step length is S=7. Figure 3 In the first fragment, the string is "bed moonlight, doubt is on the ground", and its encoding in UTF-16BE is: 5E8A, 524D, 660E, 6708, 5149, FF0C, 7591, 662F, 5730, 4E0A. Let a=256 and q=1000000007, and the following table is obtained.

[0081]

[0082]

[0083] Through the above process, the hash value Hash(1) of the first fragment (i.e., the fragment hash in the present disclosure) is obtained: 565700895. Then, the hash value of the second fragment is calculated. In the above example, the sliding step length is 7 characters, so the leftmost 7 characters of the hash value need to be removed, and then the new 7 characters are added. In combination with the example, the UTF-16BE encoding of the new 7 characters "frost. Looking up at the bright moon" is: 971C, 3002, 4E3E, 5934, 671B, 660E, 6708; then, the leftmost 7 characters of the hash value need to be removed. These characters are "bed moonlight, doubt", and their UTF-16BE encoding is: 5E8A, 524D, 660E, 6708, 5149, FF0C, 7591; thus, the following table is obtained.

[0084] Reject character Encoding Decimal code New character Encoding Decimal code Iterated hash value Bed 5E8A 24202 Cream 971C 38684 110680137 Before 524D 21069 。 3002 12290 444548630 Bright 660E 26126 Hold 4E3E 20030 485022009 Month 6708 26120 Head 5934 22836 498369576 Light 5149 20841 Look 671B 26395 19394918 ， FF0C 65292 Bright FF0C 65292 266899698 Suspect 7591 30097 Month 7591 30097 696545482

[0085] Through the above steps, the second fragment hash Hash(2) "696545482" is obtained, and the two sliding windows finally obtain u=15 hash fragments: Hash(1), Hash(2), Hash(3), Hash(...), Hash(15).

[0086] (2) Create a corresponding Bloom filter:

[0087] Since the Bloom filter has false positives, the false judgment rate can be controlled by setting a reasonable Bloom filter size and the number of files stored in each Bloom filter, so that the judgment rate is within an acceptable range. It should be noted that in actual operation, multiple Bloom filters can be set to form a Bloom filter group to store a large number of files, and the present disclosure only illustrates a single Bloom filter.

[0088] (2.1) Initialize the Bloom filter:

[0089] For each individual shard hash, the following steps will be performed to add it to the Bloom filter: Set the maximum number of files to be stored X, each file is divided into u shards (as the previous example article is cut into 15 pieces), the overall article false positive rate that can be tolerated is C, then according to the false positive rate formula of the Bloom filter: P = (1-e -kn / m ) k C = (1-(1-e -kXu / m ) k ) u , the formula describes the relationship between the accuracy C of the entire article and the number of bits m of the Bloom filter, the number of hash functions k, the number of articles X and the number of shards u of each article.

[0090] For example, in the above example, set m = 491474 bits, X = 1000, u = 15, k = 3, then C = 99%, the false positive rate P = 0.067%. Understandably, the characteristic of the Bloom filter is to search whether an element is in a set, only false positive (false alarm), and no false negative (miss). That is, if the Bloom filter indicates that an element (shard) exists in the set, it may be wrong (false positive); but if the Bloom filter indicates that an element does not exist in the set, it must be correct (i.e. no false negative).

[0091] Therefore, in the subsequent comparison process of the degree of repetition, the highest false positive rate is all false positives, that is, the article is completely repeated, in which case there is only false alarm and no miss, that is, each shard involves the false alarm problem of the Bloom filter, and the accuracy of the entire article is 99% in this case. Therefore, in other cases (i.e. partial repetition: both false positive and true negative), the accuracy of the article can be more than 99%.

[0092] Set P_ as the judgment accuracy of each shard, then P1·P2·P … ·P u In the process, if P represents repetition, then P_ = 1-0.067%; if P represents non-repetition, then P_ = 1, that is, C can represent the lowest accuracy of the entire file.

[0093] Further, for example, only a Bloom filter of about 60Kb can store 15000 shards (1000 articles), and the false positive rate (false positive rate) of each shard is 0.067%, since each article is divided into 15 shards, the minimum accuracy of each article is more than 99%, which shows strong practicability. As can be seen from the above, the present disclosure can reduce storage overhead under the condition of setting the overall minimum accuracy of the article.

[0094] (2.2) The slice hash is stored in the Bloom filter:

[0095] Using the Bloom filter initialized in the foregoing, each slice hash to be stored is further compressed to obtain k numbers with sizes in (0, m-1), and the corresponding bits in the Bloom filter are set to 1.

[0096] For example, in the foregoing example, K1(h1) = 543; K2(h1) = 16543 K3(h1) = 35533, and then the Bloom storage array of h1, i.e., the first slice, is (543, 16543, 35533). The 542nd, 16542nd, and 35532nd bits of the Bloom filter are set to 1, and the storage of the current first slice is completed. The foregoing method is cycled, and all the slices are stored in the Bloom filter. Thus, the file is stored in the Bloom filter through the fuzzy hash method.

[0097] In this way, the foregoing text forms 15 corresponding arrays, and the arrays show which bits of the Bloom filter are 1, and there are corresponding slice hash values.

[0098] (3) Data on-chain:

[0099] The relevant data is stored on-chain, and can specifically include:

[0100] The Bloom filter itself: including the hash information of all elements;

[0101] Bloom filter hash function information: k Bloom hash functions;

[0102] False positive rate: the false positive rate of the entire Bloom filter, to facilitate understanding of the storage performance;

[0103] and other data: such as timestamp, file slice number, and the like.

[0104] Four, file duplication detection steps (combined with Figure 4 a schematic diagram of a file to be detected):

[0105] (1) Obtain the blockchain data: interact with the blockchain to obtain the Bloom filter information stored in the blockchain.

[0106] (2) Analyze the Bloom filter: after obtaining the block data, the block is analyzed through a predefined data structure to extract the stored Bloom filter data.

[0107] (3) For the file to be detected, a predetermined initial step S is used for head-to-tail bidirectional sliding. In the sliding process, the hash of each sliding window (a sequence of bytes with a length of L) is calculated, and is matched with the content of the Bloom filter. The number of slices that are successfully matched and not duplicated is calculated, and until the unmatching condition occurs, the corresponding sliding window is paused.

[0108] (4) When both sides of the sliding window are stopped, the remaining text between the two sliding windows is extracted, and this part of the text is the part that may be modified, that is, the part that may be different between the current measured file and the stored file content.

[0109] (5) For this part of the text that may be modified, a dynamic change step is used to continue the sliding hash. At the beginning of this stage, the step can be set to a relatively large value and less than the initial S, and then it can be gradually reduced, for example, but not limited to, using the following moving step dynamic adjustment strategy: S n = S n-1 / 2. Generally, fuzzy hash cannot accurately distinguish different levels of similarity, and the present disclosure can more finely find out which parts of the text suspected to have modified content have a higher degree of repetition with the original file fragments by dynamically adjusting the step, thereby making the accuracy of the entire repetition detection higher.

[0110] (6) Repeat (5) until all possible steps adjusted have been used (for example, the moving step is reduced to only contain the smallest data unit of the measured file), or the number of matched fragments meets the previously mentioned fragment value u (that is, the stored content has been compared, and there is no comparison left).

[0111] (7) Calculate the degree of repetition: for example, the similarity between the detected file and the stored file can be represented by the ratio of the number of matched fragments to the total number of fragments u; of course, in addition to using the ratio for similarity analysis, different structured data (i.e., fragments) can also be weighted (i.e., importance), and higher weight parameters can be set for more important segment content, and then the similarity can be calculated according to the weight, thereby better reflecting the actual content repetition degree (also corresponding to the modification degree) between files, and this similarity calculation method is also more flexible and adaptable.

[0112] The above describes a data repetition degree detection method based on structured storage evidence according to an embodiment of the present disclosure. In the following, a data repetition degree detection device based on structured storage evidence for implementing the above detection method will be further described. Figure 5 is a functional block diagram illustrating a data repetition degree detection device based on structured storage evidence according to an embodiment of the present disclosure.

[0113] As Figure 5 shown, the data repetition degree detection device based on structured storage evidence 500 according to an embodiment of the present disclosure comprises:

[0114] a storage fragment module 501 for obtaining structured data information of a file to be stored in advance by using a bidirectional sliding window;

[0115] The fragment information processing module 502 is configured to store the structured data information into a preset data structure and complete the uplink storage;

[0116] The file verification module 503 is configured to, when performing the duplication detection, obtain the structured data information of the to-be-detected file by using the bidirectional sliding window, and match the information in the data structure extracted from the blockchain and parsed; and in the matching process, the moving step of the sliding window is dynamically adjusted to match the remaining file content when the matching fails.

[0117] The duplication calculation module 504 is configured to, after the matching ends, obtain the duplication of the to-be-detected file through all the matching results.

[0118] Further, the file verification module comprises:

[0119] The matching pause unit is configured to, when the structured data information of the to-be-detected file does not match the information in the data structure, stop the sliding window from moving.

[0120] The dynamic step adjustment unit is configured to, after both sliding windows stop, continue to execute the window sliding and matching on the remaining content between the two sliding windows by using a dynamically decreasing step.

[0121] Further, the condition for confirming that the matching ends comprises that all available steps adjusted dynamically have been used, or the information content stored in the data structure has been compared and matched.

[0122] Further, the fragment information processing module is configured to construct an array corresponding to the structured data information by using the data structure, and use the array as the storage fingerprint of the to-be-stored file in the data structure.

[0123] Further, the duplication calculation module comprises:

[0124] The first similarity calculation unit is configured to obtain the similarity between the to-be-detected file and the corresponding stored file according to the ratio of the successfully matched structured data to the total number of structured data of the corresponding stored file; or the second similarity calculation unit is configured to set a weight for different structured data of the stored file, and calculate the similarity between the to-be-detected file and the corresponding stored file according to the successfully matched result and in combination with the weight.

[0125] Figure 6 FIG. 6 is a hardware block diagram of an electronic device 600 according to an embodiment of the present disclosure. The electronic device according to the embodiment of the present disclosure at least includes a processor; and a memory for storing computer readable instructions. When the computer readable instructions are loaded and run by the processor, the processor performs the data duplication detection method based on structured storage according to the above description.

[0126] Figure 6 The illustrated electronic device 600 specifically includes a central processing unit (CPU) 601, a graphics processing unit (GPU) 602, and a main memory 603. These units are interconnected via a bus 604. The CPU 601 and / or GPU 602 can function as the aforementioned processors, and the main memory 603 can function as the aforementioned memory storing computer-readable instructions. Furthermore, the electronic device 600 may also include a communication unit 605, a storage unit 606, an output unit 607, an input unit 608, and an external device 609, all of which are also connected to the bus 604.

[0127] Figure 7 This is a schematic diagram illustrating a computer-readable storage medium according to an embodiment of the present disclosure. Figure 7 As shown, a computer-readable storage medium 700 according to an embodiment of the present disclosure stores computer-readable instructions 701 thereon. When the computer-readable instructions 701 are executed by a processor, the data duplication detection method based on structured evidence storage according to an embodiment of the present disclosure, as described with reference to the above figures, is performed. The computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, optical disk, magnetic disk, etc.

[0128] Finally, it can be added that, according to an embodiment of this disclosure, a computer program product (which may include the aforementioned apparatus) runs on a terminal device, causing the terminal device to execute the data duplication detection method based on structured evidence storage as described in the foregoing embodiments or equivalent implementations. As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the above implementation methods can be implemented using software plus the necessary general-purpose hardware platform. Based on this understanding, the aforementioned computer program product may include, but is not limited to, an APP.

[0129] The above describes the data duplication detection method, device, electronic equipment, computer readable storage medium and computer program product based on structured storage evidence according to the embodiments of the present disclosure with reference to the drawings, and the main idea is that, unlike the traditional block chain storage evidence which only focuses on data authenticity, the core idea of the present disclosure is to introduce structured data information, a preset data structure, a bidirectional sliding window and a dynamic step length mechanism for data processing to realize storage and verification, so that the storage process can not only ensure the consistency and integrity of the data, but also accurately reflect the duplication of the verification data and the storage file. In the electronic evidence mode based on the block chain, the present disclosure can detect the duplication of a large-scale data set while ensuring the effectiveness and accuracy of data storage, taking into account a lower false positive rate and a higher storage space efficiency, thereby reducing storage overhead and significantly expanding the functionality and application range of storage.

[0130] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0131] The basic principles of the present disclosure are described above in combination with specific embodiments, but it should be noted that the advantages, advantages, effects and the like mentioned in the present disclosure are only examples and not limitations, and these advantages, advantages, effects and the like cannot be considered as the must-have of each embodiment of the present disclosure. In addition, the above specific details are only for the purpose of example and for the purpose of understanding, and the above details do not limit the present disclosure to the above specific details.

[0132] The block diagrams of the devices, apparatuses, equipment, systems involved in the present disclosure are only illustrative examples and are not intended to require or imply the connection, arrangement, configuration shown in the block diagram. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any way. Words such as "include", "contain", "have" and the like are open-ended words, which mean "include but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0133] Also, as used in the description herein and throughout the claims that follow, the meaning of "or" includes both exclusive and inclusive or. Unless the context clearly requires otherwise, throughout the description and the claims that follow, the word "comprise", and variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps. Also, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Further, the term "example" does not mean that a described example is preferred or better than other examples.

[0134] It is also important to note that the systems and methods of the disclosure can be embodied in a variety of forms depending on the particular requirements of a given application. Some embodiments can be implemented in software, while other embodiments can be implemented in hardware or a combination of software and hardware. As used herein, the term "software" includes but is not limited to one or more computer program(s) and / or associated data structures.

[0135] Various changes, modifications and alterations to the techniques described herein can be made without departing from the teachings of the attached claims. Moreover, the scope of the claims should not be limited to the particular aspects described in the above description and drawings. Rather, the scope of the claims should be understood to include any and all aspects that are within the scope of the claims. Accordingly, the above description is not intended to limit the scope of the claims but to describe various aspects of the disclosure.

[0136] The above description of the disclosed aspects is given for illustrative purposes only and is not intended to limit the scope of the disclosure. Although various embodiments have been discussed, those skilled in the art will readily devise numerous other variations of these embodiments that fall within the scope of the present disclosure. Accordingly, the scope of the disclosure is not intended to be limited to the aspects described herein, but is intended to include all variations falling within the scope of the claims.

[0137] The above description has been given for illustrative purposes only and is not intended to limit the scope of the disclosure. Although various embodiments have been discussed, those skilled in the art will readily devise numerous other variations of these embodiments that fall within the scope of the present disclosure. Accordingly, the scope of the disclosure is not intended to be limited to the aspects described herein, but is intended to include all variations falling within the scope of the claims.

Claims

1. A data duplication detection method based on structured evidence storage, characterized in that, The detection method includes: The structured data information of the document to be stored is obtained in advance using a two-way sliding window; The structured data information is stored in a preset data structure and uploaded to the blockchain; When performing duplicate detection, a bidirectional sliding window is used to obtain the structured data information of the file to be detected, and it is matched with the information in the data structure extracted from the blockchain and parsed. During the matching process, the sliding window's movement step size is dynamically adjusted to match the remaining file content when a mismatch occurs. This includes stopping the sliding window movement when the structured data information of the file to be detected does not match the information in the data structure. After both sliding windows stop, the window continues to slide and match the remaining content between the two sliding windows by dynamically decreasing the step size. After the matching is completed, the duplication rate of the file to be detected is obtained from all the matching results, including the similarity between the file to be detected and the corresponding evidence file based on the ratio of the total number of structured data of the successfully matched structured data to the total number of structured data of the corresponding evidence file.

2. The data duplication detection method based on structured evidence storage as described in claim 1, characterized in that, The conditions for confirming the end of the matching process include: All available step sizes that have been dynamically adjusted have been adopted, or the information content stored in the data structure has been compared.

3. The data duplication detection method based on structured evidence storage as described in claim 1, characterized in that, The step of storing the structured data information into a preset data structure includes: An array corresponding to the structured data information is constructed using the data structure, and the array is used as the storage fingerprint of the document to be certified in the data structure.

4. The data duplication detection method based on structured evidence storage as described in claim 1, characterized in that, The process of determining the duplication rate of the file to be detected by matching all results also includes: Weights are assigned to different structured data in the evidence storage files. Based on the successful matching results and the weights, the similarity between the file to be detected and the corresponding evidence storage file is calculated.

5. The data duplication detection method based on structured evidence storage as described in any one of claims 1 to 4, characterized in that, Methods for obtaining structured data information using a two-way sliding window include: A sliding window is set at the beginning and end of the file, respectively. The two sliding windows move towards each other from the beginning and end of the file based on a preset step size. During the movement of each sliding window, the corresponding hash value is calculated for the file area within the sliding window.

6. A data duplication detection device based on structured evidence storage, characterized in that, The detection device includes: The evidence storage segmentation module is used to obtain the structured data information of the document to be stored in advance using a two-way sliding window; The sharding information processing module is used to store the structured data information into a preset data structure and complete the on-chain processing. The file verification module is used to obtain the structured data information of the file to be detected using a two-way sliding window when performing duplicate detection, and to match it with the information in the data structure extracted and parsed from the blockchain; and during the matching process, the sliding window's movement step size is dynamically adjusted to match the remaining file content when a mismatch occurs, including stopping the sliding window movement when the structured data information of the file to be detected does not match the information in the data structure, and after both sliding windows stop, continuing to slide the window and continue matching the remaining content between the two sliding windows using a dynamically decreasing step size; The repetition calculation module is used to calculate the repetition of the file to be detected after the matching is completed, based on all matching results. This includes obtaining the similarity between the file to be detected and the corresponding evidence file based on the ratio of the total number of successfully matched structured data to the total number of structured data in the corresponding evidence file.

7. An electronic device, characterized in that, include: Memory, used to store computer-readable instructions; as well as A processor for executing the computer-readable instructions, causing the electronic device to perform the data duplication detection method based on structured evidence as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium for storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by a processor, the processor performs the data duplication detection method based on structured evidence as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the data duplication detection method based on structured evidence as described in any one of claims 1 to 5.

Images