Data detection method, device, non-volatile storage medium and computer equipment

Abstract

The present invention discloses a data detection method, apparatus, non-volatile storage medium, and computer equipment. The method comprises: obtaining file data stored in a cloud storage system and file data cached in a first node; aggregating data that does not have locality in the file data stored in the cloud storage system to obtain a target data set; generating a first list based on the target data set; generating a second list based on the file data cached in the first node; determining whether a digital fingerprint in the second list is located in the first list; and merging the file data cached in the first node into the cloud storage system if the first digital fingerprint in the second list is located in the first list. The present invention solves the technical problem of low storage space utilization caused by duplicate data in a cloud storage system.

Description

Technical Field

[0001] The present invention relates to the fields of cloud computing and big data, and in particular to a data detection method, device, non-volatile storage medium and computer equipment. Background Art

[0002] With the rapid development of information technology, the total amount of digital information is experiencing explosive growth, exceeding the growth rate of physical storage space. This makes reducing data storage capacity a crucial and practical issue. Two common technologies for reducing data storage capacity are traditional data compression and the recently emerging deduplication technology (Dedup). This technology identifies identical data units in a storage system and stores only one copy, thereby reducing the space required for data storage.

[0003] Deduplication technology initially emerged and was applied in the field of backup storage, specifically secondary storage. However, duplicate data also exists in cloud storage systems, resulting in significant waste of storage space. When deduplication technology was adopted in the internet environment, the lack of block locality in cloud data meant that it could not be directly applied to indexing services, thus failing to address the problem of wasted storage space.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] Embodiments of the present invention provide a data detection method, apparatus, non-volatile storage medium, and computer equipment to at least solve the technical problem of low storage space utilization caused by duplicate data in a cloud storage system.

[0006] According to one aspect of an embodiment of the present invention, a data detection method is provided, comprising: obtaining file data stored in a cloud storage system and file data cached in a first node, wherein the first node is located in the cloud storage system; aggregating data that is not local in the file data stored in the cloud storage system to obtain a target data set; generating a first list based on the target data set, wherein the first list includes digital fingerprints corresponding to the file data in the target data set; generating a second list based on the file data cached in the first node, wherein the second list includes digital fingerprints corresponding to the file data cached in the first node; determining whether the digital fingerprint in the second list is located in the first list; and merging the file data cached in the first node into the cloud storage system if the first digital fingerprint in the second list is located in the first list.

[0007] Optionally, before obtaining the file data cached in the first node, the method includes: selecting the first node according to a resource scheduling identifier method.

[0008] Optionally, after obtaining the target data set, the method includes: calculating the false positive rate of the target data set, wherein the false positive rate represents the probability of an error in the aggregation result; and determining that the target data set is correct when the false positive rate is lower than a preset threshold.

[0009] Optionally, merging the file data cached in the first node into the cloud storage system includes: storing a portion of the file data cached in the first node that is different from the file data in the cloud storage system into the cloud storage system.

[0010] Optionally, when none of the digital fingerprints in the second list are in the first list, the method includes: obtaining the digital fingerprints corresponding to the file data cached in the nodes other than the first node in the cloud storage system; generating a list corresponding to the node based on the digital fingerprints corresponding to the file data cached in the node; judging whether the digital fingerprint in the list corresponding to the node is in the first list; and when there is a second node in the node and there is a second digital fingerprint in the list corresponding to the second node that is in the first list, merging the file data cached in the node corresponding to the second digital fingerprint into the cloud storage system.

[0011] Optionally, when none of the digital fingerprints in the list corresponding to the node is located in the first list, the method includes: storing the first node and the file data cached in the node in a cloud storage system.

[0012] According to another aspect of an embodiment of the present invention, a data detection device is also provided, including: an acquisition module for acquiring file data stored in a cloud storage system and file data cached in a first node, wherein the first node is located in the cloud storage system; an aggregation module for aggregating data that does not have locality in the file data stored in the cloud storage system to obtain a target data set; a first generation module for generating a first list based on the target data set, wherein the first list includes digital fingerprints corresponding to the file data in the target data set; a second generation module for generating a second list based on the file data cached in the first node, wherein the second list includes digital fingerprints corresponding to the file data cached in the first node; a judgment module for judging whether the digital fingerprint in the second list is located in the first list; and a merging module for merging the file data cached in the first node into the cloud storage system if the first digital fingerprint in the second list is located in the first list.

[0013] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided. The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above-mentioned data detection methods.

[0014] According to yet another aspect of an embodiment of the present invention, a computer device is provided. The computer device includes a processor, and the processor is configured to run a program. When the program is run, any one of the above-mentioned data detection methods is executed.

[0015] According to yet another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, which implements any one of the above-mentioned data detection methods when executed by a processor.

[0016] In an embodiment of the present invention, a data detection method is adopted, by obtaining file data stored in a cloud storage system and file data cached in a first node, wherein the first node is located in the cloud storage system; aggregating data that does not exist locally in the file data stored in the cloud storage system to obtain a target data set; generating a first list based on the target data set, wherein the first list includes digital fingerprints corresponding to the file data in the target data set; generating a second list based on the file data cached in the first node, wherein the second list includes digital fingerprints corresponding to the file data cached in the first node; judging whether the digital fingerprint in the second list is located in the first list; if there is a first digital fingerprint in the second list that is located in the first list, merging the file data cached in the first node into the cloud storage system, thereby achieving the purpose of comparing the file data in the cache with the file data already existing in the cloud storage system and avoiding storing the same data in the cloud storage system again, thereby achieving the technical effect of saving data storage space, and further solving the technical problem of low storage space utilization caused by the existence of duplicate data in the cloud storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a computer terminal for implementing a data detection method is shown;

[0019] Figure 2 is a flow chart of a data detection method according to an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of the structural relationship between a cloud storage system and nodes provided in an optional embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of an index service cache and pre-fetching workflow in a data detection method provided according to an optional embodiment of the present invention;

[0022] Figure 5 is a structural block diagram of a data detection device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] According to an embodiment of the present invention, an embodiment of a data detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal for implementing a data detection method. Figure 1As shown, the computer terminal 10 may include one or more (illustrated as 102a, 102b, ..., 102n in the figure) processors (the processor may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices), a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0027] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0028] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data detection method in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the data detection method of the above-mentioned application. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0030] Figure 2 FIG. 1 is a flow chart of a data detection method according to an embodiment of the present invention. Figure 2 As shown, the method includes the following steps:

[0031] Step S201: Acquire file data stored in a cloud storage system and file data cached in a first node, wherein the first node is located in the cloud storage system.

[0032] In this step, the cloud storage system is a distributed data storage method that allows users to store data on remote servers rather than local devices. A node is a data storage unit in the cloud storage system, responsible for storing, managing, and distributing data. Users can upload data to a node, while other users or applications can retrieve data from the node. Nodes can cache file data as temporary storage, making it easier for users to retrieve data or store data in the cloud storage system. The first node can be the node with the strongest locality between the file data cached by all nodes and the file data in the cloud storage system.

[0033] Step S202 : Aggregate the data that does not have locality in the file data stored in the cloud storage system to obtain a target data set.

[0034] In this step, data deduplication technology was initially applied to local storage systems. Data in local storage is local, and duplicate data can be quickly detected and processed based on this locality. However, data in cloud storage systems lacks locality for various reasons, making it difficult to detect duplicate data based on locality. Consequently, deduplication technology cannot be directly applied, and processing duplicate data in storage is difficult. Therefore, this problem can be addressed by aggregating data in cloud storage systems that lack data locality. In computer science and programming, locality refers to the tendency of programs to access adjacent or nearby memory regions when accessing memory addresses. Specifically, it can be categorized as temporal locality and spatial locality. Temporal locality means that once a piece of data is accessed, it is likely to be accessed again in the near future. Spatial locality means that programs tend to access adjacent locations in memory. For example, if an array is accessed, other data within the array is likely to be accessed in sequence. Data locality in this context can be spatial locality.

[0035] There are three reasons why data locality is absent in cloud storage systems: First, files are relatively small compared to disks and may be distributed across different physical locations during storage. Therefore, data locality may be affected by this distributed storage. Second, only modified files are re-uploaded, not the entire disk, resulting in discontinuous data distribution within the storage system, breaking the original data locality. Finally, the order in which multiple modified files are uploaded may differ from the order in which they were last uploaded, leading to data locality loss. This difference in upload order can lead to discontinuous data distribution across physical storage, thus affecting locality. To aggregate data in cloud storage systems that lack data locality, the k-means clustering model can be used. k-means clustering is an unsupervised learning clustering algorithm used to classify data into k categories or clusters. Its main idea is to group data points by assigning them to the nearest cluster center. It should be noted that this k-means clustering approach is merely an example and is not intended to be limiting.

[0036] For example, when using the k-means clustering model for aggregation, the following steps can be taken: First, define the data in the cloud storage system as a sample dataset X (i) ={x1 (i) ,x2 (i) ,...,x n (i)}, where i = 1, 2, ..., m, m represents the number of samples, n represents the number of features, x n (i) Represents the feature value in each sample. Then set the clustering parameter. Based on the classification of data block application scenarios, it is divided into three levels: A, B, and C. Among them, A is a deterministic data block, which means a data block without locality; B is an uncertain data block, which means a data block with locality; C is a non-existent data block, which means a data block that does not exist in the cloud storage system. Therefore, the clustering parameter can be set to 3.

[0037] Next, specify the class center. The number of clusters is 3. Therefore, 3 class centers need to be specified. There are two methods for selecting class centers: random method and farthest distance method. Among them, the farthest distance method randomly specifies a sample in the sample data set as the first class center, and then calculates the distance from the remaining samples to the sample, and selects the farthest distance sample as the second class center. This is repeated until 3 class centers are selected. Set the class center as C, C = {C1 i ,C2 i ,...,C n i}, i = 1, 2, 3, where C n iRepresents the center of the i-th cluster. The distance can be calculated using the Euclidean distance calculation formula, which is as follows:

[0038]

[0039] Among them, x i ,x j Representing two different samples, the Euclidean distance formula can be applied to calculate the distance between each sample data and the three cluster centers in turn and assign them to the nearest cluster.

[0040] Finally, after all the sample data are distributed, it is necessary to update the class centers of the three classes of data and calculate the mean of the sample data in each class as the new class center. The calculation formula is as follows:

[0041]

[0042] Among them, X represents the sample value in the class center, and n represents the number of samples.

[0043] The above assignment and update process is repeated. To ensure classification quality, the class center must remain constant after multiple iterations. You can determine whether to stop iteration by specifying the number of iterations or by setting the range of class center variation. For example, when specifying the number of iterations, iteration stops when the number reaches a preset threshold; when setting the range of class center variation, iteration stops when the distance between the new class center and the old class center falls below a preset threshold. After the iteration stops, the sample data is assigned to the corresponding class based on the determined class center, and the classification result is obtained.

[0044] Step S203: Generate a first list based on the target data set, wherein the first list includes digital fingerprints corresponding to the file data in the target data set.

[0045] In this step, the file data in the target data set is digitally fingerprinted and encrypted to generate a corresponding list. A digital fingerprint is a technology used to identify and track digital files or data, and can be used for copyright protection, content identification, and preventing data tampering. For example, a digital fingerprint can be generated by a hash function, which is a mathematical function that can convert an input of any length into an output of a fixed length, called a hash value or hash digest. The hash value is very sensitive to the input data, and even a small change will cause a significant change in the hash value. An ideal hash function should be able to generate a unique hash value for each different input, which makes the digital fingerprint unique. Therefore, a digital fingerprint list can be formed by calculating the hash value corresponding to the file data in the target data set as a digital fingerprint. The digital fingerprint has a one-to-one correspondence with the file data, and the digital fingerprint can be used as an index when detecting the file data, which is conducive to improving the efficiency of detection.

[0046] Step S204: Generate a second list based on the file data cached in the first node, wherein the second list includes digital fingerprints corresponding to the file data cached in the first node.

[0047] In this step, a hash value corresponding to the file data cached in the first node is calculated, and the hash value is used as the digital fingerprint corresponding to the file data to form a digital fingerprint list, wherein the digital fingerprints in the digital fingerprint list have a one-to-one correspondence with the data blocks corresponding to the file data in the cache of the first node. Generating a second list allows for subsequent duplicate file data detection, eliminating the need to compare the contents of the file data one by one. Instead, the determination of data duplication can be determined by comparing the digital fingerprints to see if they match, thus reducing the time spent on data comparison and improving efficiency.

[0048] Step S205: Determine whether the digital fingerprint in the second list is in the first list.

[0049] In this step, it is detected in turn whether the digital fingerprints in the second list are already in the first list, wherein the digital fingerprints and the file data are in a one-to-one correspondence. Therefore, it is detected whether the digital fingerprints in the second list are already in the first list, that is, it is detected whether the file data cached in the first node already exists in the cloud storage system. This can avoid repeatedly storing the file data cached in the node in the cloud storage system, resulting in a waste of storage space.

[0050] Step S206: When the first digital fingerprint exists in the first list in the second list, the file data cached in the first node is merged into the cloud storage system.

[0051] In this step, when the first digital fingerprint exists in the second list and is located in the first list, it means that the file data corresponding to the first digital fingerprint in the node has been stored in the cloud storage system. Therefore, in order to avoid repeated storage of file data, other data in the first node cache that does not exist in the cloud storage system can be stored in the cloud storage system to achieve an update of the cloud storage system.

[0052] Figure 3 Schematic diagram of the structural relationship between the cloud storage system and nodes provided by an optional embodiment of the present invention. Figure 3As shown, a cloud storage system includes multiple nodes. When storing files in the cloud storage system, the files are first cached in the nodes. That is, the file data cached in the nodes is the data that will be stored in the cloud storage system. To avoid duplicate data storage, it is possible to first detect whether the file data cached in the node already exists in the cloud storage system. If the file data already exists, the file data cached by the node will not be stored in the cloud storage system again. This prevents duplicate storage of files in the cloud storage system, which would otherwise waste storage space in the cloud storage system. This can achieve the technical effect of improving the utilization of storage space in the cloud storage system, and can reduce the amount of data stored to 1 / 20 or even 1 / 60 of the original amount.

[0053] Through the above steps, the purpose of comparing the file data in the cache with the file data already existing in the cloud storage system is achieved, avoiding storing the same data in the cloud storage system again, thereby achieving the technical effect of saving data storage space, and further solving the technical problem of low storage space utilization caused by duplicate data in the cloud storage system.

[0054] As an optional embodiment, before obtaining the file data cached in the first node, the method includes: selecting the first node according to a resource scheduling identifier method.

[0055] Optionally, in a cloud storage system, different scheduling methods can be used to select nodes to perform tasks according to different task requirements. For example, a round-robin scheduling method can be used to select nodes in sequence. This method can achieve load balancing, but may not be optimized according to the actual performance and load status of the nodes. A random scheduling method can also be used to randomly select nodes to perform tasks. This method is simple and easy to implement, but it cannot guarantee load balancing and optimal performance. A minimum connection scheduling method can also be used to select the node with the least current connections for data operations. This method helps to balance the load of the nodes, but may ignore the performance differences of the nodes. The above scheduling methods can all be used when selecting nodes, but in an embodiment of the present invention, the resource identifier scheduling method can be selected based on the similarity of the locality list, and the ID corresponding to the node is used for scheduling. Compared with other scheduling methods, it can more effectively select the node that is most likely to cache duplicate data, thereby improving data processing efficiency.

[0056] Among them, the Resource Control Identifier (RCID) method is a system for resource scheduling and management, which mainly realizes the control, tracking and scheduling of resources by assigning a unique identifier to each resource. In an embodiment of the present invention, the locality lists stored in different nodes are different, and different IDs can be assigned to each node. Then, based on the locality list in the node cache, a suitable node ID is selected as the first node. For example, when selecting the first node, the locality list in the node cache can be compared with the locality list in the cloud storage system, and the node with the highest similarity to the locality list in the cloud storage system is selected as the first node. The scheduling of the first node can be achieved through the corresponding ID, which is conducive to more efficient operation of caching and prefetching technology. During the scheduling process, the scale of digital fingerprint data can reach several TB or even tens of TB. Traditional databases cannot meet the storage and fast search requirements of such large-scale data. Therefore, data block layering can be performed to complete the optimization process of caching and prefetching based on file locality.

[0057] First, the core application service topology can be mined and associated with the network topology by leveraging the relationships between databases, middleware (software that enables communication and data exchange between different applications), and basic server monitoring indicators in the log service application database. The core application service topology is a graphical representation that shows the performance indicator relationships between different servers and components such as the database, middleware, server CPU, memory, disk, and process. Secondly, a data set based on the first-layer core application service topology relationships, called the topology data set, can be mined from each service's log. This data records monitoring data and operational health status between servers and databases, middleware, server CPU, memory, disk, and processes across different time dimensions. Then, after locating the network device through the first two layers, drill down to the board layer to more accurately locate the root cause. Network devices are composed of multiple boards, each of which includes an independent CPU, memory, disk, and other components. Finally, this problem can be solved by implementing hierarchical indirect access based on data blocks. First, a unique identifier (UID) can be generated for the locality list to be searched. This UID is used as the key and the locality list as the value, and the data is stored in the DHT. "Key" and "value" are commonly used to describe key-value pairs, data structures such as dictionaries, hash tables, or maps. This data structure allows for the storage and retrieval of data, where each data item consists of two parts: a key and a value. The key is a unique identifier used to index or find data. The value is the data associated with the key. In a key-value pair, the value is the data stored according to the key. Second, a digital fingerprint is used as the key and the UID as the value, and the data is stored in the DHT. When reading the digital fingerprint, if the UID is obtained instead of the locality list, another DHT search is performed using the UID as the key to obtain the corresponding locality list.

[0058] The DHT, a distributed hash table, refers to a list of digital fingerprints corresponding to file data cached across all nodes in the cloud storage system, including a list for each node. To maintain file locality information in the index service, when a digital fingerprint is stored in the DHT, a list of all digital fingerprints with file locality associated with that key, called a locality list, is used as a key and stored as a value. When large files cause an excessive number of digital fingerprints in the locality list, the same locality list is stored multiple times as a value, resulting in significant DHT storage waste. Cache and prefetch nodes provide the same key / value access interface as the DHT, similarly using the digital fingerprint as the key and the locality list as the value. Furthermore, due to the distributed nature of AegeanStore (the cloud storage system), it is difficult for the database to guarantee quality of service for requests from different networks. Therefore, digital fingerprints in the index service are stored in a one-hop distributed hash table, which improves the scalability and distributed access characteristics of the index storage.

[0059] As an optional embodiment, after obtaining the target data set, it includes: calculating the false positive rate of the target data set, wherein the false positive rate represents the probability of an error in the aggregation result; when the false positive rate is lower than a preset threshold, determining that the target data set is correct.

[0060] Optionally, in the indexing service, because non-existent data blocks may be identified as existing and removed, leading to data loss, the Bloom filter (fast filtering module) false positive rate formula can be used to calculate the error rate of aggregated data. Based on the error rate, the correctness of the aggregated data can be determined, which helps avoid deleting new data as duplicates during duplicate detection, which could lead to data loss. The Bloom filter false positive rate (FPR) refers to the probability of incorrectly determining whether an element exists in a set. Because Bloom filters are in-memory data structures, they offer excellent access speed and concurrency. Furthermore, through optimizations for file-based batch digital fingerprint searches, network requirements are significantly reduced. Specifically, the FPR refers to the probability of incorrectly reporting an element as existing in a set when it is not. For example, the FPR of a target data set refers to the probability of incorrectly classifying a piece of data as existing in the target data set when it is not. The FPR depends primarily on two factors: the number of hash functions and the size of the bit array. Generally speaking, increasing the number of hash functions and the size of the bit array can reduce the FPR, but this also increases space and time complexity. The formula for calculating the false positive rate of Bloom filter is as follows:

[0061]

[0062] Among them, R FP is the false positive rate, M is the size of the bit array, K is the number of hash functions, and N is the number of existing elements. As the number of elements N increases, the false positive rate R FPIt will also improve. To ensure operational efficiency, the false positive rate (FPR) must be kept below a certain threshold. When the FPR exceeds the preset threshold, it indicates an error in the aggregation result. The aggregation operation can be repeated or parameters in the aggregation operation can be adjusted, for example, by modifying clustering parameters and changing the method for selecting cluster centers. In an embodiment of the present invention, a SHA-1 hash function can be used to calculate a digital fingerprint, which has a size of 20 Bytes. When the amount of digital fingerprint data in the indexing service grows to 100 GB, the number of data blocks in the cloud storage system will be 500,000,000,000. Based on an average block size of 4 KB, the storage capacity reaches 20 TB. If the number of hash functions is 3 and the maximum allowable FPR is 10%, calculation shows that the required bit array is approximately 24 GB. In other words, the Bloom filter structure requires approximately 3 GB of memory space for effective and fast data block filtering. Therefore, the Bloom filter module can meet data storage requirements ranging from tens to hundreds of TB. If the data storage scale in the cloud storage system increases further, technologies such as scalable Bloom filters and layered Bloom filter arrays can be adopted to improve the scalability of the module.

[0063] In addition, different methods can be used to detect the accuracy of the calculation results of the formula. For example, multiple calculations can be performed to avoid errors; the target results can be calculated using different formulas, and the accuracy of the initial formula calculation results can be determined by comparing whether the two are consistent; the accuracy of the calculation results can also be detected using a cell mutation algorithm. Compared with other detection methods, the cell mutation algorithm is adaptive and can maintain stability in the face of complex and dynamically changing environments, thereby reducing the risk of misjudgment and missed judgment. At the same time, the cell mutation algorithm does not require a large amount of computing resources and also has advantages in computing cost and time.

[0064] Cell mutation algorithms are primarily used to solve optimization problems, simulating the growth and division of biological cells to search for the optimal solution in the solution space. For example, to ensure the correctness of the formula for calculating the false positive rate, the calculation process can be transformed into an optimization problem, and the result of the formula can be used as the optimization target of the cell mutation algorithm. Specifically, a fitness function can be defined that evaluates the quality of individuals based on the difference between the formula's calculation result and the expected result. The cell mutation algorithm can then search for the solution that optimizes the fitness function.

[0065] Among them, the differential evolution algorithm (DE) is a type of cell mutation algorithm. The differential evolution algorithm includes mutation, crossover and selection operations. Among them, the mutation operation refers to the generation of new individuals by linear operations on multiple individuals within the species. There are many mutation operation strategies for the differential evolution algorithm, which can be adopted as follows: in the gth iteration, three individuals X are randomly selected from the population. p1 (g),X p2 (g),X p3 (g), and p1≠p2≠p3≠i, indicating that the three individuals are different and none of them is the individual X being evaluated. i (g), the generated mutation vector is: V i (g) = X p1 (g)+F·(X p2 (g)-X p3 (g)), where X p2 (g)-X p3 (g) represents the difference vector, and F represents the scaling factor. The scaling factor determines the size of the individual difference step size. A smaller F can affect the variability between individuals in the population, causing the algorithm to fall into a local optimum. A larger F can enhance the algorithm's global search capability and facilitate the search for the optimal solution, but it will affect the algorithm's convergence speed. Therefore, F can generally be selected between [0, 2], and is usually 0.5.

[0066] In order to better select the value of F, we can use the method of adaptively adjusting F. For example, we can sort the three randomly selected individuals from best to worst to get X b ,X m ,X w and the corresponding fitness f b ,f m ,f w The mutation vector can be: V i =X b +F i (X m -X w ), where F i Represents the adaptive value of F. The value of F changes adaptively based on the two individuals that generate the difference vector: Among them, F l =0.1,F u =0.9.

[0067] Differential evolution algorithms can have multiple mutation strategies, expressed as DE / a / b, where a represents the selection scheme for the mutated individual and b represents the number of differential vectors. For example, the above mutation strategy can be expressed as DE / rand / 1. Other common mutation strategies include: DE / best / 1, where the mutation vector is V. i (g) = X best (g)+F·(X p1 (g)-X p2 (g)), where X best (g) represents the individual with the highest fitness in the gth generation; there is also a mutation strategy represented by DE / current-to-best / 1, and the mutation vector is V i (g) = X i (g)+F·(X best (g)-X i (g))+F·(X p1 (g)-X p2 (g)); There is also a mutation strategy represented by DE / best / 2, and the mutation vector is V i (g) = X best (g)+F·(X p1 (g)-X p2 (g))+F·(X p3 (g)-X p4 (g)).

[0068] As an optional embodiment, merging the file data cached in the first node into the cloud storage system includes: storing a portion of the file data cached in the first node that is different from the file data in the cloud storage system into the cloud storage system.

[0069] Optionally, when it is detected that the file data cached by the node already exists in the cloud storage system, a merge operation is performed. For example, if some of the file data cached by the first node already exists in the cloud storage system, this data cannot be stored in the cloud storage system again to avoid duplicate data storage. However, the remaining file data that does not exist in the cloud storage system can be stored in the cloud storage system, avoiding data loss caused by the failure to store new data.

[0070] As an optional embodiment, when none of the digital fingerprints in the second list are in the first list, the method includes: obtaining the digital fingerprints corresponding to the file data cached in the nodes other than the first node in the cloud storage system; generating a list corresponding to the node based on the digital fingerprints corresponding to the file data cached in the node; judging whether the digital fingerprint in the list corresponding to the node is in the first list; and when there is a second node in the node and there is a second digital fingerprint in the list corresponding to the second node that is in the first list, merging the file data cached in the node corresponding to the second digital fingerprint into the cloud storage system.

[0071] Optionally, after the detection of the file data cached by the first node is completed, different processing is performed based on whether there is duplicate data. For example, non-duplicate data can be stored in the cloud storage system, while duplicate data will not be stored again. Continue to detect the file data cached in other nodes to determine whether the file data cached in other nodes are in the cloud storage system. For example, calculate the hash value of the file data as the corresponding digital fingerprint, and then generate a list of digital fingerprints corresponding to the file data cached by each of the remaining nodes. Similarly, determine whether the digital fingerprints in the digital fingerprint lists of the remaining nodes already exist in the first list. When the second digital fingerprint exists in the first list, it means that the data block corresponding to the second digital fingerprint has been stored in the cloud storage system. In order to avoid duplicate storage of data, the remaining file data cached in the node corresponding to the second digital fingerprint is stored in the cloud storage system.

[0072] As an optional embodiment, when none of the digital fingerprints in the list corresponding to the node is located in the first list, the method includes: storing the first node and the file data cached in the node in a cloud storage system.

[0073] Optionally, each node's digital fingerprint list is checked one by one. If no fingerprints in the first list exist in the digital fingerprint lists corresponding to all nodes, it indicates that the file data cached in the node is new data that has not yet been stored in the cloud storage system and can therefore be stored in the cloud storage system. Alternatively, it indicates a false positive. A false positive occurs when data that does not belong to the cloud storage system is classified as belonging to the cloud storage system in the aggregation results, resulting in the detection of new data that does not belong to the cloud storage system during the indexing process.

[0074] Figure 4 FIG. 1 is a schematic diagram of an index service cache and pre-fetching workflow in a data detection method according to an optional embodiment of the present invention. Figure 4As shown in the figure, after the digital fingerprint list for index lookup is assigned to a cache and prefetch node, the processing flow is as follows: For each digital fingerprint not marked as new, the digital fingerprint is first searched in the cache. If a match is found, the data block already exists in the AegeanStore (cloud storage system). The file's digital fingerprint list and the locality list in the cache are merged, and the block is marked as existing in the result. If a match is not found, the distributed hash table is searched. If a match is found, the locality list stored in the distributed hash table is added to the cache, completing the prefetch process. The subsequent processing is the same as for a cache hit. If a match is not found, the data block is not in the AegeanStore (cloud storage system), resulting in a false positive. AegeanStore refers to a distributed redundancy removal storage system in a wide area network environment, i.e., a cloud storage system. Removing redundant data before uploading it to the AegeanStore can improve storage and network resource utilization. A distributed hash table (DHT) is a list of digital fingerprints corresponding to file data cached in all nodes in the cloud storage system, including a list corresponding to each node. In addition, when the cache is full, the LRU (Least Recently Used) cache replacement strategy can be used to update the locality list of the digital fingerprint replaced from the cache into the distributed hash table. Service nodes in multiple caches and prefetch modules may process the same digital fingerprint simultaneously, resulting in inconsistent locality lists. However, the locality list is only used during prefetching and will not affect the correctness of the indexing service.

[0075] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0076] Through the description of the above embodiments, those skilled in the art can clearly understand that the data detection method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0077] According to an embodiment of the present invention, a data detection device for implementing the above data detection method is also provided. Figure 5 is a structural block diagram of a data detection device provided according to an embodiment of the present invention, such as Figure 5 As shown, the data detection device includes: an acquisition module 51, an aggregation module 52, a first generation module 53, a second generation module 54, a judgment module 55 and a merging module 56. The data detection device is described below.

[0078] The acquisition module 51 is configured to acquire file data stored in the cloud storage system and file data cached in the first node, wherein the first node is located in the cloud storage system.

[0079] The aggregation module 52 is connected to the acquisition module 51 and is used to aggregate the data that does not have locality in the file data stored in the cloud storage system to obtain a target data set.

[0080] The first generating module 53 is connected to the aggregating module 52 and is configured to generate a first list based on the target data set, wherein the first list includes digital fingerprints corresponding to the file data in the target data set.

[0081] The second generating module 54 is connected to the first generating module 53 and is configured to generate a second list based on the file data cached in the first node, wherein the second list includes digital fingerprints corresponding to the file data cached in the first node.

[0082] The judging module 55 is connected to the second generating module 54 and is used to judge whether the digital fingerprint in the second list is in the first list.

[0083] The merging module 56 is connected to the judging module 55 , and merges the file data cached in the first node into the cloud storage system if the first digital fingerprint exists in the first list in the second list.

[0084] Optionally, the acquisition module may be configured to select the first node according to a resource scheduling identifier method before acquiring the file data cached in the first node.

[0085] Optionally, the aggregation module can be used to calculate the false positive rate of the target data set after obtaining the target data set, where the false positive rate represents the probability of errors in the aggregation result; when the false positive rate is lower than a preset threshold, the target data set is determined to be correct.

[0086] Optionally, the merging module may be configured to store the portion of the file data cached in the first node that is different from the file data in the cloud storage system into the cloud storage system.

[0087] Optionally, the device further includes a module that can be used for judgment, including: an acquisition unit, a generation unit, a judgment unit, and a merging unit. The acquisition unit is used to acquire the digital fingerprint corresponding to the file data cached in the nodes other than the first node in the cloud storage system; the generation unit is connected to the acquisition unit and is used to generate a list corresponding to the node based on the digital fingerprint corresponding to the file data cached in the node; the judgment unit is connected to the generation unit and is used to judge whether the digital fingerprint in the list corresponding to the node is in the first list; the merging unit is connected to the judgment unit and is used to merge the file data cached in the node corresponding to the second digital fingerprint into the cloud storage system if there is a second node in the node and the list corresponding to the second node has a second digital fingerprint in the first list.

[0088] Optionally, the apparatus further comprises a storage module configured to store the first node and the file data cached in the node into a cloud storage system when none of the digital fingerprints in the list corresponding to the node are located in the first list.

[0089] It should be noted that the acquisition module 51, aggregation module 52, first generation module 53, second generation module 54, judgment module 55, and merging module 56 correspond to steps S201 to S206 in the embodiment. The examples and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in the embodiment.

[0090] An embodiment of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one of a plurality of network devices in a computer network. The computer device includes a memory and a processor.

[0091] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the data detection method and device in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned data detection method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.

[0092] An embodiment of the present invention provides a data detection method, which obtains file data stored in a cloud storage system and file data cached in a first node, wherein the first node is located in the cloud storage system; aggregates data that does not exist locally in the file data stored in the cloud storage system to obtain a target data set; generates a first list based on the target data set, wherein the first list includes digital fingerprints corresponding to the file data in the target data set; generates a second list based on the file data cached in the first node, wherein the second list includes digital fingerprints corresponding to the file data cached in the first node; determines whether the digital fingerprint in the second list is located in the first list; if the first digital fingerprint in the second list is located in the first list, merges the file data cached in the first node into the cloud storage system, thereby achieving the purpose of comparing the file data in the cache with the file data already existing in the cloud storage system and avoiding storing the same data in the cloud storage system again, thereby achieving the technical effect of saving data storage space, and further solving the technical problem of low storage space utilization due to duplicate data in the cloud storage system.

[0093] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a non-volatile storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0094] The embodiment of the present invention further provides a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the data detection method provided in the embodiment.

[0095] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0096] An embodiment of the present invention further provides a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the data detection method provided in the above embodiment can be implemented.

[0097] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0098] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0100] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0101] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, and other media that can store program code.

[0103] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A data detection method, characterized in that: include: Acquire file data stored in a cloud storage system and file data cached in a first node, wherein the first node is located in the cloud storage system; Aggregating data that does not have locality among the file data stored in the cloud storage system to obtain a target data set; generating a first list based on the target data set, wherein the first list includes digital fingerprints corresponding to the file data in the target data set; generating a second list based on the file data cached in the first node, wherein the second list includes digital fingerprints corresponding to the file data cached in the first node; Determine whether the digital fingerprint in the second list is in the first list; In a case where the first digital fingerprint in the second list is located in the first list, the file data cached in the first node is merged into the cloud storage system.

2. The method according to claim 1, characterized in that Before obtaining the file data cached in the first node, the method includes: The first node is selected according to a resource scheduling identifier method.

3. The method according to claim 1, characterized in that After obtaining the target data set, the method includes: Calculating a false positive rate (FPR) of the target data set, wherein the FPR represents a probability that an aggregation result is erroneous; When the false positive rate is lower than a preset threshold, the target data set is determined to be correct.

4. The method according to claim 1, wherein The merging of the file data cached in the first node into the cloud storage system includes: The portion of the file data cached in the first node that is different from the file data in the cloud storage system is stored in the cloud storage system.

5. The method according to any one of claims 1 to 4, characterized in that When none of the digital fingerprints in the second list are in the first list, the method includes: Obtaining digital fingerprints corresponding to file data cached in nodes other than the first node in the cloud storage system; Generate a list corresponding to the node based on the digital fingerprint corresponding to the file data cached in the node; Determine whether the digital fingerprint in the list corresponding to the node is in the first list; If there is a second node among the nodes and a second digital fingerprint exists in the list corresponding to the second node and is located in the first list, the file data cached in the node corresponding to the second digital fingerprint is merged into the cloud storage system.

6. The method according to claim 5, characterized in that When none of the digital fingerprints in the list corresponding to the node is in the first list, the method includes: The first node and the file data cached in the node are stored in the cloud storage system.

7. A data detection device, characterized in that: include: an acquisition module, configured to acquire file data stored in the cloud storage system and file data cached in a first node, wherein the first node is located in the cloud storage system; an aggregation module, configured to aggregate data that does not have locality in the file data stored in the cloud storage system to obtain a target data set; A first generating module, configured to generate a first list based on the target data set, wherein the first list includes digital fingerprints corresponding to file data in the target data set; a second generating module, configured to generate a second list based on the file data cached in the first node, wherein the second list includes digital fingerprints corresponding to the file data cached in the first node; A judging module, configured to judge whether the digital fingerprint in the second list is in the first list; A merging module is configured to merge the file data cached in the first node into the cloud storage system when the first digital fingerprint in the second list is located in the first list.

8. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the data detection method according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: memory and processor, The memory stores a computer program; The processor is configured to execute a computer program stored in the memory, and when the computer program is run, the processor is enabled to execute the data detection method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data detection method according to any one of claims 1 to 6 is implemented.

Images