Cloud file query method and related device

Abstract

The embodiment of the application provides a cloud file query method and related equipment, and belongs to the technical field of data query. The method comprises the following steps: performing encryption processing on a first service pool and a first full pool respectively by using a preset encryption algorithm to obtain a corresponding second service pool and a second full pool; wherein the preset encryption algorithm comprises a reversible encoding algorithm; constructing a preset Bloom filter by using a preset non-encryption hash function according to the second full pool; performing comparison and analysis on the second service pool by using the preset Bloom filter to determine a target file identifier; wherein the target file identifier comprises a file identifier that exists in the second service pool and does not exist in the second full pool; performing decryption processing on the target file identifier to obtain a target query result; wherein the decryption processing corresponds to the encryption processing. The embodiment of the application can effectively improve the efficiency and accuracy of the mainless file check, and provides support for performance optimization of cloud storage services.

Description

Technical Field

[0001] This application relates to the field of data query technology, and in particular to a cloud file query method and related equipment. Background Technology

[0002] With the rapid development of big data and cloud computing, various public and private clouds have become indispensable parts of people's lives and businesses' development. Correspondingly, to better provide services, the primary method is to maintain the uniqueness of the ever-growing number of data files through indiscriminate encoding. Among these, files come in a wide variety of formats and exist in massive quantities; this type of encoding used to identify file uniqueness is collectively referred to as object IDs. In a database, object IDs act like an index in a dictionary, corresponding to information such as the storage location and size of related files, and user operations are also related to object IDs. However, with the accumulation of users and billions of file operations per day, the underlying storage update rate cannot keep up with the object ID update rate, resulting in a large number of ownerless files in the underlying storage. These files have been deleted from the user list but still occupy disk storage, causing a waste of storage costs. Related technologies mainly identify the differences between a full pool of all object IDs in storage and a business pool grouped by business or region through a cyclical process. However, this method often consumes a lot of resources and time and may even lead to cluster crashes.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a cloud file query method and related equipment, which can effectively improve the efficiency and accuracy of ownerless file clearing and provide support for the performance optimization of cloud storage services.

[0005] To achieve the above objectives, one aspect of this application proposes a cloud file query method, the method comprising: The first service pool and the first full pool are encrypted using a preset encryption algorithm to obtain the corresponding second service pool and second full pool; wherein the preset encryption algorithm includes a reversible encoding algorithm. A preset Bloom filter is constructed based on the second full pool using a preset non-cryptographic hash function; The target file identifier is determined by comparing and analyzing the second service pool using the preset Bloom filter; wherein, the target file identifier includes file identifiers that exist in the second service pool but do not exist in the second full pool. The target file identifier is decrypted to obtain the target query result; wherein the decryption process corresponds to the encryption process.

[0006] In some embodiments, the step of encrypting the first service pool and the first full pool using a preset encryption algorithm to obtain the corresponding second service pool and second full pool includes: The preset file identifier of the first service pool is encoded using the Base64 encoding algorithm to obtain the second service pool; The preset file identifier of the first full pool is encoded using the Bsae64 encoding algorithm to obtain the second full pool.

[0007] In some embodiments, constructing a preset Bloom filter based on the second full pool using a preset non-cryptographic hash function includes: Construct the first Bloom filter based on the Murmur hash function; The first Bloom filter sets the target bit array length based on the preset false positive rate and expected number of elements, and then sets the target hash count based on the target bit array length to construct the second Bloom filter. The preset file identifiers of the second full pool are mapped to the bit array of the second Bloom filter to construct the preset Bloom filter.

[0008] In some embodiments, the step of performing comparative analysis on the second service pool using the preset Bloom filter to determine the target file identifier includes: Calculate the hash value of each preset file identifier in the second service pool to obtain preset hash data; The target file identifier is determined by performing bit comparison analysis on the preset hash data using the preset Bloom filter.

[0009] In some embodiments, the decryption of the target file identifier to obtain the target query result includes: The target file identifier is decoded using the Base64 decoding algorithm to obtain plaintext data of the text identifier; Based on the plaintext data of the text identifier, the target file without a master is determined, and the target query result is obtained.

[0010] In some embodiments, after performing the decryption process on the target file identifier to obtain the target query result, the method further includes: Based on the target query results, text deletion is performed using preset database operations to obtain data optimization results; A resource optimization report is generated based on the data optimization results and the target query results.

[0011] To achieve the above objectives, another aspect of this application provides a cloud file query device, the device comprising: The first module is used to encrypt the first service pool and the first full pool respectively using a preset encryption algorithm to obtain the corresponding second service pool and second full pool; wherein, the preset encryption algorithm includes a reversible encoding algorithm; The second module is used to construct a preset Bloom filter based on the second full pool using a preset non-cryptographic hash function. The third module is used to perform comparative analysis on the second service pool through the preset Bloom filter to determine the target file identifier; wherein, the target file identifier includes file identifiers that exist in the second service pool but do not exist in the second full pool; The fourth module is used to decrypt the target file identifier to obtain the target query result; wherein the decryption process corresponds to the encryption process.

[0012] To achieve the above objectives, another aspect of this application provides an electronic device, the electronic device comprising: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0014] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method. The embodiments of this application include at least the following beneficial effects: This application provides a cloud file query method, apparatus, electronic device, storage medium, and program product. This scheme encrypts a first business pool and a first full pool using a preset encryption algorithm to obtain a corresponding second business pool and a second full pool. Then, a preset Bloom filter is constructed based on the second full pool using a preset unencrypted hash function. Next, the embodiments of this invention perform comparative analysis on the second business pool using the preset Bloom filter to determine the target file identifier, i.e., the file identifier that exists in the second business pool but does not exist in the second full pool. The target file identifier is then decrypted according to the encryption process to obtain the target query result, achieving fast and accurate querying of ownerless files. It is easily understood that the embodiments of this invention encrypt the first business pool and the first full pool using a reversible encoding algorithm, combined with the comparative analysis using a preset Bloom filter optimized by a preset unencrypted hash function. This approach ensures high accuracy while effectively saving time and resources for ownerless file queries, thereby effectively improving the efficiency and accuracy of ownerless file retrieval and providing support for performance optimization of cloud storage services. Attached Figure Description

[0015] Figure 1 This is a flowchart of the steps of the cloud file query method provided in the embodiments of the present invention; Figure 2 This is a flowchart of the steps provided by an embodiment of the present invention to encrypt the first service pool and the first full pool respectively using a preset encryption algorithm to obtain the corresponding second service pool and the second full pool; Figure 3 This is a flowchart of the steps for constructing a preset Bloom filter based on a second full pool using a preset unencrypted hash function, as provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the principle of the Bloom filter provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of a Bloom filter misjudgment provided in an embodiment of the present invention; Figure 6 This is a flowchart of the steps for determining the target file identifier by comparing and analyzing the second service pool through a preset Bloom filter, as provided in an embodiment of the present invention. Figure 7 This is a flowchart of the steps for decrypting the target file identifier to obtain the target query result, provided in an embodiment of the present invention. Figure 8 This is a flowchart illustrating the steps involved in generating a data optimization and resource optimization report, as provided in this embodiment of the invention. Figure 9 This is a schematic diagram of the cloud file query process architecture provided in an embodiment of the present invention; Figure 10This is a schematic diagram of the cloud file query device provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0017] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0018] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0021] A Bloom filter is a highly space-efficient probabilistic data structure used to quickly determine whether an element might exist in a set. Its core consists of a bit array and a hash function.

[0022] A hash function is a function that converts input data of arbitrary length (such as strings, files, etc.) into a fixed-length output (called a hash value or hash code) using a specific algorithm. The same input always generates the same hash value, and the output length is fixed. It has low time complexity and can be widely used in data storage, encryption, data integrity verification, and other fields.

[0023] With the rapid development of big data and cloud computing, various public and private clouds have become indispensable parts of people's lives and businesses' development. Correspondingly, to better provide services, the primary method is to maintain the uniqueness of the ever-growing number of data files through indiscriminate encoding. Among these, files come in a wide variety of formats and exist in massive quantities; this type of encoding used to identify file uniqueness is collectively referred to as object IDs. In a database, object IDs act like an index in a dictionary, corresponding to information such as the storage location and size of related files, and user operations are also related to object IDs. However, with the accumulation of users and billions of file operations per day, the underlying storage update rate cannot keep up with the object ID update rate, resulting in a large number of ownerless files in the underlying storage. These files have been deleted from the user list but still occupy disk storage, causing a waste of storage costs. Related technologies mainly identify the differences between a full pool of all object IDs in storage and a business pool grouped by business or region through a cyclical process. However, this method often consumes a lot of resources and time and may even lead to cluster crashes.

[0024] In view of this, this application provides a cloud file query method, apparatus, electronic device, storage medium, and program product. This scheme encrypts a first business pool and a first full pool using a preset encryption algorithm to obtain a corresponding second business pool and a second full pool. Then, a preset Bloom filter is constructed based on the second full pool using a preset unencrypted hash function. Next, this embodiment compares and analyzes the second business pool using the preset Bloom filter to determine the target file identifier—that is, the file identifier that exists in the second business pool but not in the second full pool. The target file identifier is then decrypted according to the encryption process to obtain the target query result. This enables fast and accurate querying of ownerless files, effectively improving the efficiency and accuracy of ownerless file retrieval and providing support for performance optimization of cloud storage services.

[0025] The cloud file query method provided in this application relates to the field of data query technology. The cloud file query method provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the cloud file query method, but is not limited to the above forms.

[0026] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0027] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0028] Figure 1 This is an optional flowchart of the cloud file query method provided in the embodiments of this application. Figure 1The method may include, but is not limited to, steps S110 to S140.

[0029] Step S110: Encrypt the first service pool and the first full pool using a preset encryption algorithm to obtain the corresponding second service pool and second full pool. The preset encryption algorithm includes a reversible encoding algorithm.

[0030] Step S120: Construct a preset Bloom filter based on the second full pool using a preset non-cryptographic hash function.

[0031] Step S130: Perform comparative analysis on the second service pool using a preset Bloom filter to determine the target file identifier. The target file identifier includes file identifiers that exist in the second service pool but not in the second full pool.

[0032] Step S140: Decrypt the target file identifier to obtain the target query result; wherein, the decryption process corresponds to the encryption process.

[0033] In this specific embodiment, the invention first encrypts the first business pool and the first full pool using a preset encryption algorithm to obtain the corresponding second business pool and second full pool. Specifically, the preset encryption algorithm in this embodiment includes reversible encoding algorithms, such as Hex encoding, Base32 encoding, or Base64 encoding. Correspondingly, this embodiment ultimately queries and returns the file identifier of the ownerless file, such as ObjectID, which is used to identify the uniqueness of the file. Therefore, this embodiment uses a reversible encoding algorithm to encode and encrypt the first business pool and the first full pool to obtain higher diversity during subsequent hash calculations, while ensuring consistency after decryption, so as to output the original file identifier. Correspondingly, in this embodiment, the first full pool refers to a resource pool that centrally stores all original data, and the first business pool refers to a storage resource pool dynamically allocated for a specific business or application, storing data directly related to the business, such as a business pool grouped by business or region. In this embodiment of the invention, the first business pool is mechanically encrypted using a preset encryption algorithm to obtain a second business pool, and the first full pool is also encrypted using the same preset encryption algorithm to obtain a second full pool, thus preprocessing the business pool and the full pool. Next, this embodiment constructs a preset Bloom filter based on the second full pool using a preset unencrypted hash function. Specifically, since traditional Bloom filters are highly efficient but also have a high probability of false positives, this embodiment optimizes the Bloom filter by combining it with a preset unencrypted hash function, thereby incorporating the data from the second full pool into the optimized Bloom filter, thus constructing the preset Bloom filter. Further, this embodiment uses the preset Bloom filter to perform comparative analysis on the second business pool to determine the target file identifier. Specifically, this embodiment uses the constructed preset Bloom filter to compare and analyze the data in the second business pool to identify files that exist in the second business pool but not in the second full pool (i.e., ownerless files in the full pool), and returns the corresponding file identifier as the target file identifier. This invention utilizes the characteristics of a Bloom filter—namely, comparison analysis determines that a non-existent file identifier definitely does not exist—and returns the corresponding target file identifier. This leverages the advantages of Bloom filters to effectively compress runtime and space requirements. Furthermore, by optimizing the Bloom filter using a preset unencrypted hash function, the accuracy of query identification is significantly improved. Finally, this invention decrypts the target file identifier to obtain the target query result. Specifically, the target file identifier returned in this invention is an encrypted file identifier preprocessed using a reversible encoding algorithm.Therefore, this embodiment of the invention performs decryption processing corresponding to encryption processing on the target file identifier, thereby decoding to obtain the plaintext file identifier, that is, the file identifier of the ownerless file, realizing accurate and fast query of the ownerless file and obtaining the target query result.

[0034] Reference Figure 2 To further improve the diversity obtained in subsequent hash calculations using a Bloom filter, in some embodiments of the present invention, the first business pool and the first full pool are encrypted using a preset encryption algorithm to obtain the corresponding second business pool and second full pool, including but not limited to the following steps: Step S210: Encode the preset file identifier of the first service pool using the Base64 encoding algorithm to obtain the second service pool.

[0035] Step S220: Encode the preset file identifier of the first full pool using the Bsae64 encoding algorithm to obtain the second full pool.

[0036] In this specific embodiment, the present invention uses the Base64 encoding algorithm as the preset encryption algorithm. Accordingly, the present invention encodes the preset file identifiers of the first service pool using the Base64 encoding algorithm to obtain the second service pool, and simultaneously encodes the preset file identifiers of the first full pool using the Base64 encoding algorithm to obtain the second full pool. Specifically, in this embodiment, the encryption process not only considers encryption efficiency, but also, since the final output is the object ID of the original text (ObjectID), i.e., the text identifier, it is necessary to ensure consistency after decryption, i.e., to consider the reversibility of encryption. Mainstream MD5 and SHA series encryption algorithms are irreversible; even if they could achieve higher encryption efficiency and lower collision rates in subsequent calculations, they should be discarded. The present invention fully considers the above requirements and uses Base64 to encrypt each text identifier in the full pool and the service pool, enabling higher diversity when performing hash calculations using a Bloom filter. For example, when using the Base64 encoding algorithm, the present invention first defines an array, as shown in the following formula:

[0037] In this embodiment of the invention, the above formula is used as the index table for the Base64 encoding algorithm. The characters selected are 64 printable characters: "AZ, az, 0-9, +, / ", which is specified by the standard BASE64 protocol. Additionally, in everyday use, the encoding result of this embodiment may also contain "=" or "==" signs. Here, "=" appears as a padding character.

[0038] For example, if a text identifier that needs to be encoded and encrypted using the Base64 encoding algorithm in an embodiment of the present invention is: eccc7247-9ad511e99835b8ee-6591991d, then the encrypted file identifier obtained is: ZWNjYzcyNDctOWFkNTExZTk5ODM1YjhlZS02NTkxOTkxZA==, which is more complex and has more bits than the original text identifier.

[0039] Reference Figure 3 In order to improve the accuracy of ownerless file queries while maintaining efficiency, in some embodiments of the present invention, a preset Bloom filter is constructed based on the second full pool using a preset unencrypted hash function, including but not limited to the following steps: Step S310: Construct the first Bloom filter based on the Murmur hash function.

[0040] Step S320: Based on the first Bloom filter, set the target bit array length according to the preset false positive rate and expected number of elements, and then set the target hash count according to the target bit array length to construct the second Bloom filter.

[0041] Step S330: Map each preset file identifier of the second full pool to the bit array of the second Bloom filter to construct the preset Bloom filter.

[0042] In this specific embodiment, the present invention first constructs a first Bloom filter based on the Murmur hash function. Specifically, a Bloom filter is a data structure whose underlying layer is actually a bitMap. When used, it consists of a long binary vector and a series of random mapping functions. In the best case, the time complexity of the Map is O(1) and the space complexity is O(n), thus it has the characteristics of low space consumption and high query efficiency, making it very suitable for fast querying of large datasets with no duplicates. The present invention uses the Bloom filter to determine whether the data in the business pool definitely does not exist or may exist in the full pool. Accordingly, in the application of the present invention, the file identifiers in the business pool and the full pool are actually 32-bit UUIDs separated by "-", and the characteristics of UUIDs ensure the uniqueness of file identifiers. In addition, as Figure 4As shown, the main principle of a Bloom filter is to use a set of hash functions to map elements to index positions in a set of bit arrays. To check if an element is in the set, the element is hashed, and then the value of the corresponding bit array is checked for 1. If all the bit array values ​​corresponding to the hash value are 1, then the element may be in the set; otherwise, the element is definitely not in the set. It's easy to understand that the advantage of a Bloom filter is that its space efficiency and query time far exceed those of general algorithms. The disadvantage is that it has a certain false positive rate (false positives, i.e., a Bloom filter reports that an element exists in a set, but in reality, the element is not in the set). Figure 5 As shown), but there are no incorrect cases (i.e., false negatives, where if an element is not actually in the set, the Bloom filter will not report that the element exists in the set, so there will be no false negatives). Although the Bloom filter is very efficient, with both writing and judgment being O(1) and requiring very little storage space, in big data scenarios, as the number of binary sequences increases, the probability of misjudging whether a text identifier is in the file pool increases. The overall file pool can reach billions or tens of billions of records, so the probability of misjudgment is high. Accordingly, this embodiment of the invention needs to search for text identifiers of unowned files for subsequent resource optimization, such as performing deletion operations. Although this will not lead to the accidental deletion of user files, the accuracy will be greatly reduced due to the large size of the files. Based on the characteristics of big data clusters, compared to a single machine, this embodiment of the invention allows for the sacrifice of some space to achieve higher accuracy while maintaining efficiency. According to the characteristics of the Bloom filter, this embodiment of the invention first reduces the probability of misjudgment in the following two ways: a. Increase the length of the bit array. The longer the array, the lower the probability of duplicate calculated indices. Correspondingly, if the size of the Bloom filter is m, the false positive rate is p, the number of hashes is k, and the number of elements to be inserted is n, then their relationship is as follows:

[0043] b. Increase the number of hash functions. With a single hash function, this embodiment of the invention determines existence based on a single result. However, with multiple hash functions, this embodiment only needs one calculated index with a value of 0 to determine that the element does not exist. This method effectively increases the probability of 0 appearing.

[0044] Next, this embodiment of the invention optimizes the Bloom filter using a hash function with a lower collision rate. The more complex the hash algorithm, the lower the probability of duplicates. It's easy to understand that although the false positive rate can be reduced in the above ways, neither the length of the bit array nor the number of hash functions can be increased indefinitely, and the accuracy and efficiency still cannot meet expectations when dealing with massive amounts of data. Therefore, this embodiment of the invention optimizes the data calculation process based on the above characteristics. The default hash function used in Java is the BKDRHash function, which has significant advantages for hashing strings compared to other hash algorithms. The BKDRHash function is outstanding in both practical and coding implementation; however, its obfuscation is insufficient. For example, without overflow, when the value of the last byte is incremented by 1, the hash value is only incremented by 1. However, when the input data volume is very large and the length is insufficient (e.g., no more than 30 bits), overflow will occur, and the overflow will greatly increase the probability of collision. In this embodiment of the invention, although the length of the encrypted string has reached 64 bits, the input data still far exceeds the level in general cases, such as reaching hundreds of millions or even billions. Therefore, this embodiment of the invention must select a hash algorithm with stronger obfuscation capabilities while also considering efficiency.

[0045] Accordingly, this embodiment of the invention optimizes the Bloom filter using the Murmur hash function to obtain a first Bloom filter. Compared to other popular hash functions, the Murmur hash function exhibits better random distribution characteristics for keys with strong regularity. When used in hash tables, Bloom filters, etc., elements are evenly distributed, effectively reducing the probability of collisions. The Murmur hash function is based on two core ideas: mixing and rotation. Mixing involves dividing the input data into several blocks, performing a hash operation on each block, and finally mixing the hash values ​​of all blocks together to generate the final hash value. Rotation involves cyclically shifting the hash value to increase its randomness and distribution.

[0046] The implementation process of the Murmur hash function in this embodiment of the invention can be divided into the following steps: (1) Initialize hash value: Use a random number as the initial hash value.

[0047] (2) Block hashing: Divide the input data into several blocks, perform a hash operation on each block, and generate a block hash value.

[0048] (3) Mixed hash: Mix the hash values ​​of all blocks together to generate a mixed hash value.

[0049] (4) Final hash: Rotate and mix the mixed hash value to generate the final hash value.

[0050] Compared to the default hash algorithm, the Murmur hash function is faster, has a lower hash collision rate, and a more uniform distribution, making it more suitable for the business scenarios in this embodiment of the invention. Therefore, considering the actual situation, this embodiment of the invention adopts the Murmur hash function as the hash method for the Bloom filter to optimize the construction of the first Bloom filter.

[0051] Next, in this embodiment of the invention, the target bit array length is set according to the first Bloom filter based on a preset false positive rate and the expected number of elements. Then, the target hash count is set according to the target bit array length to construct a second Bloom filter. Each preset text identifier in the second full pool is mapped to the bit array of the second Bloom filter to construct a preset Bloom filter. Specifically, this embodiment of the invention first estimates the number of text identifiers in the second full pool (expected number of elements) and sets an acceptable false positive rate (preset false positive rate) to calculate the target bit array length. Next, based on the calculated target bit array length and the estimated number of text identifiers in the second full pool, the embodiment of the invention calculates the target hash count, and then sets the first Bloom filter according to the target bit array length m and the target hash count k to construct the second Bloom filter. Next, for each file identifier in the second full pool, the embodiment of the invention calculates the target hash count k hash values ​​and sets the corresponding bits in the bit array to 1, thereby mapping each preset file identifier in the second full pool to the bit array to construct the preset Bloom filter.

[0052] Reference Figure 6 To further improve the accuracy of ownerless file queries, in some embodiments of the present invention, a pre-set Bloom filter is used to compare and analyze the second business pool to determine the target file identifier, including but not limited to the following steps: Step S410: Calculate the hash value of each preset file identifier in the second service pool to obtain preset hash data.

[0053] Step S420: Perform bit comparison analysis on the preset hash data using a preset Bloom filter to determine the target file identifier.

[0054] In this specific embodiment, the present invention first calculates the hash value of each preset file identifier in the second service pool to obtain preset hash data. Then, a preset Bloom filter is used to perform bit comparison analysis on the preset hash data to determine the target file identifier. Specifically, the present invention first traverses the second service pool and calculates the hash value of each file identifier in the second service pool to obtain preset hash data. Accordingly, the present invention performs bit checks on the calculated preset hash data. For example, when k hash values ​​are calculated, if all k positions are 1, the file identifier is marked as possibly existing; otherwise, if any bit is 0, the file identifier is marked as definitely not existing, thereby determining the target file identifier that exists in the second service pool but does not exist in the second full pool. It is easy to understand that the preset Bloom filter in the present invention maps the file identifiers in the second full pool to a bit pattern through a combination of bit array and hash function. Then, during comparison analysis, by checking whether the file identifiers in the second service pool can be mapped to the same bit pattern, the present invention determines the target file identifier that exists in the second service pool but does not exist in the second full pool, achieving fast existence determination by trading space for time.

[0055] Reference Figure 7 To ensure the consistency of the file identifiers obtained from the query, in some embodiments of the present invention, the target file identifier is decrypted to obtain the target query result, including but not limited to the following steps: Step S510: Decode the target file identifier using the Base64 decoding algorithm to obtain plaintext data of the text identifier.

[0056] Step S520: Determine the target ownerless file based on the plaintext data of the text identifier, and obtain the target query results.

[0057] In this specific embodiment, the present invention first decodes the target file identifier using a Base64 decoding algorithm to obtain plaintext data of the text identifier. Then, based on the plaintext data of the text identifier, the target ownerless file is determined, resulting in the target query result. Specifically, to improve the diversity of hash calculations performed by the Bloom filter, the file identifiers in the first full pool are encoded and encrypted using a Base64 encoding algorithm during the preprocessing stage. Therefore, the target file identifier determined by comparison and analysis using the optimized preset Bloom filter is the encrypted file identifier. Thus, the present invention decodes the target file identifier using a decoding algorithm corresponding to the preset encryption algorithm to obtain plaintext data of the text identifier. Next, the present invention determines the corresponding file, i.e., the target ownerless file, based on the decoded plaintext text identifier, completing the query for ownerless files in the full pool and obtaining the target query result.

[0058] Reference Figure 8 To improve the performance of cloud storage services, in some embodiments of the present invention, after performing decryption processing on the target file identifier to obtain the target query result, the cloud file query method provided by the embodiments of the present invention further includes, but is not limited to, the following steps: Step S610: Based on the target query results, perform text deletion processing through preset database operations to obtain data optimization results.

[0059] Step S620: Generate a resource optimization report based on the data optimization results and the target query results.

[0060] In this specific embodiment, the present invention first performs text deletion processing based on the target query results through preset database operations to obtain data optimization results, and then generates a resource optimization report based on the data optimization results and the target query results. Specifically, after quickly and efficiently obtaining the file identifiers of ownerless (invalid) files in cloud storage through a preset Bloom filter, the present invention has the DBA (Database Administrator) perform database-related file deletion operations to delete invalid files in the full pool, thereby freeing up storage space, reducing resource consumption, and improving system performance and response speed. Next, the present invention analyzes the data optimization results and the target query results to generate a resource optimization report, providing support for resource optimization decisions, so as to iteratively optimize the cleanup rules by analyzing historical data in the report.

[0061] The following section provides a detailed introduction and explanation of the solution in this embodiment of the invention, using a specific cloud file query scenario as an example: For example, refer to Figure 9 , Figure 9This is a schematic diagram of the cloud file query process provided in an embodiment of the present invention. Specifically, this embodiment of the present invention needs to query ownerless files in the full pool B. Accordingly, this embodiment of the present invention first encodes and encrypts each ObjectID (file identifier) ​​in business pool A and full pool B using the Base64 encoding algorithm. Next, this embodiment of the present invention optimizes the Bloom filter using the Murmur hash algorithm and incorporates the encoded and encrypted file identifiers of full pool B into the Bloom filter to construct a preset Bloom filter. Next, this embodiment of the present invention compares the encoded and encrypted file identifier data of business pool A and full pool B using the constructed preset Bloom filter, and returns the ObjectID that exists in business pool A but not in full pool B. Accordingly, since the returned ObjectID has been encoded and encrypted using the Base64 encoding algorithm, this embodiment of the present invention decodes the returned ObjectID using the Base64 decoding algorithm to obtain the plaintext ObjectID, that is, the ObjectID of the ownerless file in full pool B, realizing fast and accurate querying of ownerless files, effectively improving the efficiency and accuracy of ownerless file clearing, and providing support for the performance optimization of cloud storage services.

[0062] Please see Figure 10 This application also provides a cloud file query device that can implement the above-described method. The device includes: The first module 710 is used to encrypt the first service pool and the first full pool respectively using a preset encryption algorithm to obtain the corresponding second service pool and second full pool. The preset encryption algorithm includes a reversible encoding algorithm.

[0063] The second module 720 is used to construct a preset Bloom filter based on the second full pool using a preset unencrypted hash function.

[0064] The third module 730 is used to perform comparative analysis on the second service pool using a preset Bloom filter to determine the target file identifier. The target file identifier includes file identifiers that exist in the second service pool but do not exist in the second full pool.

[0065] Module 740 is used to decrypt the target file identifier to obtain the target query result. The decryption process corresponds to the encryption process.

[0066] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0067] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0068] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0069] Please see Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 810 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 820 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 820 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 820 and is called and executed by the processor 810 using the methods described in the embodiments of this application. The input / output interface 830 is used to implement information input and output; The communication interface 840 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 850 transmits information between various components of the device (e.g., processor 810, memory 820, input / output interface 830, and communication interface 840); The processor 810, memory 820, input / output interface 830 and communication interface 840 are connected to each other within the device via bus 850.

[0070] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0071] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0072] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0073] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0074] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0075] The cloud file query method, apparatus, electronic device, storage medium, and program product provided in this application preprocesses file identifiers in a pool by comparing them, and then performs comparative analysis using a Bloom filter optimized by the Murmur hash function. This achieves efficient, accurate, and resource-efficient clearing of ownerless files, making the task universally applicable, saving enterprise resources, and improving cluster efficiency. In particular, this invention innovatively integrates dynamic Bloom filtering with hash bucketing technology. The dynamic Bloom filter enables initial rapid filtering of cloud files, dynamically adjusting the filter's capacity and error rate based on file metadata characteristics, effectively eliminating a large number of invalid candidate files and significantly reducing the size of the candidate set for subsequent queries. Simultaneously, a hash bucketing strategy is used to hierarchically store and index the filtered candidate files, constructing a multi-level hash bucketing structure based on key file feature values ​​to achieve precise positioning from global to local levels. Correspondingly, through the three-level collaborative mechanism of "dynamic filtering - hierarchical binning - precise matching", the embodiments of the present invention can significantly improve the query response speed of dynamically changing files in the cloud storage environment while ensuring query accuracy, and reduce the system's computing and storage overhead. It is especially suitable for efficient retrieval scenarios of ultra-large-scale cloud file systems, and provides strong technical support for the performance optimization of cloud storage services.

[0076] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0077] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0078] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0079] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0080] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0081] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0083] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0085] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A cloud file query method, characterized in that, The method includes the following steps: The first service pool and the first full pool are encrypted using a preset encryption algorithm to obtain the corresponding second service pool and second full pool; wherein the preset encryption algorithm includes a reversible encoding algorithm. A preset Bloom filter is constructed based on the second full pool using a preset non-cryptographic hash function; The target file identifier is determined by comparing and analyzing the second service pool using the preset Bloom filter; wherein, the target file identifier includes file identifiers that exist in the second service pool but do not exist in the second full pool. The target file identifier is decrypted to obtain the target query result; wherein the decryption process corresponds to the encryption process. The step of constructing a preset Bloom filter based on the second full pool using a preset non-cryptographic hash function includes: Construct the first Bloom filter based on the Murmur hash function; The first Bloom filter sets the target bit array length based on the preset false positive rate and expected number of elements, and then sets the target hash count based on the target bit array length to construct the second Bloom filter. The preset file identifiers of the second full pool are mapped to the bit array of the second Bloom filter to construct the preset Bloom filter.

2. The method according to claim 1, characterized in that, The step of encrypting the first service pool and the first full pool using a preset encryption algorithm to obtain the corresponding second service pool and second full pool includes: The preset file identifier of the first service pool is encoded using the Base64 encoding algorithm to obtain the second service pool; The preset file identifier of the first full pool is encoded using the Base64 encoding algorithm to obtain the second full pool.

3. The method according to claim 1, characterized in that, The step of comparing and analyzing the second service pool using the preset Bloom filter to determine the target file identifier includes: Calculate the hash value of each preset file identifier in the second service pool to obtain preset hash data; The target file identifier is determined by performing bit comparison analysis on the preset hash data using the preset Bloom filter.

4. The method according to claim 2, characterized in that, The process of decrypting the target file identifier to obtain the target query result includes: The target file identifier is decoded using the Base64 decoding algorithm to obtain plaintext data of the text identifier; Based on the plaintext data of the text identifier, the target ownerless file is determined, and the target query result is obtained.

5. The method according to claim 1, characterized in that, After performing the decryption process on the target file identifier to obtain the target query result, the process further includes: Based on the target query results, text deletion is performed using preset database operations to obtain data optimization results; A resource optimization report is generated based on the data optimization results and the target query results.

6. A cloud file query device, characterized in that, The device includes: The first module is used to encrypt the first service pool and the first full pool respectively using a preset encryption algorithm to obtain the corresponding second service pool and second full pool; wherein, the preset encryption algorithm includes a reversible encoding algorithm; The second module is used to construct a preset Bloom filter based on the second full pool using a preset non-cryptographic hash function. The third module is used to perform comparative analysis on the second service pool through the preset Bloom filter to determine the target file identifier; wherein, the target file identifier includes file identifiers that exist in the second service pool but do not exist in the second full pool; The fourth module is used to decrypt the target file identifier to obtain the target query result; wherein the decryption process corresponds to the encryption process. The step of constructing a preset Bloom filter based on the second full pool using a preset non-cryptographic hash function includes: Construct the first Bloom filter based on the Murmur hash function; The first Bloom filter sets the target bit array length based on the preset false positive rate and expected number of elements, and then sets the target hash count based on the target bit array length to construct the second Bloom filter. The preset file identifiers of the second full pool are mapped to the bit array of the second Bloom filter to construct the preset Bloom filter.

7. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images