Data processing method, computing device and computer-readable storage medium

Abstract

The embodiments of the present invention provide a data processing method, a computing device and a computer-readable storage medium. The data processing method comprises: determining a data fingerprint to be verified, wherein the data fingerprint is an identifier used for distinguishing between different pieces of data content; from an index table of the data fingerprint, acquiring metadata corresponding to the data fingerprint, wherein the index table is configured to store an index entry of the data fingerprint and the metadata corresponding to the data fingerprint, and the metadata is metadata of a file that references the data fingerprint; and using the acquired metadata to locate an actually present file to obtain a verification result for the consistency between the data fingerprint and the actually present file.

Description

[0001]TECHNICAL FIELD: Embodiments of the present invention relate to the field of computer technology, and more particularly to data processing methods, computing devices, and computer-readable storage media. Background: Cloud disk services provide storage resources on demand and in an easily scalable manner over the internet. Cloud disks enable storage services such as personal network disks, enterprise network disks, and cloud photo albums. Cloud disks provide storage resources for businesses or individuals, enabling customers to enjoy fast, efficient, and massively scalable services. Systems that manage hundreds of billions of user files on cloud disks focus on securely and efficiently storing and accessing large amounts of data. However, current data index structures for cloud disk files have shortcomings in performance and data security, impacting user experience. SUMMARY OF THE INVENTION: In light of this, embodiments of the present invention provide a data processing method. One or more embodiments of the present invention also relate to a computing device, a computer-readable storage medium, and a computer program product to address the technical deficiencies of prior art data index structures, such as insufficient performance and security. According to a first aspect of an embodiment of the present invention, a data processing method is provided, comprising: determining a data fingerprint to be verified, wherein the data fingerprint is an identifier used to distinguish different data contents; obtaining metadata corresponding to the data fingerprint from an index table of the data fingerprint, wherein the index table is used to store index items of the data fingerprint and metadata corresponding to the data fingerprint, wherein the metadata is metadata of a file that references the data fingerprint; and locating an actual file using the obtained metadata to obtain a consistency verification result between the data fingerprint and the actual file. According to a second aspect of an embodiment of the present invention, a computing device is provided, comprising: a memory and a processor; the memory is configured to store a computer program / instructions, and the processor is configured to execute the computer program / instructions, wherein the computer program / instructions, when executed by the processor, implement the steps of the data processing method described in any embodiment of the present invention. According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer program / instructions are stored, wherein the computer program / instructions, when executed by the processor, implement the steps of the data processing method described in any embodiment of the present invention. According to a fourth aspect of an embodiment of the present invention, a computer program product is provided, comprising a computer program / instructions, wherein the computer program / instructions, when executed by the processor, implement the steps of the data processing method described in any embodiment of the present invention.One embodiment of the present invention implements a data processing method. The method determines a data fingerprint to be verified and obtains metadata corresponding to the data fingerprint from an index table of the data fingerprint. The index table is used to store index items of the data fingerprint and metadata corresponding to the data fingerprint. The metadata is metadata of a file that references the data fingerprint. The obtained metadata is used to locate an actual file and obtain a consistency verification result between the data fingerprint and the actual file. As can be seen, by backing up the metadata of the file that references the data fingerprint in the index table, the method provides the index table with self-describing file capabilities. That is, the index table itself can locate the ownership of data. If an actual file is located, the data fingerprint is consistent with the actual file; otherwise, they are inconsistent. This allows for rapid verification of the consistency between the data fingerprint and the actual file, achieving faster data consistency verification, effectively improving indexing performance, and ensuring data security. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a schematic diagram of a data processing method provided by an embodiment of the present invention in a cloud storage application scenario; Figure 2 is a flow chart of a data processing method provided by an embodiment of the present invention; Figure 3a is a schematic diagram of an index table provided by an embodiment of the present invention; Figure 3b is a schematic diagram of the relationship between a file metadata table and an index table provided by an embodiment of the present invention; Figure 4 is a flow chart of the file creation process of a data processing method provided by an embodiment of the present invention; Figure 5 is a flow chart of the file deletion process of a data processing method provided by an embodiment of the present invention; Figure 6 is a flow chart of the garbage collection process of a data processing method provided by an embodiment of the present invention; Figure 7 is a flow chart of the table content change notification channel-based process of a data processing method provided by an embodiment of the present invention; Figure 8 is a schematic diagram of the structure of a data processing device provided by an embodiment of the present invention; Figure 9 is a block diagram of the structure of a computing device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS The following description sets forth numerous specific details to facilitate a thorough understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar generalizations without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The terminology used in one or more embodiments of the present invention is for the purpose of describing specific embodiments only and is not intended to limit the present invention. As used in one or more embodiments of the present invention and the appended claims, the singular forms "a," "an," "said," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.It should also be understood that the terms "and / or" used in one or more embodiments of the present invention refer to and include any or all possible combinations of one or more associated listed items. It should be understood that although the terms "first", "second", etc. may be used to describe various information in one or more embodiments of the present invention, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present invention, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to a determination." In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals are provided for the user to choose to authorize or refuse. First, the terminology involved in one or more embodiments of the present invention is explained. Cloud disk, It provides users with information storage, access, and download services via the internet, featuring massive storage capabilities. Online disks and cloud photo albums provide users with file and photo storage, access, and download services via the internet, and can also implement automatic upload, automatic synchronization, and easy sharing, featuring massive storage capabilities. File metadata contains the mapping between the file's logical address in logical space and its actual physical storage location. Garbage collection (GC) is a mechanism used to periodically reclaim space occupied by unreferenced objects during idle time, freeing up space occupied by garbage data. A file metadata table is a database table used to store file metadata. File metadata can include, for example, a unique file identifier, creation time, modification time, creator, file size, file type, and other basic file information. A data fingerprint is an identifier calculated based on the file's data content, used to distinguish different data contents and uniquely identify the corresponding data content. In other words, any data content in a file has a unique data fingerprint. Different from the data fingerprint of other data contents, the index table of the data fingerprint is used to store the primary key, which is used to point to the data fingerprint.An index entry is a record in the index table that stores the primary key, also known as a sentinel fingerprint index entry, and is used for data fingerprint queries. A reference entry is a record in the index table that stores the reference relationship between a file and a data fingerprint. A reference relationship between a file and a data fingerprint indicates that the file references the file data pointed to by the data fingerprint. A reference count indicates how many files reference a data fingerprint. The table content change notification channel is a notification mechanism that notifies users of any modification operations to rows within a table in the order in which the changes were made. The scale of file data managed by cloud disks has reached hundreds of billions, making efficient access, inspection, and data verification crucial. Currently, the data index structure of cloud disk files has poor verification capabilities and cannot meet customer access performance and data security requirements. Therefore, there is an urgent need for data processing methods that can efficiently verify data indexes. In view of this, the present invention provides a data processing method. This method backs up the metadata of files referencing data fingerprints in an index table, enabling the index table to have self-describing file capabilities. Specifically, the index table itself can locate the ownership of data, thereby enabling rapid data verification and effectively ensuring client access performance and data security. Specifically, the present invention provides a data processing method, which also relates to a data processing apparatus, a computing device, and a computer-readable storage medium, each of which will be described in detail in the following embodiments. Referring to FIG1 , FIG1 shows a schematic diagram of a data processing method provided according to one embodiment of the present invention in a cloud storage application scenario. As shown in FIG1 , the cloud includes a computing cluster and a storage cluster. The computing cluster includes one or more cloud servers. The storage cluster includes one or more storage nodes for supporting storage on the cloud servers' cloud disks. The cloud servers' cloud disks can be configured to implement storage such as network disks and cloud photo albums. Based on the cloud storage service provided by the cloud, a user can send a request to the cloud to access data. In response to the request, one or more cloud servers in the cloud computing cluster access the cloud disk to obtain the corresponding file data and return the file data to the user. A cloud disk is a virtual storage device for file-level data provided by the cloud. For external access, the file-level data storage provided by the cloud disk can present a storage experience similar to that of a local file system. The underlying layer of a cloud disk can be a physical block storage device such as a disk. Just like a disk, users can perform operations such as partitioning, formatting, and creating file systems on the cloud disk mounted to the cloud server, as well as persistently storing data. Unlike disks, this persistent storage requires no additional user intervention.A cloud server manages files on a cloud disk based on metadata. File data content is uniquely identified by a data fingerprint, which is quickly queried using an index table. The index table stores index items for the data fingerprint. These index items store primary keys, which point to data fingerprints. In the method provided in an embodiment of the present invention, the index table not only stores index items for the data fingerprint, but also stores metadata corresponding to the data fingerprint. This metadata is metadata for files that reference the data fingerprint. Therefore, the cloud server's data verification process can include: determining the data fingerprint to be verified; obtaining metadata corresponding to the data fingerprint from the data fingerprint index table; this metadata is metadata for files that reference the data fingerprint; locating the actual file using the obtained metadata; and obtaining a consistency verification result between the data fingerprint and the actual file. In this application scenario, the data processing method provided in an embodiment of the present invention, by backing up the metadata of files that reference the data fingerprint in the index table, enables the index table to be self-describing. That is, the index table itself can locate the ownership of the data, thereby quickly achieving data consistency verification. It should be noted that the application scenario shown in FIG1 is merely illustrative of the method provided by an embodiment of the present invention and does not constitute a limitation of the method provided by an embodiment of the present invention. For example, according to the method provided by an embodiment of the present invention, the cloud disk may be a cloud disk of a cloud server configured to provide any cloud computing capabilities. The cloud server may be a distributed server cluster comprising multiple servers or a single server. Services provided by the cloud server may include cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms. Referring to FIG2 , FIG2 shows a flowchart of a data processing method provided according to an embodiment of the present invention, specifically comprising the following steps: Step 202: Determine a data fingerprint to be verified. The data fingerprint is an identifier used to distinguish different data contents. The data fingerprint is a unique identifier calculated based on the data content of a file and is used to uniquely identify the corresponding data content. For example, the data fingerprint may be calculated using algorithms including but not limited to SHA1 (Secure Hash Algorithm 1), MD5 (Message-Digest Algorithm), and the like.The data fingerprint is queried using index items in the index table. Therefore, determining the data fingerprint to be verified can also be understood as determining the index item corresponding to the verified data fingerprint. Step 204: Obtain metadata corresponding to the data fingerprint from the index table of the data fingerprint. The index table is used to store the index items of the data fingerprint and the metadata corresponding to the data fingerprint. The metadata is metadata of the file that references the data fingerprint. The index item is a record of the primary key of the data fingerprint stored in the index table. The index item is used to query the data fingerprint, and the primary key is used to point to the data fingerprint. For example, in the index table shown in Figure 3a, an index item (also called guard_fp_entry, sentinel index item) is used to store the primary key of a data fingerprint. In addition to index items, the index table may also include reference items, which are used to store the reference relationship between the file and the data fingerprint. The reference relationship between the file and the data fingerprint can be expressed as a correspondence between the file's unique identifier and the primary key of the data fingerprint. The metadata corresponding to the data fingerprint can be stored in the reference item corresponding to the primary key. Since the index table is used not only to store the index items of the data fingerprint but also to store the metadata corresponding to the data fingerprint, the metadata corresponding to the data fingerprint can be obtained from the index table of the data fingerprint. For example, in the index table shown in Figure 3a, assuming that the data fingerprint A to be verified is pointed to by primary key A1, after determining primary key A1, the metadata recorded in the reference item corresponding to primary key A1 can be obtained from the index table. Step 206: Use the obtained metadata to locate the actual file and obtain a consistency verification result between the data fingerprint and the actual file. Since the file metadata can be used to locate the ownership of the data, the obtained metadata can be used to verify the consistency between the data fingerprint and the actual file. The specific implementation method of the verification is not limited. For example, data content can be obtained through metadata, a data fingerprint can be constructed based on the data content, and the constructed data fingerprint can be compared with the data fingerprint corresponding to the index item to achieve consistency verification. For another example, the consistency between the data fingerprint and the actual file can be verified by comparing the metadata recorded in the file metadata table with the obtained metadata. This method backs up the metadata of files that reference data fingerprints in the index table, making the index table self-describing. That is, the index table itself can locate the ownership of the data, thereby quickly implementing data consistency verification, effectively ensuring customer access performance and data security.In one or more embodiments of the present invention, consistency verification between a data fingerprint and an actual file is achieved by inspecting a file metadata table. Specifically, using the obtained metadata to locate an actual file and obtaining a consistency verification result between the data fingerprint and the actual file includes: obtaining metadata to be verified for a file to be verified from a file metadata table, where the file to be verified is a file that references the data fingerprint; the file metadata table is used to store metadata for actual files; comparing the obtained metadata with the metadata to be verified to determine whether they are identical; if not, determining that an anomaly exists in the consistency between the data fingerprint and the actual file. The file to be verified can be any file in the file metadata table. For example, each file metadata row in the file metadata table can be traversed, and the metadata in each file metadata row can be used as the metadata to be verified for the file to be verified, thereby verifying the consistency between each data fingerprint and the actual file. In one or more embodiments of the present invention, in combination with the above embodiments, a primary key can be constructed based on the data content of the actual file to determine the data fingerprint to be verified. Specifically, determining the data fingerprint to be verified includes: obtaining data content from the file to be verified for use in constructing the data fingerprint; constructing a primary key for the data fingerprint to be queried based on the data content; and using the primary key of the data fingerprint to be queried, searching the index table of the data fingerprint for a data fingerprint corresponding to an index item that matches the primary key of the data fingerprint to be queried, as the data fingerprint to be verified, where the index item stores the primary key of the corresponding data fingerprint. The data content used to construct the data fingerprint refers to the data content uniquely identified by the data fingerprint in the file. The primary key, used to point to the data fingerprint, can be understood as the primary key of the database table used to store the data fingerprint. The primary key can be calculated based on the data content. For example, the primary key can be calculated based on the SHA1 value, MD5 value, and data size of the data content. Therefore, in the above embodiment, the primary key of the data fingerprint to be queried can be constructed based on the obtained data content. In the above embodiment, the relationship between the file metadata table and the index table is shown in the schematic diagram of the relationship between the file metadata table file and the index table fp shown in FIG3b : any row record in the file metadata table file is used to store the metadata of a file, and the reference relationship between the file and the data fingerprint is recorded in the index item of the index table fp. The reference relationship between the file and the data fingerprint can be represented by the correspondence between the file unique identifier and the primary key of the data fingerprint.Specifically, in Figure 3b, file_x represents the metadata row stored in the file metadata table file for a single file; fp_entry_x, also known as the reference entry, represents a reference to a data fingerprint by file file_x. A data fingerprint can be referenced by multiple files to support file-level data deduplication; multiple identical files will reference the same data fingerprint; and gurad_fp_entry, also known as the index entry, stores the primary key of the data fingerprint, which is used for querying the data fingerprint table. Therefore, the metadata to be verified for the file to be verified is obtained from the file metadata table file shown in Figure 3b. The primary key is constructed based on the data content obtained from the metadata. The index entry with a matching primary key can then be queried from the index table fp shown in Figure 3b. The data fingerprint pointed to by the primary key in the index entry is the data fingerprint to be verified. After determining the data fingerprint to be verified, the metadata to be verified obtained from the file metadata table can be compared with the metadata obtained from the index table to verify the consistency between the data fingerprint and the actual file. For example, in the index table shown in Figure 3b, the metadata obtained from the reference item is compared with the metadata to be verified. If the two are consistent, the verification result is passed, indicating that the data fingerprint is consistent with the actual file. Otherwise, a data consistency anomaly is determined. Taking the example of traversing each file metadata row in the file metadata table file and using the metadata in each file metadata row as the metadata to be verified for the file to be verified, to implement a patrol inspection of the file metadata table file, the processing process of the above embodiment may include: scanning the file table row by row, performing the following operations on each row of the file table: constructing a corresponding primary key for the metadata recorded in the current row using the data content corresponding to the metadata; searching the index table fp for a matching record based on the constructed primary key; and checking whether the metadata in the retrieved index item is consistent with the metadata recorded in the current row of the file table. If they are consistent, the data consistency is normal; otherwise, a data consistency anomaly is detected. It should be noted that the metadata recorded in the index item can be part of the key metadata of the file or all of the metadata, and this is not limited in the present invention, as long as the data correctness can be verified. For example, metadata recorded in an index item may include, but is not limited to, a file unique identifier, creator, creation time, size, etc. In one or more embodiments of the present invention, data consistency verification is implemented by patrolling the index table.Specifically, the data fingerprint corresponding to any index item in the index table is the data fingerprint to be verified; using the obtained metadata to locate an actual file and obtaining a consistency verification result between the data fingerprint and the actual file includes: using the obtained metadata to query a file metadata table for a record matching the metadata, the file metadata table being used to store metadata for the actual file; determining whether the obtained metadata is identical to the metadata stored in the record; if not, determining that there is an anomaly in the consistency between the data fingerprint and the actual file. For example, in the index table shown in FIG3b , the index table is traversed row by row, and the metadata obtained from the traversed reference item is compared with the metadata to be verified. If the two are consistent, the verification result is passed, indicating normal data consistency; otherwise, it is determined that there is an anomaly in the data consistency. Furthermore, when traversing the index table, index items can also be traversed to verify the primary keys in the index items. For example, an interface of the metadata index library can be called to obtain the corresponding metadata and verify whether the SHA1 value is consistent with the primary key in the index item. Taking the example of traversing each row in an index table fp and performing a patrol inspection on fp, the processing process in the above embodiment may include: scanning the table fp row by row, and performing the following operations on each row of table fp: If the traversed record is an index item, verifying whether the primary key of the index item record is correct; if the traversed record is a reference item, constructing the primary key of the file metadata table based on the metadata of the reference item record; querying the record in the file metadata table based on the constructed primary key of the file metadata table; and verifying whether the metadata of the file metadata table record and the metadata of the reference item record are consistent; if they are consistent, the data consistency is normal; otherwise, the data consistency is abnormal. In the above embodiment, because the index table contains metadata corresponding to the data fingerprint and has data self-describing capabilities, data patrol based on the index table and / or file metadata table can accurately locate actual files and perform data consistency verification, effectively reducing system resource consumption, ensuring system stability, and improving user experience. In one or more embodiments of the present invention, the reference relationship between the file and the data fingerprint is also written into an index table, so that the reference relationship and the data fingerprint are stored in the same table. In this way, the process of creating a file's data fingerprint index item can be implemented by accessing a single table, reducing table access, lowering resource consumption, and improving performance.Specifically, the index table also includes a reference entry for storing a reference relationship between a file and a data fingerprint. The method further includes: in response to a file creation request, determining whether the data fingerprint referenced by the file to be created corresponding to the file creation request exists in the index table; if so, writing the reference relationship between the file to be created and the data fingerprint into the reference entry in the index table; if not, creating an index entry corresponding to the data fingerprint referenced by the file to be created in the index table and writing the reference relationship between the file to be created and the data fingerprint into the reference entry in the index table. For example, the file creation request may include a unique identifier for the file to be created, and based on the unique identifier of the file to be created, the corresponding reference entry is searched in the index table. In the above embodiment, since the reference relationship between the file and the data fingerprint is written to the index table, table accesses are reduced when creating the file, effectively alleviating performance issues caused by hot file sharing. In conjunction with the above embodiment, after writing the new reference relationship to the index table, the index table can also be read to check whether there is a concurrent issue with creation and other operations such as deletion, thereby ensuring data accuracy. Specifically, after writing the reference relationship between the file to be created and the data fingerprint into the index table, the method further includes: determining whether an index entry corresponding to the data fingerprint referenced by the file to be created exists in the index table; if so, determining whether the data fingerprint referenced by the file to be created in the index entry has changed; and if so, deleting the reference entry corresponding to the reference relationship between the file to be created and the data fingerprint. Accordingly, if no change has occurred, the creation is successful. For example, the above determination and deletion steps can be performed using a conditional update mechanism. The conditional update mechanism executes the update action when the determination condition is met, and rejects it otherwise. The determination and update are atomic operations. After writing the new reference relationship into the index table, the above embodiment checks whether the newly written reference relationship has changed to determine whether there is a concurrency issue between the creation and deletion operations. If so, a concurrency issue exists. By deleting the incorrect reference entry, the existence of the incorrect reference entry in the index table is avoided, thereby resolving the data inaccuracy issue caused by concurrent creation and deletion operations and ensuring data accuracy if the creation is successful. In addition, in order to avoid the data fingerprint referenced by the newly created file being deleted due to the garbage collection mechanism, the garbage collection status can also be processed to ensure the availability of the created data fingerprint.Specifically, after writing the reference relationship between the file to be created and the data fingerprint into the reference entry in the index table, the process further includes: determining whether an index entry corresponding to the data fingerprint referenced by the file to be created exists in the index table; if so, determining whether the data fingerprint referenced by the file to be created is in the garbage collection state; and if so, canceling the garbage collection state of the data fingerprint referenced by the file to be created. It will be appreciated that during the garbage collection process, data to be garbage collected will be marked as garbage collected before actual deletion, awaiting deletion. Therefore, the garbage collection state of a data fingerprint means that the data fingerprint is marked as garbage collected and awaiting deletion. In the above embodiment, since the garbage collection state of the newly created file is canceled, the newly created file can normally reference the data fingerprint, preventing the record corresponding to the data fingerprint from being deleted by the background garbage collection mechanism, thereby ensuring the availability of the data fingerprint. It should be noted that the above embodiments of concurrently processing creation and other operations, as well as the embodiment of garbage collection state processing, can be implemented separately or in combination, and the present invention is not limited thereto. The following is an exemplary description of the combined implementation process. FIG4 shows a flowchart of a file creation process of a data processing method provided by an embodiment of the present invention, which specifically includes the following steps. Step 402: Receive a file creation request and obtain an index table fp. Step 404: Determine whether the data fingerprint referenced by the file to be created corresponding to the file creation request exists in the index table fp. Step 406: If so, write the reference relationship between the file to be created and the data fingerprint into the reference entry of the index table. Step 408: Re-acquire the index table fp. Step 410: Determine whether the data fingerprint referenced by the file to be created exists in the index table fp. Step 412: If so, determine whether the data fingerprint referenced by the file to be created in the index entry has changed. If step 410 determines that the data fingerprint referenced by the file to be created has not changed, proceed to step 418 to delete the reference entry corresponding to the reference relationship between the file to be created and the data fingerprint. Step 414: If step 412 determines that the data fingerprint referenced by the file to be created has not changed, determine whether the data fingerprint referenced by the file to be created is in a garbage collection state. If step 414 determines that the data fingerprint referenced by the file to be created is not in a garbage collection state, then determine that the file creation is successful. Step 416: If step 414 determines that the file is in the garbage collection state, cancel the garbage collection state of the data fingerprint referenced by the file to be created. If the cancellation fails, it is determined that the file creation has failed. If the cancellation succeeds, it is determined that the file creation has succeeded.If step 412 determines that there is a change, then proceed to step 418 to delete the reference entry corresponding to the reference relationship between the file to be created and the data fingerprint. Step 418: If step 410 determines that the reference entry does not exist, or if step 412 determines that the reference entry does not exist, then delete the reference entry corresponding to the reference relationship between the file to be created and the data fingerprint. Step 420: If step 404 determines that the reference entry does not exist, then create an index entry corresponding to the data fingerprint referenced by the file to be created in the index table, and write the reference relationship between the file to be created and the data fingerprint into the reference entry in the index table. Step 422: Determine whether step 420 was successful. If successful, the creation is successful. If unsuccessful, the creation is unsuccessful. In the above embodiment, concurrency issues with file creation and deletion operations are identified and resolved, as well as concurrency issues with file creation and data fingerprint garbage collection. This resolves the issues of incorrect data and accidental deletion by garbage collection, effectively ensuring data accuracy. Next, we will exemplify a specific implementation of deleting a file by writing the reference relationship between a file and a data fingerprint into an index table so that the reference relationship and the data fingerprint are stored in the same table. In one or more embodiments of the present invention, the method further includes: in response to a file deletion request, marking the metadata to be deleted corresponding to the file deletion request in a file metadata table for deletion; the file metadata table is used to store file metadata; deleting the reference entry corresponding to the metadata to be deleted from the index table; and, upon determining that the reference relationship has been successfully deleted, deleting the record corresponding to the metadata to be deleted from the file metadata table based on the deletion mark. Through the above embodiments, the index table can be effectively updated in the event of file deletion, ensuring data accuracy. Next, we will exemplify a specific process for deleting a file. Figure 5 shows a flowchart of a file deletion process in a data processing method provided by one embodiment of the present invention, which specifically includes the following steps: Step 502: In response to the file deletion request, marking the metadata to be deleted corresponding to the file deletion request in the file metadata table for deletion. Step 504: Determining whether the marking was successful. Step 506: If the marking is successful, delete the reference item corresponding to the metadata to be deleted from the index table. If the marking fails, determine that the deletion process has failed. Step 508: Determine whether the reference item has been successfully deleted. Step 510: If the reference item has been successfully deleted, delete the record corresponding to the metadata to be deleted from the file metadata table based on the deletion mark.It is understandable that if a remote procedure call times out or a storage update fails, the reference item deletion may fail. If the reference item deletion fails, the deletion process is determined to have failed. Step 512: Determine whether the record corresponding to the metadata to be deleted has been successfully deleted from the file metadata table. If successful, the deletion process is completed; otherwise, the deletion fails. Below, an exemplary embodiment of maintaining a reference count for a data fingerprint by writing the reference relationship between a file and a data fingerprint into an index table so that the reference relationship and the data fingerprint are stored in the same table will be described. Specifically, in one or more embodiments of the present invention, the method further includes: in response to a change in a record in the index table, determining whether the change is a change in the reference relationship between the file and the data fingerprint; if so, incrementing the reference count corresponding to the data fingerprint if the change is a change in the reference relationship between the file and the data fingerprint; and decrementing the reference count corresponding to the data fingerprint if the change is a change in the reference relationship between the file and the data fingerprint. In the above embodiment, since the index table is used to store reference relationships between files and data fingerprints, the index table can be promptly updated when a new reference relationship is added, allowing the reference count of the data fingerprint to be updated promptly based on changes in the index table records. This simplifies the reference count maintenance process and improves performance. Furthermore, to ensure timely updates to the index table during garbage collection of the data fingerprint and ensure data accuracy, the method further includes: in response to a change in a record in the index table, determining whether the change indicates that the data fingerprint has completed garbage collection; if so, marking the record corresponding to the data fingerprint in the index table as deleted. The record corresponding to the data fingerprint includes the index item and reference item corresponding to the data fingerprint. In the above embodiment, when a data fingerprint changes and garbage collection is completed, the corresponding index item and reference item in the index table can be promptly deleted, ensuring the accuracy of the record in the index table. Furthermore, in conjunction with the above-mentioned embodiment of maintaining reference counts for data fingerprints, in order to promptly delete data no longer referenced by files through garbage collection and release storage resources, one or more embodiments of the present invention further include: determining, based on the reference count of the data fingerprint in the index table, whether the data fingerprint needs to be set to a garbage collection state; and if so, setting the data fingerprint to a garbage collection state. Setting the data fingerprint to a garbage collection state may include setting the index item and reference item corresponding to the data fingerprint to a garbage collection state. For example, in the above-mentioned embodiment, garbage collection of a data fingerprint with a reference count of zero may be performed in both foreground and background stages.The foreground process may refer to the GC service querying the reference count of a data fingerprint and marking a data fingerprint with a reference count of zero as being in the garbage collection state. The backend process may refer to the GC service monitoring the table content change notification channel and, upon receiving a notification indicating that a record corresponding to a data fingerprint has been marked as being in the garbage collection state, performing the actual garbage collection action. The following is an exemplary description of the specific process for setting the garbage collection state. Figure 6 shows a flowchart of the garbage collection process of a data processing method provided by an embodiment of the present invention, which specifically includes the following steps: Step 602: Traverse the index table to retrieve the index entry with a reference count of zero for the currently traversed data fingerprint. Step 604: Determine whether the traversal is complete. Step 606: If the traversal is not complete, determine whether the currently traversed data fingerprint with a reference count of zero is in a normal reference state, that is, not in the garbage collection state or has completed garbage collection. Step 608: If so, set the currently traversed data fingerprint with a reference count of zero to the garbage collection state. If the traversal is not complete, the next record in the index table is traversed. If the traversal is complete, the process exits. In the above embodiment, by saving the reference relationship between the file and the data fingerprint in the data fingerprint index table and maintaining the reference count based on index table updates, records corresponding to data fingerprints with a reference count of zero can be scanned as candidate records for garbage collection. Therefore, data fingerprints with a reference count of zero are set to the garbage collection state. In this way, the garbage collection mechanism will delete the records corresponding to data fingerprints in the garbage collection state, thereby effectively improving garbage collection efficiency. In the above embodiment, the determination of whether a record in the index table has changed can be based on any possible implementation method, and the present invention is not limited to this. For example, in one or more embodiments of the present invention, the determination is based on the mechanism of the table content change notification channel. Specifically, the method further includes: obtaining a notification from a table content change notification channel of the index table; determining whether a record in the index table has changed based on the notification; if the processing step of marking for deletion is successfully executed, returning a consumption completion message to the table content change notification channel; if the processing step of marking for deletion fails, returning a consumption failure message to the table content change notification channel, and re-entering the step of obtaining a notification from the table content change notification channel of the index table.In the above embodiment, the determination of whether a record in the index table has changed can be performed based on the consumption logic of the table content change notification channel. Due to the sequential nature of the consumption logic of the table content change notification channel, subsequent processing when a record has changed can be performed more accurately, promptly, and efficiently, thereby improving processing efficiency. The following exemplary embodiment illustrates the subsequent processing when an index table record has changed based on the mechanism of the table content change notification channel. Figure 7 shows a flowchart of a data processing method based on the table content change notification channel, according to one embodiment of the present invention. The method specifically includes the following steps: Step 702: Obtain notifications from the table content change notification channel by monitoring the table content change notification channel. Step 704: Determine, based on the notifications, whether a reference item in the index table has changed. Step 706: If the reference item has changed, determine whether it is a reference relationship between a newly added file and a data fingerprint. Step 708: If it is a reference relationship between a newly added file and a data fingerprint, increment the reference count corresponding to the data fingerprint of the reference item. Step 710: If the reference relationship between the file and the data fingerprint is reduced, decrement the reference count corresponding to the data fingerprint of the referenced item. Step 712: If the referenced item is not changed, determine whether the change is a data fingerprint that has completed garbage collection. Step 714: If garbage collection has completed, mark the record corresponding to the data fingerprint as deleted. Step 716: Determine whether marking is successful. Step 718: If marking is successful, return a consumption completion message to the table content change notification channel. Step 720: If marking fails, return a consumption failure message to the table content change notification channel and wait for a retry to obtain notifications from the table content change notification channel of the index table. In the above embodiment, based on the table content change notification channel and through notifications in the consumption channel, consistent reference count updates can be achieved, improving garbage collection efficiency. Corresponding to the above method embodiment, the present invention also provides an embodiment of a data processing device. Figure 8 shows a schematic structural diagram of a data processing device provided by one embodiment of the present invention.As shown in Figure 8 , the apparatus includes: a verification and determination module 802 configured to determine a data fingerprint to be verified, where the data fingerprint is an identifier used to distinguish different data contents; an index data acquisition module 804 configured to obtain metadata corresponding to the data fingerprint from an index table of the data fingerprint, where the index table is used to store index items of the data fingerprint and metadata corresponding to the data fingerprint, where the metadata is metadata of a file that references the data fingerprint; and a data verification module 806 configured to locate an actual file using the obtained metadata and obtain a consistency verification result between the data fingerprint and the actual file. In one or more embodiments of the present invention, the data verification module is configured to obtain metadata to be verified for the file to be verified from a file metadata table, where the file to be verified is a file that references the data fingerprint, and the file metadata table is used to store metadata of an actual file. The obtained metadata is compared with the metadata to be verified to determine whether they are identical. If they are not identical, it is determined that there is an anomaly in the consistency between the data fingerprint and the actual file. In one or more embodiments of the present invention, the verification and determination module includes: a content acquisition submodule configured to acquire data content from the file to be verified for use in constructing a data fingerprint; a primary key construction submodule configured to construct a primary key for the data fingerprint to be queried based on the data content; and a data matching submodule configured to use the primary key of the data fingerprint to be queried to retrieve, from the data fingerprint index table, a data fingerprint corresponding to an index item that matches the primary key of the data fingerprint to be queried, as the data fingerprint to be verified. The index item is used to store the primary key of the corresponding data fingerprint. In one or more embodiments of the present invention, the data verification module is configured to use the acquired metadata to retrieve a record matching the metadata from a file metadata table. The file metadata table is used to store metadata for an actual file. The module then determines whether the acquired metadata is identical to the metadata stored in the record. If not, it determines that there is an inconsistency between the data fingerprint and the actual file. In one or more embodiments of the present invention, the index table also includes a reference item, which stores a reference relationship between the file and the data fingerprint.The apparatus further includes: a file creation module configured to, in response to a file creation request, determine whether a data fingerprint referenced by a file to be created corresponding to the file creation request exists in the index table; a reference writing module configured to, if the file creation module determines that the data fingerprint exists, write a reference relationship between the file to be created and the data fingerprint into a reference entry in the index table; and, if the file creation module determines that the data fingerprint does not exist, create an index entry corresponding to the data fingerprint referenced by the file to be created in the index table and write the reference relationship between the file to be created and the data fingerprint into the reference entry in the index table. In one or more embodiments of the present invention, the apparatus further includes: a fingerprint existence determination module configured to determine whether an index entry corresponding to the data fingerprint referenced by the file to be created exists in the index table; a change determination module configured to, if the fingerprint existence determination module determines that the data fingerprint referenced by the file to be created exists, determine whether a change has occurred in the index entry; and a reference deletion module configured to, if the change determination module determines that a change has occurred, delete the reference entry corresponding to the reference relationship between the file to be created and the data fingerprint. In one or more embodiments of the present invention, the apparatus further includes: a fingerprint existence determination module configured to determine whether an index entry corresponding to the data fingerprint referenced by the to-be-created file exists in the index table; a garbage collection determination module configured to determine whether the data fingerprint referenced by the to-be-created file is in a garbage collection state if the fingerprint existence determination module determines that the data fingerprint referenced by the to-be-created file is in a garbage collection state; and a collection cancellation module configured to cancel the garbage collection state of the data fingerprint referenced by the to-be-created file if the garbage collection determination module determines that the data fingerprint referenced by the to-be-created file is in a garbage collection state. In one or more embodiments of the present invention, the apparatus further includes: a deletion marking module configured to, in response to a file deletion request, mark the to-be-deleted metadata corresponding to the file deletion request in a file metadata table as deleted, the file metadata table being used to store file metadata; a reference deletion module configured to delete the reference entry corresponding to the to-be-deleted metadata in the index table; and a marked data deletion module configured to, upon determining that the reference entry has been successfully deleted, delete the record corresponding to the to-be-deleted metadata from the file metadata table based on the deletion mark.In one or more embodiments of the present invention, the apparatus further comprises: a reference change determination module configured to, in response to a change in a record in the index table, determine whether the change is a change in the reference relationship between a file and a data fingerprint; a count update module configured to, if the reference change determination module determines that the change is a change in the reference relationship between the file and the data fingerprint, increment the reference count corresponding to the data fingerprint; and, if the change is a change in the reference relationship between the file and the data fingerprint, decrement the reference count corresponding to the data fingerprint. In one or more embodiments of the present invention, the apparatus further comprises: a recycling change determination module configured to, in response to a change in a record in the index table, determine whether the change is a change in the data fingerprint causing garbage collection; a deletion marking module configured to, if the recycling change determination module determines that the change is a change in the data fingerprint causing garbage collection; and, if the recycling change determination module determines that the change is a change in the data fingerprint causing garbage collection. In one or more embodiments of the present invention, the apparatus further comprises: a count determination module configured to determine whether the data fingerprint needs to be set to garbage collection status based on the reference count of the data fingerprint in the index table; and a recycling setting module configured to set the data fingerprint to garbage collection status if the count determination module determines that the change is a change in the data fingerprint causing garbage collection. In one or more embodiments of the present invention, the apparatus further includes: a notification acquisition module configured to acquire notifications from a table content change notification channel of the index table; a record change determination module configured to determine whether a record in the index table has changed based on the notifications; a consumption success notification module configured to return a consumption completion message to the table content change notification channel if the processing step marked for deletion is successfully executed; and a consumption failure notification module configured to return a consumption failure message to the table content change notification channel and retry the step of acquiring notifications from the table content change notification channel of the index table if the processing step marked for deletion fails. The above is a schematic diagram of a data processing apparatus according to this embodiment. It should be noted that the technical solution of the data processing apparatus and the technical solution of the aforementioned data processing method share the same concept. Details not described in detail in the technical solution of the data processing apparatus can be found in the description of the technical solution of the aforementioned data processing method. Figure 9 shows a block diagram of a computing device 900 according to one embodiment of the present invention. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 and the memory 910 are connected via a bus 930. The database 950 is used to store data.The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, or a near field communication (NFC) interface. In one embodiment of the present invention, the aforementioned components of computing device 900, as well as other components not shown in FIG. 9 , may also be connected to one another, for example, via a bus. It should be understood that the computing device block diagram shown in FIG. 9 is for illustrative purposes only and does not limit the scope of the present invention. Those skilled in the art may add or replace other components as needed. Computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 900 may also be a mobile or stationary server. Processor 920 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the aforementioned data processing method. The above is a schematic diagram of a computing device according to this embodiment.It should be noted that the technical solution of this computing device and the technical solution of the aforementioned data processing method are based on the same concept. Any details not described in detail in the technical solution of the computing device can be found in the description of the technical solution of the aforementioned data processing method. One embodiment of the present invention also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, these computer-executable instructions implement the steps of the aforementioned data processing method. The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the aforementioned data processing method are based on the same concept. Any details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the aforementioned data processing method. One embodiment of the present invention also provides a computer program. When executed on a computer, this computer program causes the computer to execute the steps of the aforementioned data processing method. One embodiment of the present invention also provides a computer program product comprising a computer program / instructions. When executed by a processor, these computer program / instructions implement the steps of the aforementioned data processing method. The above is a schematic diagram of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the aforementioned data processing method share the same concept. For details not described in detail in the technical solution of the computer program, reference should be made to the description of the technical solution of the aforementioned data processing method. The above describes specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous. The computer instructions include computer program code, which may be in source code form, object code form, executable files, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a removable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.It should be noted that, for ease of description, the aforementioned method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, as certain steps may be performed in a different order or simultaneously, depending on the embodiments of the present invention. Furthermore, those skilled in the art should also be aware that the embodiments described in this specification are preferred embodiments, and the actions and modules described are not necessarily required for the embodiments of the present invention. In the aforementioned embodiments, the description of each embodiment has its own emphasis. For portions not described in detail in a particular embodiment, reference should be made to the relevant descriptions of other embodiments. The preferred embodiments disclosed above are merely intended to help illustrate the present invention. The alternative embodiments do not describe all details in detail, nor do they limit the present invention to the specific implementations described. Obviously, many modifications and variations are possible based on the content of the embodiments of the present invention. These embodiments are selected and described in detail to better explain the principles and practical applications of the embodiments of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

Claims 1. A data processing method, wherein: include: Determining a data fingerprint to be verified, where the data fingerprint is an identifier used to distinguish different data contents; Obtain metadata corresponding to the data fingerprint from an index table of the data fingerprint, wherein the index table is used to store index items of the data fingerprint and metadata corresponding to the data fingerprint, and the metadata is metadata of a file that references the data fingerprint; The obtained metadata is used to locate the actual existing file, and a verification result of the consistency between the data fingerprint and the actual existing file is obtained.

2. The method according to claim 1, wherein the using the obtained metadata to locate an actual file and obtaining a consistency verification result between the data fingerprint and the actual file comprises: Obtaining metadata of the file to be verified from a file metadata table, where the file to be verified is a file that references the data fingerprint, and the file metadata table is used to store metadata of an actual file; Comparing the obtained metadata with the metadata to be verified to determine whether they are identical metadata; If they are not the same, it is determined that there is an anomaly in the consistency between the data fingerprint and the actual existing file.

3. The method according to claim 2, wherein determining the data fingerprint to be verified comprises: Obtaining data content of the file to be verified for constructing a data fingerprint; Based on the data content, construct a primary key of the data fingerprint to be queried; Using the primary key of the data fingerprint to be queried, the data fingerprint corresponding to the index item matching the primary key of the data fingerprint to be queried is queried in the index table of the data fingerprint as the data fingerprint to be verified, and the index item is used to store the primary key of the corresponding data fingerprint.

4. The method according to claim 1, wherein the data fingerprint corresponding to any index item in the index table is the data fingerprint to be verified; The step of locating an actually existing file by using the obtained metadata and obtaining a consistency verification result between the data fingerprint and the actually existing file includes: Using the obtained metadata, searching for a record matching the metadata in a file metadata table, where the file metadata table is used to store metadata of actually existing files, to determine whether the obtained metadata is the same as the metadata stored in the record; If they are not the same, it is determined that there is an anomaly in the consistency between the data fingerprint and the actual existing file.

5. The method according to claim 1, wherein the index table further comprises a reference item, wherein the reference item is used to store a reference relationship between the file and the data fingerprint; the method further comprises: In response to a file creation request, determining whether a data fingerprint referenced by a file to be created corresponding to the file creation request exists in the index table; If it exists, write the reference relationship between the file to be created and the data fingerprint into the reference item of the index table; if it does not exist, create an index item corresponding to the data fingerprint referenced by the file to be created in the index table, and write the reference relationship between the file to be created and the data fingerprint into the reference item of the index table.

6. The method according to claim 5, wherein the reference relationship between the file to be created and the data fingerprint is written into After the reference item of the index table, it also includes: Determine whether there is an index entry corresponding to the data fingerprint referenced by the file to be created in the index table; If so, determining whether the data fingerprint referenced by the file to be created in the index item has changed; If a change occurs, the reference item corresponding to the reference relationship between the file to be created and the data fingerprint is deleted.

7. The method according to claim 5 or 6, further comprising: after writing the reference relationship between the file to be created and the data fingerprint into the reference item of the index table; Determine whether there is an index entry corresponding to the data fingerprint referenced by the file to be created in the index table; If so, determining whether the data fingerprint referenced by the to-be-created file is in a garbage collection state; if so, canceling the garbage collection state of the data fingerprint referenced by the to-be-created file.

8. The method according to claim 7, further comprising: In response to a file deletion request, marking the to-be-deleted metadata corresponding to the file deletion request in a file metadata table as deleted, where the file metadata table is used to store metadata of the file; Deleting the reference item corresponding to the metadata to be deleted in the index table; When it is determined that the reference item is deleted successfully, the record corresponding to the metadata to be deleted is deleted from the file metadata table according to the deletion mark.

9. The method according to claim 5, further comprising: In response to a change in a record in the index table, determining whether the change is a change in a reference relationship between a file and a data fingerprint; If so, if the change is to add a reference relationship between the file and the data fingerprint, the reference count corresponding to the data fingerprint is incremented; if the change is to reduce the reference relationship between the file and the data fingerprint, the reference count corresponding to the data fingerprint is decremented.

10. The method according to claim 5 or 9, further comprising: In response to a change in a record in the index table, determining whether the change is a data fingerprint that completes garbage collection; If yes, the record corresponding to the data fingerprint is marked as deleted in the index table.

11. The method according to claim 9, further comprising: Determining whether to set the data fingerprint to a garbage collection state according to the reference count of the data fingerprint in the index table; If yes, the data fingerprint is set to a garbage collection state.

12. The method according to claim 9, further comprising: Obtain notifications from a table content change notification channel of the index table; Determining whether a record in the index table has changed according to the notification; If the processing step marked as deleted is executed successfully, a consumption completion message is returned to the table content change notification channel. If the processing step marked as deleted fails, a consumption failure message is returned to the table content change notification channel, and the step of obtaining the notification from the table content change notification channel of the index table is retried.

13. A computing device, comprising: 18 Memory and Processor; The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the data processing method according to any one of claims 1 to 12 are implemented.

14. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 12.

15. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 12.

Images