Data processing method and system

Abstract

The embodiment of the present specification provides a data processing method and system, wherein the data processing method comprises: the data access node, in response to a data access request, determining corresponding to-be-accessed data of a to-be-accessed data identifier in a data storage node according to the to-be-accessed data identifier carried in the data access request, and accessing the to-be-accessed data in the data storage node; the data scanning node, in response to a data scanning request, determining corresponding to-be-scanned data of a to-be-scanned data identifier in the data storage node according to the to-be-scanned data identifier carried in the data scanning request, loading the to-be-scanned data to a memory, and scanning the to-be-scanned data in the memory, wherein the to-be-accessed data comprises the to-be-scanned data. The user access and data scanning are guaranteed in parallel, and the data loaded to the memory through scanning does not affect the user's access, thereby guaranteeing the user experience.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to data processing methods. Background Technology

[0002] With the development of computer technology, data storage systems can provide users with a wide variety of services. For example, users can store data in a data storage system and perform operations such as reading, writing, and modifying the data stored in the system. Correspondingly, the data storage system also stores metadata, which can be used to describe key information such as the directory structure, file attributes, and data distribution of the data storage system.

[0003] To ensure consistency between the metadata stored in the data storage system and the actual stored data, it is necessary to scan the data in the system to detect anomalies. However, if the data being scanned is being accessed by a user, it can affect the user's experience, leading to incorrect results and negatively impacting the user experience. Therefore, an effective technical solution is urgently needed to address this issue. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing system, another data processing method, another data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a data processing method is provided, applied to a data processing system, the data processing system including a data scanning node, a data storage node, and a data access node, the method comprising:

[0006] In response to a data access request, the data access node determines the data to be accessed corresponding to the data to be accessed identifier carried in the data access request in the data storage node, and accesses the data to be accessed in the data storage node.

[0007] In response to a data scanning request, the data scanning node determines the data to be scanned corresponding to the data to be scanned identifier in the data storage node according to the data to be scanned identifier carried in the data scanning request, loads the data to be scanned into memory, and scans the data to be scanned in the memory, wherein the data to be accessed includes the data to be scanned.

[0008] According to a second aspect of the embodiments of this specification, a data processing system is provided, including a data processing node and a data storage node, wherein the data processing node is used to execute the method described in the first aspect of the embodiments of this specification, and the data storage node is used to store data.

[0009] According to a third aspect of the embodiments of this specification, a data processing method is provided, applied to a data scanning node, comprising:

[0010] In response to a data scanning request, the data to be scanned corresponding to the data to be scanned identifier carried in the data scanning request is determined in the data storage node.

[0011] The data to be scanned is loaded into memory, and the data to be scanned is scanned in the memory.

[0012] According to a fourth aspect of the embodiments of this specification, a data processing apparatus is provided, applied to a data scanning node, comprising:

[0013] The determination module is configured to, in response to a data scan request, determine the data to be scanned corresponding to the data to be scanned identifier carried in the data scan request in the data storage node;

[0014] The scanning module is configured to load the data to be scanned into memory and scan the data to be scanned in the memory.

[0015] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0016] Memory and processor;

[0017] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above method.

[0018] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the above-described method.

[0019] According to a seventh aspect of an embodiment of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described method.

[0020] This specification provides a data processing method according to one embodiment, applied to a data processing system. The data processing system includes a data scanning node, a data storage node, and a data access node. The data access node, in response to a data access request, determines the data to be accessed corresponding to the data to be accessed identifier carried in the data access request in the data storage node, and accesses the data to be accessed in the data storage node. The data scanning node, in response to a data scanning request, determines the data to be scanned corresponding to the data to be scanned identifier carried in the data scanning request in the data storage node, loads the data to be scanned into memory, and scans the data to be scanned in the memory. The data to be accessed includes the data to be scanned.

[0021] The above method sets up a data scanning node, which can load all or part of the data accessed by the user into the memory of the data scanning node when the user accesses data, and scan the data in memory to ensure that user access and data scanning are carried out in parallel. Furthermore, by scanning the data loaded into memory, the user access is not affected, thus ensuring a good user experience. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the structure of a data processing system provided in one embodiment of this specification;

[0023] Figure 2 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification.

[0024] Figure 3 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;

[0025] Figure 4 This is a flowchart of another data processing method provided in one embodiment of this specification;

[0026] Figure 5 This is a flowchart illustrating the processing procedure of another data processing method provided in one embodiment of this specification.

[0027] Figure 6 This is a schematic diagram of the structure of another data processing device provided in one embodiment of this specification;

[0028] Figure 7 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0029] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0030] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0031] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0032] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0033] meta file: Used to index a set of SSTable files.

[0034] SSTable file: A data structure used to store a set of key-value pairs that cannot be changed once generated.

[0035] memtable file: User operations on data are written to this memtable file. This memtable file is mutable and can be converted into an sstable file.

[0036] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0037] This specification provides a data processing system, and also relates to a data processing method, another data processing method, another data processing apparatus, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0038] See Figure 1 , Figure 1 This diagram illustrates a data processing system 100 according to an embodiment of this specification. The data processing system 100 includes a data scanning node 102, a data storage node 104, and a data access node 106.

[0039] The data access node 106 is configured to respond to a data access request, determine the data to be accessed corresponding to the data to be accessed in the data storage node 104 according to the data to be accessed identifier carried in the data access request, and access the data to be accessed in the data storage node.

[0040] The data scanning node 102 is configured to respond to a data scanning request, determine the data to be scanned corresponding to the data to be scanned identifier in the data storage node 104 according to the data to be scanned identifier carried in the data scanning request, load the data to be scanned into memory, and scan the data to be scanned in the memory, wherein the data to be accessed includes the data to be scanned.

[0041] In this context, a data storage node can be understood as a server or server cluster used to store data; for example, a data storage node can be a distributed data storage system. A data scanning node can be understood as a server used to scan data. A data access node can be understood as a server used to receive user access requests for data. Data access requests and data scanning requests can be understood as requests for the same data. For example, for metadata file A stored on a data storage node, a data access request can be an access request for metadata file A, and a data scanning request can be a scan request for metadata file A. A data access request can also be understood as a request to perform operations such as reading, writing, or modifying the data to be accessed; for example, it could be a deletion request for metadata file A. Accordingly, the data to be accessed and the data to be scanned can be the same data, or they can be different data, or the data to be scanned can be a portion of the data to be accessed.

[0042] Specifically, in a data processing system, a data access node responds to a user's data access request and, based on the identifier of the data to be accessed carried in the request, identifies the data to be accessed in the data storage node and accesses it, performing operations such as reading, writing, modifying, and deleting. When a data scanning node receives a data scanning request for the same data to be accessed, it identifies the data to be scanned in the data storage node based on the identifier of the data to be scanned carried in the request. This data to be scanned is the same as the data to be accessed. At this point, the data to be scanned can be loaded from the data storage node into the data scanning node's own memory and scanned in memory.

[0043] Furthermore, it is understandable that since users will perform operations such as reading, writing, deleting, adding, and modifying the data stored in the data storage nodes, that is, the data stored in the data storage nodes will increase or change over time. The data stored in the data storage nodes can be scanned and verified according to a preset time to ensure that the consistency of incremental data can also be detected, thereby achieving comprehensive data detection.

[0044] Specifically, when loading the data to be scanned from the data storage node into memory, a data index can be used. Accordingly, the data scanning node is also configured to create a data index in the memory associated with the data to be scanned based on the identifier of the data to be scanned, and to load the data to be scanned stored in the data storage node into the memory using the data index.

[0045] The data index can be understood as a separate, physical storage structure that sorts the values ​​of one or more columns in a database table. It can be understood as a directory of the data to be scanned. Based on the data index, the data to be scanned can be retrieved from the data storage node in the memory of the data scanning node.

[0046] Based on this, a data index associated with the data to be scanned can be created in the memory of the data scanning node according to the identifier of the data to be scanned. Using this data index, the data to be scanned can be determined in the data storage node, and the data to be scanned in the data storage node can be loaded into the memory of the data scanning node.

[0047] In practical applications, the data to be scanned can also be loaded by creating hard links in the memory of the data scanning node. Specifically, a corresponding hard link can be created in memory based on the identifier of the data to be scanned, and the data to be scanned can be retrieved from the data storage node based on the hard link and loaded into memory.

[0048] In summary, by creating a data index, it is possible to determine and load the data to be scanned, providing a foundation for subsequent scanning of the data to be scanned in the memory of the data scanning node.

[0049] In practical applications, the data access node is also configured to perform an update operation on the data to be accessed in the data access node, obtain the updated data to be accessed, convert the updated data to be accessed into target data according to a preset rule, and store the target data in the data storage node.

[0050] The data scanning node is also configured to load the target data into the memory and scan the target data in the memory.

[0051] Specifically, after the data access node determines the data to be accessed in the data storage node, it can perform corresponding access operations on the data in the memory of the data access node. For example, it can update the data to obtain the updated data to be accessed. Furthermore, according to preset rules, the updated data to be accessed is converted into target data, and the target data is stored in the data storage node. Correspondingly, since the data scanning request and the data access request target the same data, the data scanning node can load the converted target data into memory and perform scanning in memory when scanning the data.

[0052] The preset rules can be understood as the rules that transform the updated data to be accessed into target data. Preset rules can be determined based on the actual needs of the data. Target data can be understood as data that, once generated, will not change and cannot be altered; target data is the data to be scanned.

[0053] Furthermore, multiple data access requests can be included for the same data. For example, for data a1 to be accessed, the first data access request updates the data a1 to be accessed to a2, and the second data access request updates a2 to a3. At this time, a2 and a3 are both updated data to be accessed (i.e., mutable data). After converting data a2 and a3 into target data a, the target data a becomes immutable data.

[0054] For example, given data a1 to be accessed, the data access node can modify the data a1 according to the data access request to obtain the updated data a2, and then convert the updated data a2 into target data a according to preset rules. The data scanning node then loads the target data a into memory for scanning.

[0055] Optionally, in response to a data access request, the data access node determines the data to be accessed corresponding to the data to be accessed identifier in the data storage node according to the data to be accessed identifier carried in the data access request, accesses the data to be accessed in the data access node, updates the data to be accessed in the data access node to obtain the updated data to be accessed, and converts the updated data to be accessed into target data according to a preset rule, and stores the target data in the data storage node.

[0056] Accordingly, the data scanning node receives a data scanning request for the target data, loads the target data from the data storage node into memory, and performs a scan in the memory.

[0057] In summary, by converting the updated data to be accessed into target data and using data scanning nodes to scan the target data, the parallel operation of data scanning and data access is ensured, so that data scanning and data access do not interfere with each other and avoid mutual influence.

[0058] In practice, the preset rule can be to convert data according to a preset time interval. Specifically, the data access node is also configured to convert the updated data to be accessed into target data according to the preset time interval.

[0059] The preset time interval can be, for example, 5 seconds or 10 days. The preset time interval can be determined based on data requirements. For example, for frequently accessed data, which needs to be updated immediately to ensure stability and accuracy, the preset time interval can be set shorter, such as 1 second or 2 seconds. Conversely, for infrequently accessed data that may not be accessed for extended periods, the preset time interval can be set longer, such as 10 days or 50 days, reducing resource consumption on data storage nodes and thus improving performance.

[0060] In addition, data can be converted according to the amount of updated data to be accessed. Specifically, the data access node is also configured to convert the updated data to be accessed into target data when the amount of updated data to be accessed reaches a preset data amount threshold.

[0061] The amount of data to be accessed can be understood as the size of the data. The preset data size threshold can be understood as a pre-set threshold for the size of the data that can be accessed, such as 10G. Once the data size reaches 10G, the data can be converted into target data to ensure the stability of data storage.

[0062] Based on this, if the amount of updated data to be accessed reaches a preset data volume threshold, the updated data to be accessed will be converted into target data.

[0063] Understandably, the updated data to be accessed is accessible, readable, writable, and modifiable. Subsequent data access requests can still access the updated data and perform read, write, and modify operations. The target data, on the other hand, is immutable once generated, but it can be read and written. In practical applications, the updated data to be accessed can be understood as a memtable file stored in the memory of the data access node, while the target data can be understood as an SStable file stored in the data storage node. Furthermore, in practical applications, the data to be scanned can be a group of SStable files stored in the data storage node. Each group of SStable files can include a single SStable file or multiple SStable files. The metadata information in the SStable and memtable files can be stored in the form of key-value pairs. When a user modifies the files corresponding to this metadata information, the metadata information will also change accordingly.

[0064] In summary, by transforming the updated data to be accessed using preset rules to obtain the target data, a delay is created between the user's real-time data access and the data storage operation, ensuring the stability of data storage and facilitating subsequent scanning of the target data. Furthermore, since the target data is immutable once generated, the scanning process is guaranteed to be accurate. Figure 1 Consistency is ensured to prevent errors in scan results caused by concurrent user access and operations.

[0065] In addition, scanning the data to be scanned can be used to scan the status of the data in order to verify the consistency of the data. Specifically, the data scanning node is also configured to obtain the scan results for the data to be scanned and update the data to be scanned based on the scan results.

[0066] The scan result can be understood as the consistency scan result of the state of the data to be scanned. The data to be scanned may include a set of metadata files stored in the data storage node. The scan result can be used to determine whether the metadata files stored in the data storage node are consistent. For example, if the metadata is a directory and the state of the directory is that it contains 100 files, this scan will determine whether the state of the directory is true and accurate, that is, it will scan whether there are 100 files stored in the directory. The scan result may also be a snapshot of the data to be scanned at a certain moment.

[0067] Based on this, the scanned data can be updated according to the scan results obtained.

[0068] Using the previous example, if the scan results show that there are only 90 files stored in the directory, the directory status will be updated, or the files stored in the directory will be updated.

[0069] In summary, by updating the data to be scanned based on the scan results, the accuracy of data storage in the data storage nodes can be guaranteed, thereby ensuring a better user experience.

[0070] In specific implementation, the data scanning node is further configured to determine the status information of the data to be scanned based on the scanning results, determine the access information of the data to be scanned based on the access log of the data storage node, and determine whether the status information and the access information match. If they match, the scanning ends; otherwise, the data to be scanned is updated based on the access information.

[0071] The access log of a data storage node can be understood as access records of metadata files stored within the data storage node. These metadata files can be directories, for example. For instance, a directory within a data storage node might record the attribute information and storage address of file A. If file A is stored in directory 1, and a data access request moves file A to directory 2, thus modifying the metadata about file A recorded in the directory, then the access log of the data storage node should store this move operation and update the directory and file states to ensure a correspondence between the access behavior recorded in the log and the actual storage of the data in the data storage node. The status information of the data to be scanned can be understood as the current data storage status of the data to be scanned, such as its data storage address and data information.

[0072] Based on this, the status information of the data to be scanned can be determined according to the scan results, and the actual access information of the data to be scanned can be determined according to the access logs of the data storage nodes. It is then determined whether the status information and access information of the data to be scanned match. If they match, it means that the status information of the data to be scanned is accurate and does not need to be changed, and the scan ends. If not, it means that there is a problem with the current status information of the data to be scanned. In this case, it is necessary to update the data to be scanned according to the access information recorded in the access logs, such as updating the status information of the data to be scanned.

[0073] For example, according to the scan results for file A, file A is stored in directory 1. However, according to the access log, file A has been moved to directory 2. This means that the actual storage location of file A is inconsistent with the recorded storage location. In this case, it is necessary to update the storage location of file A and move it to directory 2.

[0074] Alternatively, based on the scan results for directory 1, it can be known that directory 1 stores files A, B, and C. However, according to the access log for directory 1, file A in directory 1 has been deleted. In this case, it means that the status information of directory 1 does not match the actual storage information. At this point, the status information of directory 1 can be changed, that is, directory 1 stores files B and C.

[0075] In addition, the data scanning node can send an access log retrieval request to the data storage node to obtain the data storage node's access logs. Alternatively, the data scanning node can also include a data identifier of the data to be scanned in the access log retrieval request. The data storage node can then use this data identifier to retrieve the actual access information of the data to be scanned from the access logs and send this access information to the data scanning node so that the data scanning node can make a judgment based on the access information.

[0076] In summary, by utilizing access logs to determine the accuracy of the status information of the data to be scanned, status detection and consistency detection of the data to be scanned can be achieved, further ensuring the accuracy of data storage on data storage nodes and further improving the user experience.

[0077] In summary, the above system, by setting up a data scanning node, can load all or part of the data accessed by the user into the memory of the data scanning node when the user accesses data, and scan the data in memory, ensuring that user access and data scanning are carried out in parallel. Furthermore, by scanning the data loaded into memory, the system ensures that user access is not affected, thus guaranteeing a good user experience.

[0078] The following is in conjunction with the appendix Figure 2 Taking the application of the data processing method provided in this specification in metadata scanning as an example, the data processing method will be further explained. Figure 2 A flowchart illustrating the processing procedure of a data processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0079] Step 202: In response to a data access request, the data access node determines the data to be accessed corresponding to the data to be accessed in the data storage node based on the data to be accessed identifier carried in the data access request, and accesses the data to be accessed in the data storage node.

[0080] Step 204: Data access node, in the data access node, update the data to be accessed, obtain the updated data to be accessed, and convert the updated data to be accessed into target data according to preset rules, and store the target data in the data storage node.

[0081] In practical applications, when a user performs read or write operations on the data to be accessed, the data access node writes the memtable file (i.e., the updated data to be accessed) to memory. At preset intervals, when the amount of data in the memtable file reaches a preset threshold, the memtable file is converted into an SSTable file (the target data). Then, when the subsequent data scanning node wants to scan the data to be accessed, it loads the SSTable file from the data storage node into memory and scans the SSTable file in memory.

[0082] Step 206: The data scanning node, in response to a data scanning request for the target data, loads the target data into memory and scans the target data in the memory.

[0083] Step 208: The data scanning node obtains the scanning results for the data to be scanned, determines the status information of the data to be scanned based on the scanning results, determines the access information of the data to be scanned based on the access log of the data storage node, and determines whether the status information and the access information match. If they match, the scanning ends; otherwise, the data to be scanned is updated based on the access information.

[0084] In summary, the above method, by setting up a data scanning node, can load all or part of the data accessed by the user into the memory of the data scanning node when the user accesses data, and scan the data in memory, ensuring that user access and data scanning are carried out in parallel. Furthermore, by scanning the data loaded into memory, the user access is not affected, thus ensuring a good user experience.

[0085] Corresponding to the above system embodiments, see [link to relevant documentation]. Figure 3 , Figure 3 A flowchart of a data processing method according to an embodiment of this specification is shown. The data processing method is applied to a data processing system, which includes a data scanning node, a data storage node, and a data access node. The method specifically includes the following steps.

[0086] Step 302: In response to a data access request, the data access node determines the data to be accessed corresponding to the data to be accessed in the data storage node according to the data to be accessed identifier carried in the data access request, and accesses the data to be accessed in the data storage node.

[0087] Step 304: In response to a data scanning request, the data scanning node determines the data to be scanned corresponding to the data to be scanned identifier in the data storage node according to the data to be scanned identifier carried in the data scanning request, loads the data to be scanned into memory, and scans the data to be scanned in the memory, wherein the data to be accessed includes the data to be scanned.

[0088] In an optional embodiment, loading the data to be scanned into memory includes:

[0089] Based on the identifier of the data to be scanned, a data index associated with the data to be scanned is created in the memory, and the data to be scanned stored in the data storage node is loaded into the memory using the data index.

[0090] In an optional embodiment, accessing the data to be accessed in the data storage node includes:

[0091] In the data access node, the data to be accessed is updated to obtain the updated data to be accessed, and the updated data to be accessed is converted into target data according to a preset rule, and the target data is stored in the data storage node.

[0092] Accordingly, loading the data to be scanned into memory and scanning the data to be scanned in the memory includes:

[0093] The target data is loaded into the memory, and the target data is scanned in the memory.

[0094] In an optional embodiment, converting the updated data to be accessed into target data according to a preset rule includes:

[0095] The updated data to be accessed is converted into target data according to a preset time interval.

[0096] In an optional embodiment, converting the updated data to be accessed into target data according to a preset rule includes:

[0097] If the amount of updated data to be accessed reaches a preset data amount threshold, the updated data to be accessed will be converted into target data.

[0098] In an optional embodiment, the method further includes:

[0099] Obtain the scan results for the data to be scanned, and update the data to be scanned based on the scan results.

[0100] In an optional embodiment, the method further includes:

[0101] Based on the scan results, the status information of the data to be scanned is determined. Based on the access log of the data storage node, the access information of the data to be scanned is determined. It is then determined whether the status information and the access information match. If they match, the scan ends. If not, the data to be scanned is updated based on the access information.

[0102] In summary, the above method, by setting up a data scanning node, can load all or part of the data accessed by the user into the memory of the data scanning node when the user accesses data, and scan the data in memory, ensuring that user access and data scanning are carried out in parallel. Furthermore, by scanning the data loaded into memory, the user access is not affected, thus ensuring a good user experience.

[0103] Corresponding to the above method embodiments, this specification also provides a data processing system, including a data processing node and a data storage node, wherein the data processing node is used to execute the above data processing method, and the data storage node is used to store data.

[0104] Specifically, data processing nodes can include data access nodes and data scanning nodes.

[0105] Corresponding to the above system embodiments, see [link to relevant documentation]. Figure 4 , Figure 4 A flowchart of another data processing method according to an embodiment of this specification is shown. The method is applied to a data scanning node and specifically includes the following steps.

[0106] Step 402: In response to the data scanning request, determine the data to be scanned corresponding to the data to be scanned identifier carried in the data scanning request in the data storage node;

[0107] Step 404: Load the data to be scanned into memory, and scan the data to be scanned in the memory.

[0108] Specifically, in response to a data scan request, the system can determine the data to be scanned corresponding to the data identifier carried in the data scan request in the data storage node, load the data to be scanned into the memory of the data scan node, and scan the data to be scanned in memory. By using a separate server to scan the data to be scanned, the system avoids affecting access to the data in the data storage node, thereby further realizing the concurrency of data access and data scanning.

[0109] In summary, the above method, by setting up a data scanning node, can load all or part of the data accessed by the user into the memory of the data scanning node when the user accesses data, and scan the data in memory, ensuring that user access and data scanning are carried out in parallel. Furthermore, by scanning the data loaded into memory, the user access is not affected, thus ensuring a good user experience.

[0110] The above is an illustrative scheme of a data processing method according to this embodiment. It should be noted that the technical solution of this data processing method and the technical solution of the data processing system described above belong to the same concept. For details not described in detail in the technical solution of the data processing method, please refer to the description of the technical solution of the data processing system described above.

[0111] In practical applications, the data processing method provided in this specification is used as an example in metadata scanning within a distributed data storage system to further illustrate the data processing method. Specifically, Figure 5 A flowchart illustrating another data processing method provided in one embodiment of this specification is shown, specifically including the following steps.

[0112] Step 502: In response to the data scanning request, the scanning engine retrieves the SSTable file of the data to be scanned from the distributed data storage system.

[0113] The scanning engine can be understood as a server used to scan data, i.e., the aforementioned data scanning node. The distributed data storage system is the aforementioned data storage node.

[0114] Specifically, the scanning engine can read the meta file in the distributed data storage system, obtain a list of SStable files for the data to be scanned based on the meta file, and then retrieve any SStable file from that list.

[0115] Step 504: The scanning engine creates a directory in memory and hard links the sstable files through this directory.

[0116] Step 506: Load the sstable file into the scanning engine's memory.

[0117] Step 508: Scan the sstable file in the scanning engine's memory.

[0118] In summary, the above method uses a separate server (i.e., a scanning engine) to scan the data, scanning the state of metadata in the distributed data storage system at a certain moment, thereby verifying the consistency of metadata, without affecting users' operations on metadata in the distributed data storage system.

[0119] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 6 A schematic diagram of another data processing apparatus provided in one embodiment of this specification is shown. Figure 6 As shown, the device includes:

[0120] The determination module 602 is configured to, in response to a data scan request, determine the data to be scanned corresponding to the data to be scanned identifier carried in the data scan request in the data storage node;

[0121] The scanning module 604 is configured to load the data to be scanned into memory and scan the data to be scanned in the memory.

[0122] In summary, the above-mentioned device, by setting up a data scanning node, can load all or part of the data accessed by the user into the memory of the data scanning node when the user accesses data, and scan the data in memory, ensuring that user access and data scanning are carried out in parallel, and by scanning the data loaded into memory, it does not affect the user's access, thus ensuring a good user experience.

[0123] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0124] Figure 7 A structural block diagram of a computing device 700 according to one embodiment of this specification is shown. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0125] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0126] In one embodiment of this application, the aforementioned components of the computing device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0127] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.

[0128] The processor 720 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0129] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0130] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0131] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data processing method described above.

[0132] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data processing method.

[0133] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the data processing method described above belong to the same concept. Details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the data processing method described above.

[0134] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0135] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0136] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0137] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0138] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method applied to a data processing system, the data processing system comprising a data scanning node, a data storage node, and a data access node, the method comprising: In response to a data access request, the data access node determines the data to be accessed corresponding to the data to be accessed identifier carried in the data access request in the data storage node, and accesses the data to be accessed in the data storage node. When a user accesses the data to be accessed, the data scanning node, in response to a data scanning request, determines the data to be scanned corresponding to the data identifier carried in the data scanning request in the data storage node, loads the data to be scanned into memory, scans the data to be scanned in memory to obtain a scan result for the data to be scanned, determines the status information of the data to be scanned based on the scan result, determines the access information of the data to be scanned based on the access log of the data storage node, and determines whether the status information and the access information match. If they match, the scanning ends; if not, the data to be scanned is updated based on the access information. The data to be accessed includes the data to be scanned.

2. The method according to claim 1, wherein loading the data to be scanned into memory comprises: Based on the identifier of the data to be scanned, a data index associated with the data to be scanned is created in the memory, and the data to be scanned stored in the data storage node is loaded into the memory using the data index.

3. The method according to claim 1, further comprising: The data access node performs an update operation on the data to be accessed to obtain the updated data to be accessed, and converts the updated data to be accessed into target data according to a preset rule, and stores the target data in the data storage node; The data scanning node loads the target data into the memory and scans the target data in the memory.

4. The method according to claim 3, wherein converting the updated data to be accessed into target data according to a preset rule comprises: The updated data to be accessed is converted into target data according to a preset time interval.

5. The method according to claim 3, wherein converting the updated data to be accessed into target data according to a preset rule comprises: If the amount of updated data to be accessed reaches a preset data amount threshold, the updated data to be accessed will be converted into target data.

6. A data processing method applied to a data scanning node, comprising: In response to a data scanning request, the data to be scanned corresponding to the data to be scanned identifier carried in the data scanning request is determined in the data storage node. When a user accesses data to be accessed, including data to be scanned, the data to be scanned is loaded into memory, and the data to be scanned is scanned in memory to obtain scan results. Based on the scan results, the status information of the data to be scanned is determined. Based on the access logs of the data storage node, the access information of the data to be scanned is determined. It is then determined whether the status information and the access information match. If they match, the scan ends; otherwise, the data to be scanned is updated based on the access information.

7. A data processing system, comprising data processing nodes and data storage nodes, wherein, The data processing node is used to perform the method according to any one of claims 1-5, and the data storage node is used to store data.

8. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-6.

Images