Data processing method and device and computer program product

Abstract

The embodiment of the invention discloses a data processing method and device and a computer program product, and the data processing method comprises the steps: based on a target access moment in metadata corresponding to target data, storing the target data in a target storage space matched with the target access moment; wherein the target access moment is determined based on the historical access moment of the target data and the target access period, and the target storage space at least comprises a cold data storage space and a hot data processing space. High-concurrency data congestion can be avoided, and the data processing efficiency is improved.

Description

Technical Field

[0001] This application relates to data storage technology, and to, but is not limited to, a data processing method and device, and a computer program product. Background Technology

[0002] In data processing scenarios, it is often necessary to classify the data, and the time of data access is usually used as a criterion for classification.

[0003] In related technologies, high-concurrency scenarios may encounter problems such as task backlog caused by frequent data access and updates, which reduces the data processing efficiency in high-concurrency scenarios. Summary of the Invention

[0004] In view of this, the data processing method, apparatus, and computer program product provided in the embodiments of this application can avoid high-concurrency data congestion and improve data processing efficiency. The data processing method, apparatus, device, and storage medium provided in the embodiments of this application are implemented as follows: One aspect of this application provides a data processing method, including: Based on the target access time in the metadata corresponding to the target data, the target data is stored in the target storage space that matches the target access time; wherein, the target access time is determined based on the historical access time of the target data and the target access cycle, and the target storage space includes at least: cold data storage space and hot data processing space.

[0005] Another aspect of the embodiments of this application provides a data processing apparatus, including: a storage allocation module; The storage allocation module is used to store the target data in the target storage space that matches the target access time based on the target access time in the metadata corresponding to the target data. The target access time is determined based on the historical access time of the target data and the target access cycle. The target storage space includes at least: cold data storage space and hot data processing space.

[0006] The computer device provided in this application includes a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement the method of this application.

[0007] The computer-readable storage medium provided in this application embodiment stores a computer program thereon, which, when executed by a processor, implements the method provided in this application embodiment.

[0008] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.

[0009] The data processing method, apparatus, and computer program product provided in this application can store target data in a target storage space matching the target access time based on the target access time in the metadata corresponding to the target data. The target access time is determined based on the historical access times and target access cycles of the target data. The target storage space includes at least a cold data storage space and a hot data processing space. Since the target access time is determined based on the historical access times and target access cycles of the target data, the process of determining the target access time can determine whether it needs to be updated by identifying the access cycle of the historical access times. This reduces the update frequency of the target access time in the metadata corresponding to the target data, saving computing resources in high-concurrency scenarios, thereby improving data processing efficiency and avoiding data congestion and other problems. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of the data storage scenario provided in the embodiments of this application; Figure 2 This is a schematic diagram of the data processing scenario provided in the embodiments of this application; Figure 3 This is a flowchart illustrating the data processing method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the process for determining whether to update metadata, provided in an embodiment of this application. Figure 5 This is a schematic diagram illustrating the logic for determining whether to update metadata, as provided in an embodiment of this application. Figure 6 This is a schematic diagram of the process for updating based on target identification information provided in the embodiments of this application; Figure 7 This is a schematic diagram illustrating the logic of updating based on target identifier information provided in the embodiments of this application; Figure 8 This is another logical diagram illustrating the determination of whether to update metadata, provided in an embodiment of this application. Figure 9 This is a schematic diagram of the process for determining the target storage space provided in the embodiments of this application; Figure 10 This is another flowchart illustrating the process of determining the target storage space provided in the embodiments of this application; Figure 11 This is a schematic diagram of the computer device involved in the data processing method provided in the embodiments of this application; Figure 12 This is a schematic diagram of the process for hierarchical migration of metadata provided in the embodiments of this application; Figure 13 This is a schematic diagram of the structure of the metadata provided in the embodiments of this application; Figure 14 This is a schematic diagram illustrating the target access cycle of the target data provided in the embodiments of this application; Figure 15 This is a schematic diagram of the structure of the data processing device provided in the embodiments of this application; Figure 16 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0014] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0015] It should be noted that the terms "first, second, third" used in the embodiments of this application are used to distinguish similar or different objects and do not represent a specific order of objects. It can be understood that "first, second, third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0016] In data processing scenarios, it is often necessary to classify the data. For example, frequently accessed data can be classified as hot data, and infrequently accessed data as cold data. The boundary between "frequently accessed" and "infrequently accessed" can be distinguished based on preset judgment conditions. Furthermore, in actual implementation, more regions can be set according to actual needs, such as "warm data," whose access frequency can fall between hot and cold data. These regions can be configured accordingly based on actual needs, without specific restrictions here.

[0017] In the process of classifying different types of data based on the above methods, the time of data access is usually used as a criterion for classification.

[0018] The access time of the data can be information recorded in the metadata of the data. It can record the last or previous access times of the data. The specific level of granularity can be set according to actual needs. For example, the specific access time can be defined with minutes, seconds, milliseconds or even microseconds as the smallest unit.

[0019] However, in high-concurrency scenarios, there may be access to massive amounts of data. In related technologies, for each data access, the access time needs to be updated accordingly. Since the number of access time updates required in high-concurrency scenarios is huge, there may be problems such as task backlog caused by frequent data access time updates. These problems reduce the data processing efficiency in high-concurrency scenarios.

[0020] To address the aforementioned problems in related technologies, this application provides a data processing method. The specific scenarios in which the data processing method provided in this application is applied will be explained below.

[0021] One feasible scenario is a data storage scenario.

[0022] Figure 1 This is a schematic diagram illustrating the data storage scenario provided in the embodiments of this application. Please refer to... Figure 1 This scenario can be a storage scenario for a computer device, in which a data processing unit 110 can be set up, and the data storage determination can be performed through the data processing unit 110.

[0023] The computer device may include, but is not limited to, mobile phones, wearable devices (such as smartwatches, smart bracelets, smart glasses, etc.), tablet computers, laptops, in-vehicle terminals, PCs (Personal Computers), etc. In the embodiments of this application, the computer device may also be a cloud server, a dedicated server, or other server-side equipment. The function implemented by this method can be achieved by the processor in the computer device calling program code. Of course, the program code can be stored in the computer storage medium. Therefore, the computer device includes at least a processor and a storage medium.

[0024] The data processing unit 110 can determine the storage space that the data should be allocated based on the access time in the metadata of each data. The storage space can be set in various ways. In this embodiment, three different storage spaces can be set, namely cold data storage space 120, warm data storage space 130 and hot data storage space 140.

[0025] If the data is already in the corresponding storage space, no data migration is required. If the data is not in the corresponding storage space, the data needs to be migrated from the current storage space to the corresponding storage space for storage.

[0026] Among them, the access time in the data's metadata is the main basis for allocating storage space.

[0027] It should be noted that for the access time in the data's metadata, a corresponding update process can be set, the process of which is as follows: Figure 2 This is a schematic diagram of the data processing scenario provided in the embodiments of this application. Please refer to... Figure 2 During the process of updating the metadata of data, a metadata update task can be generated after the data is accessed, and the access time in the metadata can be updated based on the update task.

[0028] Alternatively, the task generation unit 210 can generate the corresponding update task and send it to the metadata database 220, thereby updating the corresponding access time in the metadata database.

[0029] In related technologies, in high-concurrency scenarios, that is, due to the generation of a large number of update tasks, the access time of the metadata database needs to be updated, resulting in frequent updates and thus causing problems such as task backlog.

[0030] Based on this, in order to solve such problems, data processing methods can be further improved on the basis of the above application scenarios to avoid problems such as task backlog and improve data processing efficiency in high-concurrency scenarios.

[0031] The following explains one possible implementation of the data processing method provided in the embodiments of this application.

[0032] Figure 3 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Please refer to it. Figure 3 A data processing method, comprising: S310: Based on the target access time in the metadata corresponding to the target data, store the target data in the target storage space that matches the target access time.

[0033] It should be noted that the subject executing this method can be the aforementioned computer device. In this embodiment, the computer device can be explained as a cloud server.

[0034] The target data can be any data already stored in the computer device. For example, it can be a file, such as an audio file, video file, text file, or image file, or any other type of file. Alternatively, it can be application data, such as various runtime data generated during the operation of an application. There are no specific restrictions here, and the definition of the target data can be set according to actual needs.

[0035] Metadata of target data can be descriptive information about that target data, providing key information such as the data's background, structure, source, quality, and usage. Metadata typically includes: data identification information, source information, structural information, quality information, usage information, maintenance information, and other relevant information.

[0036] The above information includes identification information such as dataset name, unique identifier, and version number. The dataset name can be the official name of the dataset for easy identification and reference. The unique identifier is used to uniquely identify the dataset's code or number. The version number is used to provide version information for the dataset, making it easy to track data updates and changes.

[0037] Source information may include data provider, data access time, data collection method, and data collection location, where the data access time may specifically be the target access time mentioned above.

[0038] Data structure information can include data format, data fields, field descriptions, and data relationships.

[0039] Quality information can include data integrity, data accuracy, data consistency, and data timeliness.

[0040] Usage information can include the intended use of the data, usage restrictions, and data licenses. Data purpose refers to the intended use of the data, such as for research, business analysis, or decision support. Usage restrictions refer to whether there are legal, policy, or ethical limitations on the use of the data, such as whether authorization is required or whether it can be used for commercial purposes. Data licenses refer to the permissions granted for the use of the data, such as open-source licenses or commercial licenses.

[0041] Maintenance information may include data update frequency, data maintenance manager, and data update log. Data update frequency refers to the data update cycle, such as daily, weekly, or monthly. The data maintenance manager is the individual or team responsible for data maintenance. The data update log records historical information about data updates, including update time and content.

[0042] Other information may include data annotations, data associations, and data citations. Data annotations refer to additional descriptions or notes about the data. Data associations refer to the relationships between the data and other datasets or resources. Data citations refer to the relevant citation information if the data has been cited or published.

[0043] It is important to note that metadata plays a crucial role in data management, data analysis, and data sharing. It helps users understand the context and structure of data, ensuring its correct use and effective management.

[0044] The target access time can be stored as one of the data in the aforementioned source information, and the last one or more access times of this data can be recorded.

[0045] The target access time is determined based on the historical access times of the target data and the target access cycle.

[0046] It should be noted that the historical access time refers to the actual time when the target data was accessed in the historical time, and the target access period can be a preset duration.

[0047] For example, the historical access time could be 4:12:36 on a certain day, and the target access period could be 5 minutes. The target access time can be the access time recorded in the metadata, and it is an access time that can be updated.

[0048] In one embodiment, the time can be divided into multiple specific access cycles based on the duration of the target access cycle. For example, 4:00 to 4:05 is one access cycle, 4:05 to 4:10 is one access cycle, and 4:10 to 4:15 is one access cycle. The target access time can be determined based on historical access times and any specific access cycle determined according to the target access cycle.

[0049] For example, if the target access time and the historical access time are in the same access period, the target access time does not need to be updated. If the target access time and the historical access time are not in the same access period, the target access time can be updated. The historical access time can be used as the target access time, or a new time corresponding to the access period in which the historical access time is located can be determined as the target access time. No specific restrictions are imposed here.

[0050] In one embodiment, after determining the target access time based on the above method, the target data can be stored in a target storage space that matches the target access time.

[0051] The target storage space includes at least: cold data storage space and hot data processing space.

[0052] In one embodiment, the target storage space may further include the warm data storage space explained above.

[0053] It should be noted that the hot data storage space uses high-performance SSD (Solid State Drive) storage, providing millisecond-level access latency, which is suitable for business scenarios that require real-time response; the warm data storage space uses standard storage or low-frequency access storage, striking a balance between performance and cost; the cold data storage space uses archive storage or deep archive storage, which significantly reduces storage costs and is suitable for data that needs to be stored for a long time but is accessed very infrequently.

[0054] In the data processing method provided in this application embodiment, the target data can be stored in a target storage space matching the target access time based on the target access time in the metadata corresponding to the target data. The target access time is determined based on the historical access times and target access cycles of the target data. The target storage space includes at least a cold data storage space and a hot data processing space. Since the target access time is determined based on the historical access times and target access cycles of the target data, the process of determining the target access time can determine whether it needs to be updated by identifying the access cycle of the historical access times. This reduces the update frequency of the target access time in the metadata corresponding to the target data, saving computing resources in high-concurrency scenarios, thereby improving data processing efficiency and avoiding data congestion and other problems.

[0055] The following explains one feasible implementation process for determining whether to update the metadata of the target data, as provided in the embodiments of this application.

[0056] It should be noted that, in the embodiments of this application, the updates to the metadata related to the target data are all updates to the target access time in the metadata.

[0057] Figure 4 This is a flowchart illustrating the process of determining whether to update metadata, as provided in this application embodiment. Please refer to... Figure 4 Before storing the target data in the target storage space matching the target access time, based on the target access time in the metadata corresponding to the target data, the method further includes: S410: Monitors the reading or writing of target data.

[0058] It should be noted that the existence of reading or writing of target data can be determined based on whether reading or writing operations on the target data occur, thereby enabling monitoring of reading or writing of target data.

[0059] Here, writing data refers to the process of writing data into any storage space of the computer device, while reading data refers to reading data from any storage space of the computer device.

[0060] When reading or writing target data, S420: Determine whether to update the target access time in the metadata corresponding to the target data based on the historical access time of the target data and the target access cycle.

[0061] It should be noted that when reading or writing to target data is detected, the target access time can be updated. For example, it can be determined whether the target access time needs to be updated based on the historical access time of the target data and the target access cycle.

[0062] The historical access time can be determined in different ways depending on whether data is read or written.

[0063] During the data writing process, the moment when the computer device receives the data is the aforementioned historical access moment; during the data reading process, the moment when the computer device reads the data from the storage space is the aforementioned historical access moment.

[0064] After obtaining the historical access times using the above method, the period of the target access time can be determined.

[0065] S430: Do not update the target access time if no target data is read or written.

[0066] Correspondingly, if no reading or writing of the target data is detected, there is no need to update the target access time.

[0067] The data processing method provided in this application embodiment can monitor the reading or writing of target data. When target data is read or written, it determines whether to update the target access time in the metadata corresponding to the target data based on the historical access time and target access cycle of the target data. When target data is not read or written, the target access time is not updated. By monitoring the writing or reading of target data, it is possible to accurately monitor whether the metadata of the target data needs to be updated.

[0068] The following section will provide a more detailed explanation of the steps involved in determining the update of the aforementioned metadata.

[0069] Figure 5 This is a schematic diagram illustrating the logic for determining whether to update metadata, as provided in the embodiments of this application. Please refer to... Figure 5 Based on the historical access times and target access cycles of the target data, determine whether to update the target access time in the metadata corresponding to the target data, including: S510: Determine whether the access period corresponding to the historical access time and the access period corresponding to the target access time are the same access period.

[0070] It should be noted that, as explained above, the target access period can indicate a duration, and the access period can be specifically divided based on this duration. For example, starting from 0:00, every 5 minutes is considered an access period. Each specific access period has a corresponding time interval, and the access period corresponding to the target access time and the access period corresponding to the historical access time can be determined separately, thereby determining whether these two access periods are the same access period.

[0071] For example, if the target access time is 4:11 and the historical access time is 4:13, and the target access period is 5 minutes, the corresponding access period is from 4:10 to 4:15. Then it can be determined that the access period corresponding to the target access time and the access period corresponding to the historical access time are the same period. Conversely, if the historical access time is 4:20, then it can be determined that the access period corresponding to the target access time and the access period corresponding to the historical access time are not the same period.

[0072] If the access period corresponding to the historical access time is not the same as the access period corresponding to the target access time, S520: Update the target access time.

[0073] It should be noted that if the access period corresponding to the historical access time is not the same as the access period corresponding to the target access time, it can be determined that the target access time needs to be updated, and the update steps for the target access time can be executed.

[0074] If the access period corresponding to the historical access time is the same as the access period corresponding to the target access time, then S530: Maintain the target access time.

[0075] It should be noted that if the access period corresponding to the historical access time is determined to be the same as the access period corresponding to the target access time, it can be determined that the target access time does not need to be updated, and the target access time stored in the current metadata can be maintained.

[0076] The data processing method provided in this application embodiment can determine whether the access period corresponding to the historical access time and the access period corresponding to the target access time are the same access period. If the access period corresponding to the historical access time and the access period corresponding to the target access time are not the same access period, the target access time is updated. If the access period corresponding to the historical access time and the access period corresponding to the target access time are the same access period, the target access time is maintained. By determining whether the access period corresponding to the target access time and the access period corresponding to the historical access time are the same period, it is possible to quickly and accurately determine whether the target access time needs to be updated.

[0077] The following explains another feasible implementation process for updating and determining the target access time in the metadata, as provided in the embodiments of this application.

[0078] Figure 6 This is a schematic diagram of the update determination based on target identifier information provided in the embodiments of this application. Please refer to... Figure 6 When reading or writing target data, the method further includes: S610: Generate target identification information.

[0079] It should be noted that a cache component can be set up in the computer device to store the aforementioned target identification information. After the target data is read or written, target identification information for that target data can be generated.

[0080] The target identification information is used to indicate whether the target data has been accessed within a preset time threshold, which is determined based on the target access cycle.

[0081] It should be noted that each target identifier can be timed after it is generated. The duration of the timer can be the aforementioned preset time threshold, which can be determined based on the target access cycle. For example, if the target access cycle is 5 minutes, the corresponding specific access cycle is between 4:05 and 4:10. If the historical access time is 4:06, then the preset time threshold can be the time difference between the historical access time and the end time of the corresponding access cycle. There are 4 minutes between 4:06 and 4:10, which means the preset time threshold is 4 minutes.

[0082] Correspondingly, if the historical access time is 4:08, there are 2 minutes between 4:08 and 4:10, which means the preset time threshold is 2 minutes.

[0083] In determining whether the target access time and the historical access time belong to the same access period, the determination can be made based on the existence of the aforementioned target identification information.

[0084] In other words, the target identification information can be used to mark whether the target data has been accessed during the corresponding access period.

[0085] In one embodiment, after generating the target identification information, the method further includes: S620: If the generation time of the target identification information is greater than or equal to a preset time threshold, delete the target identification information.

[0086] It should be noted that the duration of the target identification information can be determined by a timer countdown. If it is within the preset time threshold, the target identification information can be retained; if it exceeds the preset time threshold, the target identification information can be deleted.

[0087] In one embodiment, since the target identification information is essentially an identifier used to mark whether the target data has been accessed within a certain period of time, if the time interval to be determined is exceeded, that is, the corresponding access period, the target identification information can be deleted to indicate that the target data has not been accessed in the new access period.

[0088] The data processing method provided in this application embodiment can generate target identification information and delete it if the generation time of the target identification information is greater than or equal to a preset time threshold. By generating and deleting target identification information, it is possible to more quickly and accurately determine whether the target access time in the metadata needs to be updated. In particular, updating based on target identification information can improve the accuracy and efficiency of the determination.

[0089] The following is a detailed explanation of the specific implementation process of updating and determining based on target identification information in the data processing method provided in the embodiments of this application.

[0090] Figure 7 This is a schematic diagram illustrating the logic of updating based on target identifier information provided in the embodiments of this application. Please refer to... Figure 7 Determining whether the access period corresponding to the historical access time and the access period corresponding to the target access time are the same access period includes: S710: Detects whether target identification information exists.

[0091] It should be noted that after receiving a target data, you can first check whether the target identifier information corresponding to the target data exists in the cache component.

[0092] Among them, the target identification information for the same target data can be fixed. The detection process can be achieved by detecting whether a certain target identification information exists, and then the corresponding steps can be executed based on whether the target identification information exists.

[0093] If target identification information is detected, S720: Determine that the access period corresponding to the historical access time and the access period corresponding to the target access time are the same access period.

[0094] It should be noted that when target identification information is detected, it can be determined that there is access to the target data within a preset time threshold. Since the preset time threshold is determined based on the target access cycle, it can be determined that the access cycle corresponding to the historical access time and the access cycle corresponding to the target access time are the same access cycle.

[0095] If no target identification information is detected, S730: Determine that the access period corresponding to the historical access time is not the same access period as the access period corresponding to the target access time.

[0096] It should be noted that if no target identification information is detected, it can be determined that there is no access to the target data within the preset time threshold. Since the preset time threshold is determined based on the target access cycle, it can be determined that the access cycle corresponding to the historical access time is different from the access cycle corresponding to the target access time.

[0097] The data processing method provided in this application embodiment can detect whether target identification information exists; if target identification information is detected, it is determined that the access period corresponding to the historical access time and the access period corresponding to the target access time are the same access period; if target identification information is not detected, it is determined that the access period corresponding to the historical access time and the access period corresponding to the target access time are not the same access period. Specifically, by determining whether target identification information is detected, it is possible to accurately determine whether the access period corresponding to the historical access time and the access period corresponding to the target access time are the same access period, thereby further determining whether to update the target access time.

[0098] It should be noted that when it is determined that metadata needs to be updated, the metadata can be updated by generating an update task. The following explains one of the feasible implementation processes for updating metadata based on update tasks provided in the embodiments of this application.

[0099] Figure 8 For another logical diagram illustrating the determination of whether to update metadata provided in this application embodiment, please refer to... Figure 8 Update the target access time, including: S810: Generate target access time update task.

[0100] In one embodiment, if it is determined that the target access time needs to be updated, a target access time update task can be generated, which can be used to access the metadata database to update the target access time.

[0101] Optionally, in order to further reduce the frequency of metadata access, after generating the update task, it can be determined whether the target data has been accessed in the current access period. This determination can be based on whether an update task for the target access time has been generated in the current access period.

[0102] If other target access time update tasks for the target data are generated during the current access period, it can be determined that the metadata of the target data has been updated during the current access period. Conversely, if no other target access time update tasks for the target data are generated during the current access period, it can be determined that the metadata of the target data has not been updated during the current access period.

[0103] S820: If the metadata of the target data has not been updated within the current access period, update the metadata of the target data in the task access metadata database based on the target access time to update the target access time.

[0104] In one embodiment, if the metadata of the target data has not been updated within the current access period, it can be determined that this update is the first update within the current access period. Therefore, the update step can be performed, and the metadata of the target data in the task access metadata database can be updated based on the target access time, thereby updating the target access time, that is, updating the target access time to the historical access time.

[0105] The target access time update task can store relevant information about historical access times.

[0106] In one embodiment, the target access time can be updated using historical access times. For example, if the target access time is 4:16 and the historical access time is 4:31, and the metadata of the target data has not been updated within the current access period, the target access time can be updated to 4:31.

[0107] The target access time update task can also store relevant information for the current access period. For example, it can normalize historical access times within the same access period. Specifically, it can normalize all historical access times from 4:30 to 4:35 to 4:30. Therefore, if the target access time is 4:16 and the normalized historical access time is 4:30, and the target data's metadata has not been updated within the current access period, the target access time can be updated to 4:30.

[0108] It should be noted that the normalization method is not limited to the above. It can take any time within the access period, such as the start time, end time, or intermediate time. No specific restrictions are imposed here.

[0109] In one embodiment, after generating the target access time update task, the method further includes: S830: If the metadata of the target data has been updated within the current access period, the update task returns update completion information based on the target access time.

[0110] In one embodiment, if the metadata of the target data has been updated within the current access period, it can be determined that this update is not the first update within the current access period. Therefore, the update step can be skipped, and the update task can be directly updated based on the target access time to return update completion information. That is, the target access time in the metadata database will not be updated, but update completion information of the completed update task will be returned.

[0111] The data processing method provided in this application embodiment can generate a target access time update task. If the metadata of the target data has not been updated within the current access period, the target access time update task accesses the metadata of the target data in the metadata database to update the target access time. If the metadata of the target data has been updated within the current access period, the target access time update task returns update completion information. By determining whether the metadata has been updated within the current access period, frequent access to the metadata database can be further avoided, thereby reducing data processing pressure and improving data processing efficiency.

[0112] After determining the target historical moment in the metadata of the target data using the above method, the data can be stored based on that target historical moment.

[0113] Figure 9 This is a schematic diagram of the process for determining the target storage space provided in the embodiments of this application. Please refer to... Figure 9 The target data is stored in the target storage space that matches the time of target access, including: S910: Determine the corresponding target storage space based on the target access time of the target data and the target duration from the current time.

[0114] The current moment can be the moment when the storage location of the target data needs to be determined. For example, for each piece of target data, after it is stored in the storage space of the computer device, the above steps can be performed to determine whether the data needs to be migrated, that is, whether the target data needs to be migrated from the current storage space to other storage spaces.

[0115] The target duration can be the time difference between the current time and the target access time. The target duration can be compared with a preset duration threshold to determine the corresponding target storage space.

[0116] For example, a first duration threshold and a second duration threshold can be set. The first duration threshold is greater than the second duration threshold.

[0117] If the target duration is greater than or equal to the first duration threshold, the target storage space can be determined to be a cold data storage space; if the target duration is less than the first duration threshold but greater than or equal to the second duration threshold, the target storage space can be determined to be a warm data storage space; if the target duration is less than the second duration threshold, the target storage space can be determined to be a hot data storage space.

[0118] S920: Store the target data in the target storage space.

[0119] After determining the target storage space, the target data can be stored in the target storage space. If the target data is not in the target storage space, it can be migrated from the current storage space to the target storage space; if the target data is already in the target storage space, it can remain in its current storage location without migration.

[0120] In the data processing method provided in this application embodiment, the corresponding target storage space can be determined based on the target duration from the target access time of the target data to the current time; the target data is then stored in the target storage space. Determining the target storage space based on the target duration ensures the accuracy and rationality of data storage and improves data processing efficiency.

[0121] The above storage method is determined based on the target duration. In actual implementation, the target storage space corresponding to the target data can also be determined based on other information.

[0122] Figure 10 This is another flowchart illustrating the process of determining the target storage space provided in this application embodiment. Please refer to... Figure 10 The target data is stored in the target storage space that matches the time of target access, including: S1010: Determine the access frequency of the target data based on multiple target access times of the target data.

[0123] It should be noted that the target access frequency can be determined based on multiple target access times. For any two adjacent target access times, the corresponding time interval can be determined. By calculating the average value of the time interval between any two adjacent target access times, the access frequency of the target data can be obtained. The smaller the average value, the higher the access frequency of the target data; the higher the average value, the lower the access frequency of the target data.

[0124] S1020: Determine the corresponding target storage space based on the access frequency of the target data.

[0125] In one embodiment, the target storage space can be determined by comparing the access frequency of the target data with a preset frequency threshold.

[0126] For example, a first frequency threshold and a second frequency threshold can be set. The first frequency threshold is less than the second frequency threshold.

[0127] If the target frequency is less than or equal to the first frequency threshold, the target storage space can be determined to be a cold data storage space; if the target frequency is greater than the first frequency threshold and less than or equal to the second frequency threshold, the target storage space can be determined to be a warm data storage space; if the target frequency is greater than the second frequency threshold, the target storage space can be determined to be a hot data storage space.

[0128] S1030: Store the target data in the target storage space.

[0129] After determining the target storage space, the target data can be stored in the target storage space. If the target data is not in the target storage space, it can be migrated from the current storage space to the target storage space; if the target data is already in the target storage space, it can remain in its current storage location without migration.

[0130] In the data processing method provided in this application embodiment, the access frequency of the target data can be determined based on multiple target access times, and then the corresponding target storage space can be determined based on the access frequency of the target data; the target data is then stored in the target storage space. Determining the target storage space based on the access frequency ensures the accuracy and rationality of data storage and improves data processing efficiency.

[0131] In actual implementation, besides being based on Figure 9 The target duration shown determines the target storage space and Figure 10 In addition to determining the target storage space based on the access frequency shown, the two methods mentioned above can be combined. For example, simultaneously satisfying the target duration and access frequency conditions can determine the specific target storage space. No specific restrictions are imposed here; one or a combination of these methods can be chosen based on actual needs.

[0132] To more clearly explain the data processing method provided in the embodiments of this application, the metadata update process will be explained below through an overall structure.

[0133] Figure 11 This is a schematic diagram of the computer equipment involved in the data processing method provided in the embodiments of this application. Please refer to... Figure 11 First, let's explain the various levels of components involved in computer equipment: The computer device may include: a storage access layer 1110, a metadata database 1120, a data storage layer 1130, a metadata access layer 1140, a data migration component 1150, and a caching component 1160.

[0134] The storage access layer 1110 serves as the user access point for the storage system, handling all user data uploads and downloads. The metadata database 1120, within the distributed storage system, allows user-uploaded complete data files to be fragmented and stored across different disks on different physical machines. Complete file object metadata (such as filename, file size, file type, access time, creation time, and data fragment location information) is stored in the metadata database cluster. The data storage layer 1130 performs data storage; the aforementioned hot, warm, and cold data storage spaces can all be areas within this layer. Data files are partitioned into different storage layers based on their hot or cold data type. The metadata access layer 1140 collects user access information for data files and analyzes and identifies hot and cold data based on data file access analysis, deduplication, and access time stamping strategies. The metadata access layer also provides metadata database scanning capabilities to classify the analyzed hot and cold data files, generate hot-cold migration conversion tasks, and submit them to the message queue for further data migration processing. The storage access layer 1110 can send metadata update tasks, such as the target access time update task mentioned above, to the metadata access layer 1140 through the message queue. The data migration component 1150 reads the data migration tasks from the message queue and performs data input / output operations to read the files from the hot storage layer and write them to the cold storage layer. During the migration process, CRC64 and hash calculations are performed on the data to ensure data consistency. After the data migration is completed, the data shard location information in the metadata database is updated to point to the new data location, and the old hot data is deleted and space is reclaimed. The migration service can be horizontally scaled and migrated in parallel through the message queue to complete the large-scale data cooling migration. The role of the caching component 1160 has been explained above and will not be repeated here.

[0135] In one embodiment, the storage access layer 1110 can determine whether target data is being written or read, and can determine whether to update metadata. The metadata access layer 1140 can further determine whether to access the metadata database; the determination process in steps S810-S830 can be implemented through this layer. The data migration component 1150 can be used to implement data migration and storage; the execution processes in steps S910-S920 and S1010-S1030 can be implemented based on this component.

[0136] In one embodiment, data migration can be performed using a cooling method, as detailed below: Figure 12 This is a schematic diagram of the layered migration process for metadata provided in the embodiments of this application. Please refer to... Figure 12The aforementioned process for determining the target storage space can be configured with corresponding timing. For example, for data in the hot data storage space, the above determination can be performed periodically to determine whether the hot data needs to be migrated to the warm or cold data storage space.

[0137] Correspondingly, for data in the warm data storage space, the above judgment can also be performed every once in a while to determine whether the warm data needs to be migrated to the cold data storage space.

[0138] Figure 12 The diagram only shows the process of migration from hot to cold. In actual implementation, a migration process from cold to hot can also be set up. No specific restrictions are made here.

[0139] The following explains the structural relationship of the metadata stored in the metadata database provided in the embodiments of this application.

[0140] Figure 13 This is a schematic diagram of the metadata structure provided in the embodiments of this application. Please refer to... Figure 13 Metadata can be a set of data information in a metadata database for storing target data. This metadata can include various types of information. For a target data, the metadata stored in the metadata database can include: volume, file location, metadata ontology, creation time, data type, and target access time.

[0141] Here, "volume" refers to the volume in which the data is stored, and its unique identifier (key). In distributed storage systems, data may be divided into multiple volumes for easier management and expansion. "File location" indicates the specific location of the data file within the storage system. This can be a file path, an address on a storage device, or a pointer to the actual storage location of the data. "Metadata ontology" can be descriptive information about the data, describing its attributes such as file format, size, and content type. "Creation time" refers to the timestamp when the data was created, helping to understand its freshness and historical history. "Data type" may be a numeric identifier used to distinguish different types of data or files. For example, 0 might represent one type of data, while 1 might represent another.

[0142] The following explains the specific relationship between the target access cycles of the target data provided in the embodiments of this application.

[0143] Figure 14 This is a schematic diagram illustrating the target access cycle of the target data provided in the embodiments of this application. Please refer to... Figure 14The entire timeline can be divided into multiple time periods. The target time period refers to the duration of each time period, such as 5 minutes. Any target access time and historical access time will be in one of the access periods. In the process of determining whether the target access time needs to be updated, it can be determined based on the historical access time and the target access period, so as to obtain the access period in which the historical access time is located, and thus realize the process of determining whether the access periods are the same.

[0144] The data processing method provided in this application embodiment can effectively address the intelligent hot and cold data stratification of massive data in cloud storage services. It boasts high performance, high reliability, easy scalability, and strong data consistency. The method's design concept is simple and clear, its technical implementation is relatively easy, and it has strong scalability. Through modular design and standardized interfaces, this solution can be quickly integrated into existing cloud storage systems, reducing the difficulty of technology promotion. The ease of implementation provides an important foundation for its large-scale application. The solution is simple in design, easy to implement, and has strong scalability. This solution can effectively solve the challenges posed by the massive update volume of file object access time in high-concurrency and hot data access scenarios. Simultaneously, it can effectively solve the performance and scalability issues caused by hot and cold data migration at the hundreds of petabyte level, theoretically enabling unlimited-scale hot and cold data conversion. Furthermore, by performing CRC64 and hash calculations on the data during migration, data consistency can be fully guaranteed, preventing data integrity issues caused by errors or interruptions during the migration process.

[0145] It should be understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0146] Based on the foregoing embodiments, this application provides a data processing device, which includes the included modules and the units included in each module, which can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0147] Figure 15 This is a schematic diagram of the data processing device provided in the embodiments of this application. Please refer to... Figure 15 In another aspect of the embodiments of this application, a data processing apparatus is also provided, including: a storage allocation module 1510; The storage allocation module 1510 is used to store the target data in a target storage space that matches the target access time based on the target access time in the metadata corresponding to the target data; wherein, the target access time is determined based on the historical access time of the target data and the target access cycle, and the target storage space includes at least: cold data storage space and hot data processing space.

[0148] The device also includes an update determination module 1520, which is used to determine whether to update the target access time in the metadata corresponding to the target data based on the historical access time and target access cycle of the target data when reading or writing the target data.

[0149] In one embodiment, the update determination module 1520 is specifically used to determine whether the access period corresponding to the historical access time and the access period corresponding to the target access time are the same access period; if the access period corresponding to the historical access time and the access period corresponding to the target access time are not the same access period, the target access time is updated; if the access period corresponding to the historical access time and the access period corresponding to the target access time are the same access period, the target access time is maintained.

[0150] In one embodiment, the update determination module 1520 is further configured to generate target identification information, which is used to indicate that target data has been accessed within a preset time threshold, and the preset time threshold is determined based on the target access cycle.

[0151] In one embodiment, the update determination module 1520 is further configured to delete the target identification information if the generation time of the target identification information is greater than or equal to a preset time threshold.

[0152] In one embodiment, the update determination module 1520 is specifically used to determine, when the existence of target identification information is detected, that the access period corresponding to the historical access time and the access period corresponding to the target access time are the same access period; and to determine, when the existence of target identification information is not detected, that the access period corresponding to the historical access time and the access period corresponding to the target access time are not the same access period.

[0153] In one embodiment, the update determination module 1520 is specifically used to generate a target access time update task; if the metadata of the target data has not been updated within the current access period, the target access time update task accesses the metadata of the target data in the metadata database to update the target access time.

[0154] In one embodiment, the update determination module 1520 is further configured to, within the current access period, if the metadata of the target data has been updated, return update completion information based on the target access time update task.

[0155] In one embodiment, the storage allocation module 1510 is specifically used to determine the corresponding target storage space based on the target access time of the target data and the target duration from the current time; and to store the target data in the target storage space.

[0156] In one embodiment, the storage allocation module 1510 is specifically used to determine the access frequency of the target data based on multiple target access times of the target data; determine the corresponding target storage space based on the access frequency of the target data; and store the target data in the target storage space.

[0157] The data processing apparatus provided in this application embodiment can store target data in a target storage space matching the target access time based on the target access time in the metadata corresponding to the target data. The target access time is determined based on the historical access times and target access cycles of the target data. The target storage space includes at least a cold data storage space and a hot data processing space. Since the target access time is determined based on the historical access times and target access cycles of the target data, the process of determining the target access time can determine whether it needs to be updated by identifying the access cycle of the historical access times. This reduces the update frequency of the target access time in the metadata corresponding to the target data, saving computing resources in high-concurrency scenarios, thereby improving data processing efficiency and avoiding data congestion and other problems.

[0158] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0159] It should be noted that, in the embodiments of this application... Figure 15The module division of the data processing device shown is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or be integrated into one unit with two or more units. The integrated units can be implemented in hardware, as software functional units, or a combination of both.

[0160] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0161] Figure 16 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Please refer to... Figure 16 This application provides a computer device, which may be a cloud server, and its internal structure diagram may be as follows. Figure 16 As shown, the computer device includes a processor 1620, memory, and a network interface 1640 connected via a system bus 1610. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium 1631 and internal memory 1632. The non-volatile storage medium 1631 stores an operating system, computer programs, and a database. The internal memory 1632 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium 1631. The database stores data. The network interface 1640 communicates with external terminals via a network connection. When the computer program is executed by the processor 1620, it implements the aforementioned methods.

[0162] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in the above embodiments.

[0163] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.

[0164] Those skilled in the art will understand that Figure 16 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0165] In one embodiment, the data processing apparatus provided in this application can be implemented as a computer program, and the computer program can be implemented as follows: Figure 16 The device operates on the computer device shown. The memory of the computer device can store the various program modules that make up the above-described apparatus. The computer program, composed of the various program modules, causes the processor to execute the steps of the methods in the various embodiments of this application described in this specification.

[0166] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0167] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0168] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0169] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0171] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0172] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0173] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0174] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0175] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0176] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0177] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0178] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, include: Based on the target access time in the metadata corresponding to the target data, the target data is stored in a target storage space that matches the target access time; wherein, the target access time is determined based on the historical access time and target access cycle of the target data, and the target storage space includes at least: a cold data storage space and a hot data processing space.

2. The method according to claim 1, characterized in that, Before storing the target data in a target storage space matching the target access time based on the target access time in the metadata corresponding to the target data, the method further includes: When reading or writing the target data, determine whether to update the target access time in the metadata corresponding to the target data based on the historical access time and target access period of the target data.

3. The method according to claim 2, characterized in that, The step of determining whether to update the target access time in the metadata corresponding to the target data based on the historical access time and target access period of the target data includes: Determine whether the access period corresponding to the historical access time is the same as the access period corresponding to the target access time; If the access period corresponding to the historical access time is not the same as the access period corresponding to the target access time, update the target access time. If the access period corresponding to the historical access time is the same as the access period corresponding to the target access time, the target access time is maintained.

4. The method according to claim 3, characterized in that, When reading or writing the target data, the method further includes: Generate target identification information, which indicates that the target data has been accessed within a preset time threshold, and the preset time threshold is determined based on the target access cycle.

5. The method according to claim 4, characterized in that, The step of determining whether the access period corresponding to the historical access time and the access period corresponding to the target access time are the same access period includes: If the target identification information is detected, it is determined that the access period corresponding to the historical access time and the access period corresponding to the target access time are the same access period; If the target identification information is not detected, it is determined that the access period corresponding to the historical access time is not the same access period as the access period corresponding to the target access time.

6. The method according to claim 3, characterized in that, The updating of the target access time includes: The task is updated at each target access time; If the metadata of the target data has not been updated within the current access period, the metadata of the target data in the task access metadata database is updated based on the target access time to update the target access time.

7. The method according to claim 1, characterized in that, The step of storing the target data in a target storage space that matches the target access time includes: The target storage space is determined based on the target access time of the target data and the target duration from the current time. The target data is stored in the target storage space.

8. The method according to claim 1, characterized in that, The step of storing the target data in a target storage space that matches the target access time includes: The access frequency of the target data is determined based on multiple target access times of the target data; The target storage space is determined based on the access frequency of the target data. The target data is stored in the target storage space.

9. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images