Data processing methods, systems, electronic devices and computer program products

Abstract

This application discloses a data processing method, system, electronic device, and computer program product. Relating to the field of big data technology, the method includes: obtaining data cooling rules; modifying the state of a file in a first storage device that is in a second state and meets the data cooling rules to a third state; and migrating the file in the third state from the first storage device to a second storage device. The file in the first storage device transitions from the first state to the second state when write access is prohibited. The first state indicates support for appending data, the second state indicates no support for appending data, and the third state indicates no support for appending data or real-time data reading. The read / write speed of the first storage device is higher than that of the second storage device. This application solves the technical problems of high cost and low efficiency in data cooling of files that support appending uploads in related technologies.

Description

Technical Field

[0001] This application relates to the field of big data technology, and more specifically, to a data processing method, system, electronic device, and computer program product. Background Technology

[0002] With the acceleration of digital transformation, the amount of data generated by enterprises and individuals is growing explosively. This data includes not only information needed for daily operations, but also a large amount of historical records and backup files that are not frequently accessed but need to be stored long-term. For the latter, using standard storage or high-performance storage methods such as hot storage is costly. Cold storage services, on the other hand, can reduce storage costs by optimizing hardware configuration and reducing redundancy, while ensuring data security and persistence. Related technologies use cold storage services to store historical records and backup files.

[0003] It's important to note that for some streaming data, such as audit logs, video surveillance, and live video streams, users can append recently updated data to the end of an existing file in real time, ensuring immediate readability. However, for products that support appending, data cooling requires either downloading the appended file locally and then re-uploading it to the cooling device, or copying the appended file as a regular file and configuring lifecycle rules to convert it into a cooled file. The problem with this is that if cooling a large number of existing files is required, manually downloading, uploading, and copying the files sequentially is inefficient and generates numerous requests, leading to resource waste.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a data processing method, system, electronic device, and computer program product to at least solve the technical problems of high cost and low efficiency in the related art when performing data cooling on files that support additional uploads.

[0006] According to one aspect of the embodiments of this application, a data processing method is also provided, applied to a backend service, comprising: obtaining data cooling rules, wherein the data cooling rules are rules for converting the data type from a hot data type to a cold data type; modifying the state of a file in a first storage device that is in a second state and satisfies the data cooling rules to a third state, and migrating the file in the third state from the first storage device to a second storage device, wherein the file in the first storage device is converted from the first state to the second state when writing is prohibited, the first state indicates that append writing is supported, the second state indicates that append writing is not supported, the third state indicates that append writing is not supported and real-time reading of data is not supported, and the read / write speed of the first storage device is higher than that of the second storage device.

[0007] Optionally, the method further includes: when a target data retrieval task associated with the target file in the second storage device is polled, constructing a copy file of the target file based on the target data retrieval task, wherein the target data retrieval task is generated based on a data read request of the target file; storing the copy file in the first storage device, and changing the state of the target file from the third state to the second state.

[0008] Optionally, after obtaining the data cooling rules, the method further includes: if the data cooling rule is a first type of cooling rule, performing a step of changing the state of files in the first storage device that are in the second state and satisfy the data cooling rule to the third state, wherein the first type of cooling rule indicates that files that have been prohibited from writing data are being cooled; if the data cooling rule is a second type of cooling rule, changing the state of the set of files in the first storage device that conform to the second type of cooling rule from the first state to the third state, and migrating the set of files from the first storage device to the second storage device, wherein the second type of cooling rule indicates that data is cooled in batches, and the set of files includes at least one file.

[0009] Optionally, obtaining data cooling rules includes: reading configuration data to obtain data cooling rules, wherein the configuration data includes data cooling rules, and the data cooling rules include a first type of cooling rule and a second type of cooling rule. The first type of cooling rule indicates that files that have been prohibited from writing data should be cooled, and the second type of cooling rule indicates that data should be cooled in batches.

[0010] Optionally, the data cooling rules include at least one of the following: cooling files associated with preset tags; cooling files whose storage time exceeds a first preset duration; cooling files whose last access time exceeds a second preset duration; and cooling files whose size exceeds a preset size.

[0011] According to one aspect of the embodiments of this application, a data processing method is provided, applied to a front-end service, comprising: determining a target file based on a data write-prohibition request sent by a client, wherein the data write-prohibition request indicates that data should not be written to the target file on a first storage device; modifying the state of the target file from a first state to a second state, wherein the target file in the second state in the first storage device is transferred to the second storage device if a data cooling rule is satisfied, wherein the data cooling rule is a rule that converts the data type from a hot data type to a cold data type, the first state indicates that append writing of data is supported, the second state indicates that append writing of data is not supported, and the read / write speed of the first storage device is higher than that of the second storage device.

[0012] Optionally, after changing the state of the target file from the first state to the second state, the method further includes: upon receiving a data read request sent by the client, if the data read request indicates reading data from the target file, generating a target data retrieval task, wherein the target data retrieval task is used to instruct the construction of a copy file of the target file and store the copy file in the first storage device; responding to the completion instruction of the target data retrieval task, reading data from the copy file from the first storage device and sending the data from the copy file to the client.

[0013] According to another aspect of the embodiments of this application, a data processing system is also provided, comprising: a client, configured to send a data read / write request or a data write-prohibition request for a target file to a front-end service, and receive data returned by the front-end service; a front-end service, configured to perform data read / write operations on the target file if the target file is in a first state upon receiving a data read / write request for the target file, and further configured to modify the state of the target file from the first state to a second state upon receiving a data write-prohibition request for the target file, wherein the first state indicates that append writing is supported, and the second state indicates that append writing is not supported; a back-end service, configured to modify the state of a file that meets the data cooling rule and is in the second state to a third state, and migrate the file in the third state from a first storage device to a second storage device, wherein the data cooling rule is a rule that converts the data type from a hot data type to a cold data type, and the third state indicates that append writing is not supported and real-time data reading is not supported; a first storage device, configured to store data written by the client; and a second storage device, configured to store data migrated from the first storage device, wherein the read / write speed of the first storage device is higher than that of the second storage device.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods in various embodiments of this application.

[0017] In this embodiment, for a target file in the first storage device that can be appended to, the front-end service responds to a data write-prohibition request actively sent by the user through the client, performing a state transition on the target file. This changes the target file from a first state that supports appending data to a second state that does not support appending data, laying the foundation for data cooling and avoiding data inconsistency issues during migration. The back-end service then further determines whether the target file in the second state meets the data cooling rules. If the target file meets both conditions, the back-end service initiates a data migration process, changing the target file from the second state to a third state that does not support appending data or real-time data reading. The target file is then migrated from the high-cost, high-speed first storage device to the low-cost, low-speed second storage device. This allows the target file to be efficiently cooled when frequent read / write operations are no longer needed, saving storage costs and meeting the long-term storage needs for large amounts of historical data. This solves the technical problems of high cost and low efficiency in data cooling of files that support appending to uploads in related technologies.

[0018] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1A hardware structure block diagram of a computer terminal (or mobile device) implementing a data processing method according to an embodiment of this application;

[0021] Figure 2 A structural block diagram of a computing environment according to an embodiment of this application;

[0022] Figure 3 This is a flowchart of a data processing method according to an embodiment of this application. Figure 1 ;

[0023] Figure 4 This is a flowchart of a data processing method according to an embodiment of this application. Figure 2 ;

[0024] Figure 5 This is a flowchart of a data processing method according to an embodiment of this application. Figure 3 ;

[0025] Figure 6 This is a schematic diagram of a data processing system according to an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of a data processing apparatus according to an embodiment of this application. Figure 1 ;

[0027] Figure 8 This is a schematic diagram of a data processing apparatus according to an embodiment of this application. Figure 2 ;

[0028] Figure 9 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0032] Example 1

[0033] According to an embodiment of this application, a data processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) implementing a data processing method according to an embodiment of this application. (See diagram below.) Figure 1 As shown, computer terminal 10 (or mobile device) may include one or more ( Figure 1 The processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions may also be included. In addition, it may include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0035] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method in the embodiments of this application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the method in the above embodiments. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0037] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0038] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0039] Figure 1The hardware structure block diagram shown can serve not only as an exemplary block diagram of the aforementioned computer terminal 10 (or mobile device), but also as an exemplary block diagram of the aforementioned server. In one optional embodiment, Figure 2 The use of the above is illustrated in a block diagram. Figure 1 The computer terminal 10 (or mobile device) shown is an embodiment of a computing node in computing environment 201. Figure 2 A structural block diagram of a computing environment according to an embodiment of this application, such as... Figure 2 As shown, the computing environment 201 includes multiple ( ) running on a distributed network. Figure 2 The compute nodes (such as servers) are represented by 210-1, 210-2, ..., . Each compute node contains local processing and memory resources, and end user 202 can remotely run applications or store data within the compute environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3, and 220-4 within the compute environment 201, representing services "A", "D", "E", and "H", respectively.

[0040] End user 202 can provide and access services through a web browser or other software application on a client. In some embodiments, the provisioning and / or requests of end user 202 can be provided to ingress gateway 230. Ingress gateway 230 may include a corresponding agent to handle the provisioning and / or requests for services (one or more services provided in computing environment 201).

[0041] Services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services may be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can simulate a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the machine is virtualized by a virtual machine, container-based virtualization can launch containers to virtualize an entire operating system (OS), allowing multiple workloads to run on a single OS instance.

[0042] In one embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, such as Figure 2As shown, service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). A Pod can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers within a Pod handle requests related to one or more corresponding functions of the service. Proxy 245 typically controls service-related network functions such as routing and load balancing. Other services can also be equipped with similar Pods.

[0043] During operation, executing a user request from end user 202 may require invoking one or more services in computing environment 201, and executing one or more functions of one service may require invoking one or more functions of another service. For example... Figure 2 As shown, service "A" 220-1 receives user requests from terminal user 202 from ingress gateway 230. Service "A" 220-1 can call service "D" 220-2, and service "D" 220-2 can request service "E" 220-3 to perform one or more functions.

[0044] The aforementioned computing environment can be a cloud computing environment, where resource allocation is managed by cloud services, allowing functionality development without needing to consider implementation, adjustment, or server scaling. This computing environment allows developers to execute event-responsive code without building or maintaining complex infrastructure. Services can be partitioned into a set of functions that can automatically and independently scale, rather than scaling a single hardware device to handle potential loads.

[0045] Under the aforementioned operating environment, this application provides the following: Figure 3 The data processing method shown is applied to the backend service. Figure 3 This is a flowchart of a data processing method according to an embodiment of this application. Figure 1 .

[0046] Step S31: Obtain data cooling rules, where data cooling rules are rules that convert the data type from hot data type to cold data type.

[0047] Among them, data cooling rules are a set of predefined strategies or conditions used to determine whether it is allowed to change the data type from a hot data type that supports high-frequency read and write to a cold data type that supports low-frequency read and write. Data cooling rules may include, but are not limited to, the last modification time of the file, access frequency, file size, specific time window, or user-defined conditions.

[0048] By obtaining the data cooling rules, the backend service can identify which files in the first storage device meet the cooling rules, providing a basis for data cooling processing.

[0049] Step S32: Change the state of the file in the first storage device that is in the second state and meets the data cooling rule to the third state, and migrate the file in the third state from the first storage device to the second storage device. The file in the first storage device changes from the first state to the second state when writing is prohibited. The first state indicates that append writing is supported, the second state indicates that append writing is not supported, and the third state indicates that append writing is not supported and real-time reading is not supported. The read and write speed of the first storage device is higher than that of the second storage device.

[0050] For example, the backend service can periodically scan the status of all files in the first storage device to check which files are in the second state. Files in the second state are those that have been explicitly indicated by the user as not to have any further data written to them, and whose status has been modified by the frontend service. In other words, they no longer need to be written to frequently and can be cooled. The backend service identifies these files and provides a dynamic list of candidate files for data cooling.

[0051] When the backend service finds a file in the second state (which prohibits writing) and meets the previously obtained data cooling rules during the scanning process, it can initiate the data migration process to change the file's state from the second state to the third state, indicating that the file will no longer accept real-time data reading, and migrate the file from the first storage device to the second storage device.

[0052] It should be noted that the first storage device is a high-speed, high-cost storage device, such as a SATA (Serial Advanced Technology Attachment hard drive) or SSD (Solid State Drive). The second storage device is a low-speed, low-cost storage device, such as an HDD (Hard Disk Drive) or magnetic tape. The backend service migrates files that are already in the second state and meet the data cooling rules from the first storage device to the second storage device, achieving the goal of transferring infrequently accessed files to low-cost storage media and reducing storage costs.

[0053] In this embodiment, for a target file in the first storage device that can be appended to, the front-end service responds to a data write-prohibition request actively sent by the user through the client, performing a state transition on the target file. This changes the target file from a first state that supports appending data to a second state that does not support appending data, laying the foundation for data cooling and avoiding data inconsistency issues during migration. The back-end service then further determines whether the target file in the second state meets the data cooling rules. If the target file meets both conditions, the back-end service initiates a data migration process, changing the target file from the second state to a third state that does not support appending data or real-time data reading. The target file is then migrated from the high-cost, high-speed first storage device to the low-cost, low-speed second storage device. This allows the target file to be efficiently cooled when frequent read / write operations are no longer needed, saving storage costs and meeting the long-term storage needs for large amounts of historical data. This solves the technical problems of high cost and low efficiency in data cooling of files that support appending to uploads in related technologies.

[0054] To enable users to easily read the cooled data, optionally, in the data processing method of this application embodiment, the method further includes: when a target data retrieval task associated with the target file in the second storage device is polled, constructing a copy file of the target file based on the target data retrieval task, wherein the target data retrieval task is generated based on the data read request of the target file; storing the copy file in the first storage device, and modifying the state of the target file from the third state to the second state.

[0055] The data retrieval task is generated by the front-end service. When it receives a request from a client to read the target file of the cooled data, the front-end service generates a corresponding data retrieval task according to the request. The back-end service executes a polling mechanism to periodically check whether there are any generated data retrieval tasks and process them in a timely manner. This ensures that the back-end service can respond promptly to the data retrieval requests generated by the front-end service, improving the response speed and efficiency of data cooling.

[0056] When the backend service identifies a target data retrieval task during polling, it constructs a copy of the target file based on that task. Specifically, the target file is retrieved from the second storage device, a complete and readable copy is created based on it, and then stored on the first storage device. The first storage device has high-speed read / write capabilities; storing the copy on the first storage device ensures that data retrieval can be completed quickly, meeting users' immediate data access needs.

[0057] While storing the copy file on the primary storage device, the backend service modifies the target file's state from a third state (not supporting real-time data reading) to a second state (supporting real-time data reading). This ensures that users will not encounter reading obstacles due to file state when accessing chilled data. This state adjustment is for the copy file, making it readable on the primary storage device. When users access chilled files through the frontend service, the copy file is retrieved directly from the high-speed read / write primary storage device, providing a better reading experience. It's important to note that readability is provided through the copy file, rather than directly migrating the target file to the primary storage device. This ensures the integrity of the chilled file's storage on the secondary storage device and reduces the complexity of data management associated with data migration.

[0058] This embodiment achieves rapid recovery and access to cooled data by having the backend service poll the data retrieval tasks generated by the frontend service, constructing a copy of the cooled target file, storing the copy on a high-speed read-only first storage device, and adjusting the state of the target file. This enhances the flexibility and efficiency of data management, maintains the integrity and security of cooled data, and improves the user experience when accessing cooled data.

[0059] When performing data cooling, individual files can be cooled, or files can be cooled in batches. Optionally, in the data processing method of this application embodiment, after obtaining the data cooling rules, the method further includes: if the data cooling rule is a first type of cooling rule, performing a step of changing the state of files in the first storage device that are in the second state and satisfy the data cooling rule to the third state, wherein the first type of cooling rule indicates that files that have been prohibited from writing data are cooled; if the data cooling rule is a second type of cooling rule, changing the state of the set of files in the first storage device that meet the second type of cooling rule from the first state to the third state, and migrating the set of files from the first storage device to the second storage device, wherein the second type of cooling rule indicates that data is cooled in batches, and the set of files includes at least one file.

[0060] The first type of cooling rule instructs the cooling of files for which write access is prohibited. This is used to handle files for which users have explicitly indicated that they will no longer be updated via write-prohibition commands. The second type of cooling rule instructs for batch cooling of data. This is suitable for scenarios where users want to automatically or in batches perform data cooling, such as automatically determining the set of files to be cooled based on file access frequency, modification time, or size. After obtaining the data cooling rules, the backend service first needs to determine which type of data cooling rule it belongs to, and then adopt different data cooling processing strategies according to the rule type to ensure the efficiency and targeting of the cooling operation.

[0061] When the data cooling rule is set to the first type, the backend service polls files in the first storage device that are in the second state, searching for files that match the data cooling rule. This ensures that all files that are write-prohibited and meet the data cooling rule are identified in a timely manner, thus preparing for the subsequent migration of data to the second storage device and avoiding data loss.

[0062] If the data cooling rule is the second type, which indicates batch data cooling, the backend service first searches for and determines the set of files that meet the data cooling rule on the first storage device. Then, it changes the status of the files in the set from a first state (supporting append-only writes) to a second state (not supporting append-only writes or real-time reads), ensuring that the files are not accidentally modified before migration and protecting data integrity.

[0063] After the state of the file collection is modified, the backend service migrates the files in the file collection from the first storage device to the second storage device with lower storage costs, thereby achieving data cooling. This ensures long-term data preservation while effectively controlling storage costs and improving the overall utilization efficiency of storage resources.

[0064] This embodiment provides two cooling methods by determining the type of data cooling rule. It can cool a single file that has been prohibited from being written to, and it also supports batch cooling of a set of files that meet specific conditions. This enhances the flexibility of data cooling processing, improves the efficiency of data cooling operations, and reduces the storage cost of cooled data while ensuring long-term storage of historical data.

[0065] Data cooling rules can be configured in advance. Optionally, in the data processing method of this application embodiment, configuration data is read to obtain data cooling rules. The configuration data includes data cooling rules, which include a first type of cooling rule and a second type of cooling rule. The first type of cooling rule indicates that files that have been prohibited from writing data should be cooled, and the second type of cooling rule indicates that data should be cooled in batches.

[0066] Among them, data cooling rules can be configured in the control center of the object storage service. The control center of the object storage service is responsible for storing and managing various configuration data. Users can specify data storage policies, lifecycle management rules and data cooling rules (including first-type cooling rules and second-type cooling rules) through the interface of the control center, making data management more personalized and efficient.

[0067] The first type of cooling rule instructs to cool files that have been prohibited from being written to. It is used to handle files that users have explicitly indicated through data write prohibition instructions will no longer be updated. This ensures that files that are no longer being updated can be migrated to a second storage device with low storage costs in a timely manner to save storage costs while maintaining data persistence and security.

[0068] The second type of cooling rule indicates batch cooling of data, which is suitable for scenarios where users want to perform data cooling automatically or in batches. For example, it can automatically determine the set of target files that need to be cooled in batches based on the access frequency, modification time or size of the files, making the data cooling operation more automated, reducing manual intervention by users and improving the efficiency of large-scale data cooling.

[0069] In this embodiment, users can customize a first data cooling rule or a second data cooling rule according to their own needs and costs, and store them in the control center. The backend service reads the data cooling rule from the control center and applies the data cooling rule to perform data cooling. Whether cooling a single file that has been prohibited from being written to, or batch cooling a large set of files, efficient and personalized implementation can be achieved.

[0070] The content of the data cooling rules can be flexibly set. Optionally, in the data processing method of this application embodiment, the data cooling rules include at least one of the following: cooling files associated with preset tags; cooling files whose storage time exceeds a first preset duration; cooling files whose last access time exceeds a second preset duration; and cooling files whose file size exceeds a preset size.

[0071] For example, data cooling rules based on preset tags can be configured. Specifically, users can set cooling rules for a set of files with preset tags. Preset tags are identifiers that users attach to specific files and can be used to categorize or mark the purpose, importance, or storage strategy of files. For example, all files with the "history" tag are automatically cooled after 30 days of upload. Data cooling rules based on preset tags allow users to categorize and manage files according to their attributes or uses, ensuring that specific types of files can be automatically migrated to cold storage when they are not frequently accessed.

[0072] For example, a data cooling rule based on upload time can be configured. Specifically, the user can set a first preset duration, such as automatically cooling files uploaded more than 90 days ago. This upload time-based data cooling rule can automatically identify and cool files that have not been updated for a long time, preventing large amounts of inactive data from consuming costly primary storage resources.

[0073] For example, a data cooling rule based on the last access time can be configured. Specifically, the user can set a second preset duration, such as automatically cooling a file if it has been more than 60 days since its last access. This ensures that files are automatically cooled when they have not been accessed within a certain period, preventing premature cooling due to occasional fluctuations in access frequency and releasing storage resources on the primary storage device in a timely manner.

[0074] For example, a data cooling rule based on file size can be configured. Specifically, the user sets a preset size, which is a user-defined file size threshold, to filter out files that occupy a large amount of storage space but are accessed infrequently for cooling. For example, files larger than 500MB are cooled. This targeted cooling of large files significantly reduces storage costs.

[0075] This embodiment provides cooling rules based on preset tags, upload time, last access time, and file size. Users can set appropriate cooling rules according to different scenarios and needs to achieve automatic and efficient data migration, helping users save storage costs, ensuring the persistence and security of important data, and avoiding the occupation of high-performance storage resources by inactive data.

[0076] Example 2

[0077] Under the aforementioned operating environment, this application provides the following: Figure 4 The data processing method shown is applied to the front-end service. Figure 4 This is a flowchart of a data processing method according to an embodiment of this application. Figure 2 .

[0078] Step S41: Determine the target file based on the data write-prohibition request sent by the client, wherein the data write-prohibition request indicates that data should not be written to the target file of the first storage device.

[0079] For example, a client refers to a software application or interface that a user can operate, such as a cloud storage client, a browser control panel, or a mobile application, which is used to initiate data read requests, data write requests (including data append write requests), and can also initiate data prohibition requests for append-only files.

[0080] It's important to note that a data write-prohibition request is a special type of request used to instruct the front-end service to modify the state of a specified appendable file (i.e., the target file), prohibiting further data write operations. When a user no longer wants to write data to a particular appendable file but instead needs to save it as a history or archive file, they can send a data write-prohibition request to the front-end service through the client. This request includes the target file's identifier information and the information of the primary storage device where the target file resides.

[0081] After the front-end service receives the write-prohibition request sent by the user through the client, it parses the write-prohibition request and identifies the target file in the first storage device that needs to be prevented from writing data.

[0082] Step S42: Change the state of the target file from the first state to the second state. The target file in the second state in the first storage device is transferred to the second storage device if the data cooling rule is met. The data cooling rule is a rule that changes the data type from a hot data type to a cold data type. The first state indicates that append writing is supported, the second state indicates that append writing is not supported, and the read / write speed of the first storage device is higher than that of the second storage device.

[0083] It should be noted that the first state indicates support for appending data, which can be called the append-only state, meaning the target file can accept new data appended and can also be read in real time. The second state indicates that appending data is not supported, which can be called the non-append-only state, meaning the target file prohibits further append operations, but can still be read in real time.

[0084] After receiving a request to prohibit writing data, the front-end service modifies the metadata of the target file, changing the target file's status from the first state to the second state, indicating that the target file will no longer accept new data appends, thereby ensuring the stability of the file content and providing conditions for the subsequent data cooling of the back-end service.

[0085] The backend service is the component responsible for performing data cooling operations. It periodically scans the file status on the primary storage device and the data cooling rules on the object storage service. When a target file is in the second state, the backend service considers it to meet the preconditions for data cooling and then further determines whether it meets the data cooling rules. Data cooling rules are a set of predefined strategies or conditions used to determine whether it is permissible to convert data from a hot data type that supports high-frequency read / write operations to a cold data type that supports low-frequency read / write operations. Data cooling rules can include, but are not limited to, the file's last modification time, access frequency, file size, a specific time window, or user-defined conditions.

[0086] If the target file is in the second state and meets the data cooling rules, the backend service can perform a data migration operation to change the target file's state from the second state to the third state, indicating that the target file will no longer accept real-time data reading, and migrate the target file from the first storage device to the second storage device.

[0087] It should be noted that the first storage device is a high-speed, high-cost storage device, such as a SATA (Serial Advanced Technology Attachment hard drive) or SSD (Solid State Drive). The second storage device is a low-speed, low-cost storage device, such as an HDD (Hard Disk Drive) or magnetic tape. The backend service migrates target files that are already in the second state and meet the data cooling rules from the first storage device to the second storage device, achieving the goal of transferring files that are no longer frequently accessed to low-cost storage media and reducing storage costs.

[0088] In this embodiment, for a target file in the first storage device that can be appended to, the front-end service responds to a data write-prohibition request actively sent by the user through the client, performing a state transition on the target file. This changes the target file from a first state that supports appending data to a second state that does not support appending data, laying the foundation for data cooling and avoiding data inconsistency issues during migration. The back-end service then further determines whether the target file in the second state meets the data cooling rules. If the target file meets both conditions, the back-end service initiates a data migration process, changing the target file from the second state to a third state that does not support appending data or real-time data reading. The target file is then migrated from the high-cost, high-speed first storage device to the low-cost, low-speed second storage device. This allows the target file to be efficiently cooled when frequent read / write operations are no longer needed, saving storage costs and meeting the long-term storage needs for large amounts of historical data. This solves the technical problems of high cost and low efficiency in data cooling of files that support appending to uploads in related technologies.

[0089] To ensure the integrity and immutability of the cooled data, optionally, in the data processing method of this application embodiment, after modifying the state of the target file from the first state to the second state, the method further includes: upon receiving a data read request sent by the client, if the data read request indicates reading data in the target file, generating a target data retrieval task, wherein the target data retrieval task is used to instruct the construction of a copy file of the target file and store the copy file in the first storage device; in response to the completion instruction of the target data retrieval task, reading the data in the copy file from the first storage device and sending the data in the copy file to the client.

[0090] It's important to note that a data write request refers to a request sent by the user through the client, which can be a request to append new data to a target file. When the target file is in state one, it indicates that appending data is allowed, and the frontend service responds to the data write request and performs the data write operation. When the target file is in state two, it indicates that appending data is prohibited, and the frontend service no longer responds to data write requests to prevent data from being scattered across different storage devices, affecting the integrity of the data in the target file, and increasing the complexity of data management.

[0091] For example, after the target file's status is changed from a first state where appending data is allowed to a second state where appending data is prohibited, the front-end service continuously listens for any data write requests from the client, especially append write requests. When a data write request is received, the front-end service checks whether the request is associated with a target file that was previously prohibited from appending data. When a data write request requests to modify a target file that has been prohibited from being written to, the front-end service no longer responds to the data write request and performs a data write operation. Instead, it immediately generates a prompt message and sends the prompt message to the client. The prompt message is used to inform the user that writing data to the target file is currently prohibited, ensuring that the user has a real-time understanding of the target file's status.

[0092] This embodiment prevents users from attempting to write to the cooled file by responding to data write requests after writing to the target file is prohibited, thereby protecting the integrity and immutability of the cooled data, further enhancing the stability and efficiency of data cooling, ensuring that historical data is not disturbed after cooling, and enabling users to clearly understand the file status and operation restrictions, thus improving the user experience.

[0093] To enable users to conveniently read the cooled data, optionally, in the data processing method of this application embodiment, after changing the state of the target file from the first state to the second state, the method further includes: upon receiving a data read request sent by the client, if the data read request indicates reading data in the target file, generating a target data retrieval task, wherein the backend service constructs a copy file of the target file based on the target data retrieval task, stores the copy file in the first storage device, and changes the state of the target file from the third state to the second state; in response to the completion instruction of the target data retrieval task, reading the data from the copy file from the first storage device, and sending the data from the copy file to the client.

[0094] It should be noted that when the front-end service receives a data read request from the client, and the request explicitly indicates that the target file to be read is in a second state where it cannot be appended to, the front-end service generates a target data retrieval task. By generating the target data retrieval task, the front-end service explicitly indicates the user's needs to the back-end service, namely, the need to access the cooled target file, thereby providing instructions for the rapid recovery and access of the cooled data.

[0095] After receiving the target data retrieval task, the backend service constructs a copy of the target file based on the task. This copy is then sent and stored on the first storage device. The first storage device has high-speed read / write capabilities; storing the copy file on the first storage device ensures that data retrieval can be completed quickly, meeting the user's need for immediate data access.

[0096] To make the copy of the target file readable, the backend service changes the target file's status from the third state to the second state. This change in the target file's status indicates to the user that the target file is readable, ensuring that the user will not encounter any reading obstacles due to the file's status when reading the cold data. It should be noted that readability is provided through the copy file, rather than directly migrating the target file to the first storage device. This ensures the storage integrity of the cold file in the second storage device and reduces the complexity of data management caused by data migration.

[0097] Once the backend service completes retrieving the target file and building a copy, and stores the copy file on the primary storage device, it sends a completion command to the frontend service. Upon receiving the completion command, the frontend service reads the data from the copy file on the primary storage device and sends this data to the client, thus fulfilling the client's data retrieval needs. This ensures that users can promptly access the cooled data that was originally stored on the low-cost but slow-access secondary storage device.

[0098] This embodiment generates a target data retrieval task through a front-end service, enabling the back-end service to build a copy file based on the target data retrieval task and store the copy file on a first storage device. Users can read and utilize the cooled data as if it were data in the first storage device, enhancing the flexibility and efficiency of data management, maintaining the integrity and security of the cooled data, and improving the user experience when accessing the cooled data.

[0099] Example 3

[0100] Under the aforementioned operating environment, this application provides the following: Figure 5 The data processing method shown. Figure 5 This is a flowchart of a data processing method according to an embodiment of this application. Figure 3 .like Figure 5 As shown, the method includes:

[0101] Files that users append to are stored on the first storage device, where they are writable and can be appended to and read in real time. The backend server does not allow files in a writable state to be migrated to the second storage device, which is a cold storage device.

[0102] For a file that is writable, a user can initiate a write-block request. Upon successful request, the file will become writable, preventing appending data while allowing real-time data reading. Subsequent append attempts will fail. Files already in a writable state can be configured to never return to a writable state.

[0103] Users can configure data cooling rules. When a file in a non-writable state meets the data cooling rules, the backend service can change the file's state to a cooled state, making it impossible to append data or read data in real time, and then migrate the file to a second storage device to achieve file cooling.

[0104] If a user needs to read a cooled file, they must first execute a data retrieval request. The data can only be read again after the backend service has migrated it to the primary storage device and generated a readable copy.

[0105] This embodiment converts an appendable file into a non-writable state by disabling write requests, clearly defining the boundaries of the file state. This clarifies the preconditions for whether data can be cooled. Without requiring additional requests from the user, historical data can be efficiently cooled by simply configuring data cooling rules, thereby reducing storage costs.

[0106] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0108] Example 4

[0109] Under the aforementioned operating environment, this application provides the following: Figure 6 The data processing system shown. Figure 6 This is a schematic diagram of a data processing system according to an embodiment of this application.

[0110] The client is used to send data read / write requests or data write-prohibition requests to the front-end service and to receive data returned by the front-end service.

[0111] In this context, the client refers to the software application or interface that users can operate, such as a cloud storage client, browser control panel, or mobile application. It is used to initiate data read and write requests (including read requests and write requests, with append requests included). It can also initiate write-prohibition requests for append-only files. A write-prohibition request is a special request used to instruct the front-end service to modify the state of a specified append-only file (i.e., the target file), prohibiting further data write operations and preparing for data cooling.

[0112] It should be noted that when a user no longer wants to write data to a file that can be appended, but instead needs to save it as a history or archive file, they can send a data write-prohibition request to the front-end service through the client. The data write-prohibition request contains the identification information of the target file and the information of the first storage device where the target file is located.

[0113] The front-end service is used to perform data read / write operations when it receives a data read / write request for the target file and the target file is in the first state. It is also used to change the state of the target file from the first state to the second state when it receives a request to prohibit writing data to the target file. The first state indicates that appending data is supported, and the second state indicates that appending data is not supported.

[0114] The front-end service is the user interaction layer of the object storage service. It is responsible for receiving client requests, including data read / write requests and requests to prohibit write operations, and performing corresponding operations based on the file status. The first status indicates that appending data is supported; this can be called the append-only state, meaning the target file can accept new data appended and can be read in real time. The second status indicates that appending data is not supported; this can be called the non-append-only state, meaning the target file prohibits further append operations, but can still be read in real time.

[0115] After receiving a request to prohibit writing data, the front-end service modifies the metadata of the target file, changing the target file's status from the first state to the second state. This signifies that the target file will no longer accept new data appends, thus ensuring the stability of the file content and providing conditions for data cooling in subsequent back-end services.

[0116] The backend service is used to change the status of files that meet the data cooling rules and are in the second state to the third state, and to migrate files in the third state from the first storage device to the second storage device. The data cooling rules are rules that change the data type from a hot data type to a cold data type. The third state indicates that appending data is not supported and real-time data reading is not supported.

[0117] The backend service is the automated processing layer of object storage, responsible for performing data cooling and migration operations. Based on data cooling rules and file status, it migrates eligible files from the primary storage device, which has higher cost and performance, to the secondary storage device, which has lower cost but slower read and write speeds.

[0118] The backend service periodically scans the file status on the first storage device and the data cooling rules on the object storage service. When a target file is in the second state, the backend service considers it to meet the preconditions for data cooling and then further determines whether it meets the data cooling rules. Data cooling rules are a set of predefined strategies or conditions used to determine whether it is permissible to convert frequently accessed "hot" data into less frequently accessed "cold" data. Data cooling rules include, but are not limited to, the file's last modification time, access frequency, file size, specific time windows, or user-defined conditions. If the target file is in the second state and meets the data cooling rules, the backend service can perform a data migration operation, changing the target file's status from the second state to the third state, indicating that the target file will no longer accept real-time data reads, and migrating the target file from the first storage device to the second storage device. By migrating a target file that is already in the second state and meets the data cooling rules from the first storage device to the second storage device, the backend service transfers files that are no longer frequently accessed to low-cost storage media, reducing storage costs.

[0119] The first storage device is used to store the data written by the client.

[0120] The first storage device is a storage device with high read and write speed and high storage cost, such as SATA (Serial Advanced Technology Attachment hard drive) or SSD (Solid State Drive).

[0121] A second storage device is used to store data migrated from the first storage device, wherein the storage cost of the first storage device is higher than that of the second storage device, and the read / write speed of the first storage device is higher than that of the second storage device.

[0122] The second storage device is a storage device with low read / write speed and low storage cost, such as an HDD (Hard Disk Drive) or magnetic tape.

[0123] In this embodiment, for a target file in the first storage device that can be appended to, the front-end service responds to a data write-prohibition request actively sent by the user through the client, performing a state transition on the target file. This changes the target file from a first state that supports appending data to a second state that does not support appending data, laying the foundation for data cooling and avoiding data inconsistency issues during migration. The back-end service then further determines whether the target file in the second state meets the data cooling rules. If the target file meets both conditions, the back-end service initiates a data migration process, changing the target file from the second state to a third state that does not support appending data or real-time data reading. The target file is then migrated from the high-cost, high-speed first storage device to the low-cost, low-speed second storage device. This allows the target file to be efficiently cooled when frequent read / write operations are no longer needed, saving storage costs and meeting the long-term storage needs for large amounts of historical data. This solves the technical problems of high cost and low efficiency in data cooling of files that support appending to uploads in related technologies.

[0124] Example 5

[0125] According to an embodiment of this application, a data processing apparatus for implementing the above-described data processing method is also provided, applied to post-service. Figure 7 This is a schematic diagram of a data processing apparatus according to an embodiment of this application. Figure 1 ,like Figure 7 As shown, the device includes:

[0126] The acquisition unit 701 is used to acquire data cooling rules, wherein the data cooling rules are rules that convert the data type from hot data type to cold data type;

[0127] Migration unit 702 is used to change the state of a file in the first storage device that is in the second state and meets the data cooling rules to the third state, and migrate the file in the third state from the first storage device to the second storage device. The file in the first storage device is changed from the first state to the second state when writing is prohibited. The first state indicates that append writing is supported, the second state indicates that append writing is not supported, and the third state indicates that append writing is not supported and real-time reading is not supported. The read and write speed of the first storage device is higher than that of the second storage device.

[0128] Optionally, the apparatus further includes: a copy file construction unit, configured to construct a copy file of the target file based on the target data retrieval task when a target data retrieval task associated with the target file is polled in the second storage device, wherein the target data retrieval task is generated based on a data read request of the target file; and a copy storage unit, configured to store the copy file in the first storage device and modify the state of the target file from the third state to the second state.

[0129] Optionally, the apparatus further includes: a first cooling unit, configured to, when the data cooling rule is a first type of cooling rule, perform a step of changing the state of files in the first storage device that are in a second state and satisfy the data cooling rule to a third state, wherein the first type of cooling rule indicates cooling files for which writing is prohibited; and a second cooling unit, configured to, when the data cooling rule is a second type of cooling rule, change the state of a set of files in the first storage device that conform to the second type of cooling rule from the first state to the third state, and migrate the set of files from the first storage device to the second storage device, wherein the second type of cooling rule indicates batch cooling of data, and the set of files includes at least one file.

[0130] Optionally, the acquisition unit 701 is further configured to read configuration data to obtain data cooling rules, wherein the configuration data includes data cooling rules, and the data cooling rules include a first type of cooling rule and a second type of cooling rule. The first type of cooling rule indicates that files that have been prohibited from writing data should be cooled, and the second type of cooling rule indicates that data should be cooled in batches.

[0131] Optionally, the data cooling rules include at least one of the following: cooling files associated with preset tags; cooling files whose storage time exceeds a first preset duration; cooling files whose last access time exceeds a second preset duration; and cooling files whose size exceeds a preset size.

[0132] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0133] Example 6

[0134] According to an embodiment of this application, a data processing apparatus for implementing the above-described data processing method is also provided, applied to a front-end service. Figure 8 This is a schematic diagram of a data processing apparatus according to an embodiment of this application. Figure 2 ,like Figure 8 As shown, the device includes:

[0135] The parsing unit 801 is used to determine the target file based on the data write-prohibition request sent by the client, wherein the data write-prohibition request indicates that data should not be written to the target file of the first storage device;

[0136] Modification unit 802 is used to modify the state of the target file from a first state to a second state. The target file in the second state in the first storage device is transferred to the second storage device if the data cooling rule is satisfied. The data cooling rule is a rule that changes the data type from a hot data type to a cold data type. The first state indicates that append writing is supported, the second state indicates that append writing is not supported, and the read / write speed of the first storage device is higher than that of the second storage device.

[0137] Optionally, the apparatus further includes: a task generation unit, configured to, after changing the state of the target file from a first state to a second state, generate a target data retrieval task if a data read request sent by the client indicates reading data from the target file, wherein the target data retrieval task is used to instruct the construction of a copy file of the target file and the storage of the copy file in a first storage device; and a response unit, configured to, in response to the completion instruction of the target data retrieval task, read data from the copy file from the first storage device and send the data from the copy file to the client.

[0138] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 2, but are not limited to the schemes provided in Embodiment 2.

[0139] Example 7

[0140] Embodiments of this application may provide an electronic device, which may be any one of a group of electronic devices. Optionally, in this embodiment, the aforementioned electronic device may also be replaced by a terminal device such as a mobile terminal.

[0141] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0142] In this embodiment, the computer terminal described above can execute the program code in the method.

[0143] Optionally, Figure 9 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 9 As shown, the electronic device A may include: one or more ( Figure 9 (Only one is shown) processor 102, memory 104, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0144] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0145] The processor can invoke information and applications stored in the memory via the transmission device to perform the following steps: determine the target file based on a data write-prohibition request sent by the client, wherein the data write-prohibition request indicates that data should not be written to the target file on the first storage device; change the status of the target file from a first status to a second status, wherein the backend service migrates the target file in the second status and satisfying the data cooling rule from the first storage device to the second storage device, wherein the data cooling rule is a rule that changes the data type from a hot data type to a cold data type, the first status indicates that append writing is supported, the second status indicates that append writing is not supported, the storage cost of the first storage device is higher than that of the second storage device, and the read / write speed of the first storage device is higher than that of the second storage device.

[0146] Optionally, the processor may also execute program code for the following steps: obtaining data cooling rules, wherein the data cooling rules are rules for converting the data type from a hot data type to a cold data type; polling files in the first storage device that are in the second state, and if the files in the second state satisfy the data cooling rules, migrating the files in the second state from the first storage device to the second storage device, wherein the front-end service, upon receiving a request from the client to prohibit writing data to the target file, modifies the state of the target file from the first state to the second state, wherein the first state indicates that append writing is supported, the second state indicates that append writing is not supported, the storage cost of the first storage device is higher than the storage cost of the second storage device, and the read / write speed of the first storage device is higher than the read / write speed of the second storage device.

[0147] Optionally, the processor may also execute program code with the following steps: upon receiving a user-input request to prohibit writing data to the target file, the processor sends the request to the front-end service. Upon receiving the request, the front-end service changes the target file's status from a first state to a second state. The back-end service migrates the target file, which is in the second state and meets the data cooling rules, from the first storage device to the second storage device. The data cooling rules are rules that convert data types from hot data types to cold data types. The first state indicates that appending data is supported, the second state indicates that appending data is not supported, the storage cost of the first storage device is higher than that of the second storage device, and the read / write speed of the first storage device is higher than that of the second storage device.

[0148] Those skilled in the art will understand that, Figure 9 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, PDAs, mobile internet devices (MIDs), tablets, and other terminal devices. This diagram does not limit the structure of the aforementioned electronic devices. For example, electronic device A may include more or fewer components (such as network interfaces, display devices, etc.) than shown in the diagram, or may have a different configuration than shown in the diagram.

[0149] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0150] Example 8

[0151] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0152] Optionally, in this embodiment, the storage medium may be located in any one of the electronic devices in the group of electronic devices in the computer network, or in any one of the mobile terminals in the group of mobile terminals.

[0153] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining data cooling rules, wherein the data cooling rules are rules for converting the data type from a hot data type to a cold data type; modifying the state of a file in the first storage device that is in the second state and satisfies the data cooling rules to the third state, and migrating the file in the third state from the first storage device to the second storage device, wherein the file in the first storage device is converted from the first state to the second state when writing is prohibited, the first state indicates that append writing is supported, the second state indicates that append writing is not supported, the third state indicates that append writing is not supported and real-time reading of data is not supported, and the read / write speed of the first storage device is higher than that of the second storage device.

[0154] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: determining a target file based on a data write-prohibition request sent by the client, wherein the data write-prohibition request indicates that data should not be written to the target file of the first storage device; modifying the state of the target file from a first state to a second state, wherein the target file in the second state in the first storage device is transferred to the second storage device if a data cooling rule is satisfied, wherein the data cooling rule is a rule for changing the data type from a hot data type to a cold data type, the first state indicates that append writing is supported, the second state indicates that append writing is not supported, and the read / write speed of the first storage device is higher than that of the second storage device.

[0155] Example 9

[0156] Embodiments of this application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which can be used to store a computer program that, when executed by a processor, implements the method provided in the above embodiments.

[0157] Example 10

[0158] Embodiments of this application also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0159] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0160] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0164] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0165] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, Applied to backend services, including: Obtain data cooling rules, wherein the data cooling rules are rules that convert the data type from a hot data type to a cold data type; The file in the first storage device that is in the second state and meets the data cooling rule is changed to the third state, and the file in the third state is migrated from the first storage device to the second storage device. The file in the first storage device is changed from the first state to the second state when writing is prohibited. The first state indicates that append writing is supported, the second state indicates that append writing is not supported, and the third state indicates that append writing is not supported and real-time reading is not supported. The read and write speed of the first storage device is higher than that of the second storage device.

2. The method according to claim 1, characterized in that, The method further includes: If a target data retrieval task associated with a target file in the second storage device is polled, a copy of the target file is constructed based on the target data retrieval task, wherein the target data retrieval task is generated based on the data read request of the target file; The copy file is stored in the first storage device, and the status of the target file is changed from the third status to the second status.

3. The method according to claim 1, characterized in that, After obtaining the data cooling rules, the method further includes: When the data cooling rule is a first type of cooling rule, the step of changing the state of the file in the first storage device that is in the second state and satisfies the data cooling rule to the third state is executed, wherein the first type of cooling rule indicates that the file for which data writing is prohibited is cooled; When the data cooling rule is the second type of cooling rule, the state of the file set in the first storage device that conforms to the second type of cooling rule is changed from the first state to the third state, and the file set is migrated from the first storage device to the second storage device. The second type of cooling rule indicates batch cooling of data, and the file set includes at least one file.

4. The method according to any one of claims 1 to 3, characterized in that, The data cooling rules include: Read the configuration data to obtain the data cooling rules, wherein the configuration data includes the data cooling rules, and the data cooling rules include a first type of cooling rule and a second type of cooling rule. The first type of cooling rule indicates that files that have been prohibited from writing data should be cooled, and the second type of cooling rule indicates that data should be cooled in batches.

5. The method according to any one of claims 1 to 3, characterized in that, The data cooling rules include at least one of the following: Cool down files that have pre-defined tags associated with them; Files stored for longer than a first preset duration are cooled. Files whose last access time exceeds a second preset time will be cooled; Files exceeding the preset size will be cooled.

6. A data processing method, characterized in that, Applied to front-end services, including: The target file is determined based on a write-prohibition request sent by the client, wherein the write-prohibition request indicates that data should not be written to the target file on the first storage device; The state of the target file is changed from a first state to a second state. The target file in the second state in the first storage device is migrated to the second storage device if the data cooling rule is satisfied. The data cooling rule is a rule that changes the data type from a hot data type to a cold data type. The first state indicates that append writing is supported, the second state indicates that append writing is not supported, and the read / write speed of the first storage device is higher than that of the second storage device.

7. The method according to claim 6, characterized in that, After changing the state of the target file from the first state to the second state, the method further includes: Upon receiving a data read request from the client, if the data read request indicates reading data from the target file, a target data retrieval task is generated, wherein the target data retrieval task is used to instruct the construction of a copy of the target file and the storage of the copy file in the first storage device; In response to the completion instruction of the target data retrieval task, the system reads data from the copy file from the first storage device and sends the data from the copy file to the client.

8. A data processing system, characterized in that, include: The client is used to send data read / write requests or data write-prohibition requests to the front-end service for the target file, and to receive data returned by the front-end service. The front-end service is configured to perform data read / write operations on the target file if the target file is in a first state when it receives a data read / write request for the target file, and is also configured to change the state of the target file from the first state to a second state when it receives a data write prohibition request for the target file, wherein the first state indicates that appending data is supported, and the second state indicates that appending data is not supported. A backend service is used to change the status of a file that meets the data cooling rule and is in the second state to the third state, and to migrate the file in the third state from the first storage device to the second storage device. The data cooling rule is a rule that changes the data type from a hot data type to a cold data type. The third state indicates that appending data is not supported and real-time data reading is not supported. The first storage device is used to store the data written by the client; The second storage device is used to store data migrated from the first storage device, wherein the read / write speed of the first storage device is higher than that of the second storage device.

9. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images