Data processing method, device and equipment

Abstract

The embodiment of the present application provides a data processing method, device and equipment, the method comprising: obtaining the data size of cached data in the memory; according to the data size, determining the dump mode for dumping the cached data, the dump mode comprising any one of the following: a first dump mode corresponding to the online erasure code technology, a second dump mode corresponding to the non-online erasure code technology; performing a dump operation on the cached data based on the dump mode. The technical solution provided by the present application effectively realizes that in different application scenarios, different dump modes can be selected to dump cached data, thereby effectively solving the problems of write traffic amplification, storage space amplification or consumption of an extra copy of read traffic when using a multi-copy strategy or erasure code technology for data processing operations in the prior art, and the method does not limit the size of the data written by the user.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device and equipment. Background Art

[0002] In order to ensure data security, distributed storage systems usually use multi-copy strategies or erasure code (EC) technology for data redundancy. When using a multi-copy strategy for data processing, there are problems of write traffic amplification and storage space amplification; when using erasure code technology for data processing, the problems of write traffic amplification and storage space amplification can be significantly reduced. However, since erasure code technology is usually used for background offline dumping, an extra read traffic will be consumed during the dump operation. Summary of the invention

[0003] The embodiments of the present application provide a data processing method, apparatus and device to solve the problems in the prior art of amplifying write traffic, amplifying storage space or consuming an extra copy of read traffic when using a multi-copy strategy or erasure coding technology for data redundancy operations.

[0004] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0005] Get the size of cached data in memory;

[0006] Determine, according to the data amount, a dump mode for performing a dump operation on the cached data, wherein the dump mode includes any one of the following: a first dump mode corresponding to an online erasure coding technology, and a second dump mode corresponding to a non-online erasure coding technology;

[0007] A dump operation is performed on the cache data based on the dump mode.

[0008] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0009] The acquisition module is used to obtain the data size of the cached data in the memory;

[0010] A determination module, configured to determine a dump mode for performing a dump operation on the cache data according to the amount of data, wherein the dump mode includes any one of the following: a first dump mode corresponding to an online erasure coding technology, and a second dump mode corresponding to a non-online erasure coding technology;

[0011] A processing module is used to perform a dump operation on the cache data based on the dump mode.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the data processing method shown in the first aspect above.

[0013] In a fourth aspect, an embodiment of the present invention provides a computer storage medium for storing a computer program, wherein the computer program enables a computer to implement the data processing method shown in the first aspect when executed.

[0014] The technical solution provided by the embodiment of the present application effectively implements that in different application scenarios, different dump methods can be selected to perform dump operations on cached data, thereby effectively solving the problems of write traffic amplification, storage space amplification, or consumption of an extra copy of read traffic when using a multi-copy strategy or erasure code technology for data processing operations in the prior art. In addition, the method does not limit the size of the data written by the user, thereby improving the scope of application of the method and the flexibility and reliability of use, so that the method can be adapted to a variety of application scenarios, further improving the practicality of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A schematic diagram of a scenario of a data processing method provided in an embodiment of the present application;

[0017] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0018] Figure 3 A flow chart of determining that the dump mode is a second dump mode corresponding to the non-online erasure coding technology provided in an embodiment of the present application;

[0019] Figure 4 A schematic diagram of a process for performing a dump operation on the cache data based on the dump method provided in an embodiment of the present application Figure 1 ;

[0020] Figure 5 A schematic diagram of a process for performing a dump operation on the cache data based on the dump method provided in an embodiment of the present application Figure 2 ;

[0021] Figure 6 A schematic diagram of a process for performing a dump operation on the cache data based on the dump method provided in an embodiment of the present application Figure 3 ;

[0022] Figure 7 A schematic diagram of a process for adjusting the cache data provided in an embodiment of the present application;

[0023] Figure 8 A flowchart of a data processing method provided for an application embodiment of the present application;

[0024] Fig. 9 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0025] Fig.10 for Fig. 9 A schematic structural diagram of an electronic device corresponding to the data processing device shown. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms of "a", "said", and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two, but does not exclude the inclusion of at least one.

[0028] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0029] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0030] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a product or system. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the product or system including the elements.

[0031] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0032] In order to facilitate those skilled in the art to understand the technical solutions provided by the embodiments of the present application, the relevant technologies are described below:

[0033] In order to ensure the security of data storage, distributed storage systems can usually adopt a multi-copy strategy or an erasure code (EC) strategy for data redundancy. Among them, when using a multi-copy strategy for data processing operations, there is a situation of write traffic amplification and storage space amplification; when using EC technology for data processing, one implementation method is: using background offline dump to generate EC files, which can significantly reduce the problems of write traffic amplification and storage space amplification and reduce data storage costs. However, the data written by users still needs to be written into multiple copy files, and there are still many problems of write traffic amplification, so an additional copy of read traffic will be required for background offline dump.

[0034] Another way to use erasure code technology is online dumping. However, this requires that the size and flow of the user's input / output data (IO) are large, so that enough data can be written into the EC file. Obviously, this method is not friendly to application scenarios with small IO and small flow. Even in some application scenarios, users can upload multiple small data volumes or multiple data with small flows. According to the processing principle of EC files, the data to be processed must meet a certain length. If the data to be processed cannot be aligned with the required length of the EC file, it is necessary to perform data padding on the data to be processed. In this way, when reading the data, there will still be a problem of traffic waste; in addition, when the data after the above data padding operation is performed online dumping operation, the EC file generated by the dump includes less valid data, which makes the EC file waste more space. For example, for an EC file, there may be 20%, 25% or 30% of the space wasted, which reduces the quality and efficiency of the dumping operation.

[0035] In summary, when using multi-copy file (Log file) technology for data storage operations, data alignment is not required, but there are problems of traffic amplification and storage space amplification; when using erasure coding technology to perform offline data dump operations, the problems of write traffic amplification and storage space amplification can be significantly reduced, reducing storage costs, but there will be an additional read traffic. When using erasure coding technology to perform online data dump operations, the memory data can be directly read and dumped to the EC file, which reduces the read traffic. However, due to the memory limitations of the distributed storage system, the memory data needs to be dumped to the EC file as soon as possible. Combined with the requirements of erasure coding technology, it is necessary to ensure that the write data is aligned, and the misaligned data will be filled in. In this way, when the data length is relatively small, more invalid data will be filled, which will cause space waste.

[0036] In order to solve the above technical problems, this embodiment provides a data processing method, device and equipment, such as Figure 1 As shown, the execution subject of the data processing method can be a data processing device, and the data processing device can be communicatively connected with a client.

[0037] Among them, the client can be any computing device with a certain computing power. The basic structure of the client can include: at least one processor. The number of processors depends on the configuration and type of the client. The client can also include a memory, which can be volatile, such as RAM, or non-volatile, such as read-only memory (ROM), flash memory, etc., or can also include both types. The memory usually stores an operating system (OS), one or more application programs, and can also store program data, etc. In addition to the processing unit and the memory, the client also includes some basic configurations, such as a network card chip, an IO bus, a display component, and some peripheral devices, etc. Optionally, some peripheral devices may include, for example, a keyboard, a mouse, an input pen, a printer, etc. Other peripheral devices are well known in the art and will not be described in detail here. Optionally, the client can be a PC (personal computer) terminal, a handheld terminal (e.g., a smart phone, a tablet computer), etc.

[0038] A data processing device refers to a device that can provide computing and processing services in a network virtual environment, and usually refers to a device that uses the network for information planning and data processing. In terms of physical implementation, a data processing device can be any device that can provide computing services, respond to service requests, and perform processing, such as a conventional server, cloud server, cloud host, virtual center, etc. The composition of a data processing device mainly includes a processor, hard disk, memory, system bus, etc., which is similar to a general computer architecture.

[0039] In the above embodiment, the client can establish a network connection with the data processing device, and the network connection can be a wireless or wired network connection. If the client and the data processing device are connected in communication, the network standard of the mobile network can be any one of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), WiMax, etc.

[0040] In an embodiment of the present application, the client is used to obtain data uploaded by a user and send the uploaded data to a data processing device so that the data processing device can analyze and process the uploaded data. The number of uploaded data may be one or more. When the number of uploaded data is multiple, the multiple data may correspond to different data formats and the multiple data may correspond to different data sizes.

[0041] A data processing device is used to receive data uploaded by a client, and then cache the uploaded data in a memory, and then obtain the data volume of the cached data in the memory; and determine a dump method for dumping the cached data according to the data volume; and then the cached data can be dumped based on the dump method.

[0042] Specifically, the dumping method may include at least one of the following: a first dumping method corresponding to the online erasure code technology, a second dumping method corresponding to the non-online erasure code technology, when the data size S is equal to or equal to the preset threshold value Sth, the dumping method is determined as the first dumping method, and then the cache data can be dumped based on the first dumping method; when the data size S is less than the preset threshold value Sth, the dumping method is determined as the second dumping method, and then the cache data can be dumped based on the second dumping method. This effectively realizes that in different application scenarios, different dumping methods can be selected to perform dump operations on cache data, thereby solving the problem of write traffic amplification, storage space amplification, or consumption of one more read traffic when using multiple copy strategies or erasure code technology for data redundancy operations in the prior art, and the method does not limit the size of the data written by the user, thereby improving the scope of application of the method and the flexibility and reliability of use, so that the method can be adapted to a variety of application scenarios.

[0043] The data processing methods provided in various embodiments of the present application are described in detail below through an exemplary application scenario.

[0044] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application; see attached Figure 2 As shown, this embodiment provides a data processing method, and the execution subject of the method may be a data processing device. It can be understood that the data processing device can be implemented as software, or a combination of software and hardware. Specifically, the data processing method may include:

[0045] Step S201: Obtain the size of cached data in the memory.

[0046] Step S202: Determine a dump mode for dumping cached data according to the amount of data, wherein the dump mode includes any one of the following: a first dump mode corresponding to online erasure coding technology, and a second dump mode corresponding to non-online erasure coding technology.

[0047] Step S203: performing a dump operation on the cache data based on the dump method.

[0048] The following is a detailed description of each of the above steps:

[0049] Step S201: Obtain the size of cached data in the memory.

[0050] Among them, the memory is one of the important components of the computer, which is used to temporarily store the calculation data in the central processing unit. Specifically, when the user writes data to the data processing device, the written data can be cached in the memory first, so that the cache data can be generated. It can be understood that as the data processing time continues to increase, or as the user continuously writes data, the amount of cached data in the memory will change accordingly. For example, if at least part of the cached data in the memory has been dumped, the amount of cached data in the memory will decrease; if the user continuously writes data, the amount of cached data in the memory will increase.

[0051] In addition, this embodiment does not limit the specific implementation method for obtaining the data size of the cached data in the memory. Technical personnel in this field can make arbitrary settings according to the specific application scenario, for example: counting the occupancy of the memory, and calculating the data size of the cached data in the memory based on the occupancy; or, using a pre-trained machine learning model to analyze and process the cached data in the memory to determine the data size of the cached data in the memory.

[0052] In some instances, obtaining the data size of cached data in the memory may include: obtaining cached data written into the memory by the user end in a write-ahead log manner; and analyzing and processing the cached data to obtain the data size of the cached data in the memory.

[0053] Specifically, write-ahead logging (WAL) is a technology used to provide atomicity and persistence in relational database systems. When a user writes data to the memory in the write-ahead logging mode, the cache data written to the memory by the user in the write-ahead logging mode can be obtained. It can be understood that all cache data in the memory can correspond to one or more users; after obtaining the cache data, the cache data can be analyzed and processed to obtain the amount of cache data in the memory.

[0054] It should be noted that in this embodiment, after obtaining the data written by the user in the pre-write log manner, when counting the amount of cached data in the memory, the cached data written by more users can be integrated or synthesized as much as possible, which is conducive to making the amount of cached data counted meet the conditions for dumping using online erasure coding technology.

[0055] Step S202: Determine a dump mode for dumping cached data according to the amount of data, wherein the dump mode includes any one of the following: a first dump mode corresponding to online erasure coding technology, and a second dump mode corresponding to non-online erasure coding technology.

[0056] Since the size of the cached data in the memory is associated with the implementation method for performing a dump operation on the cached data, after obtaining the size of the data, the size of the data can be analyzed and processed to determine the dump method for performing a dump operation on the cached data according to the analysis and processing results. In some instances, determining the dump method for performing a dump operation on the cached data according to the size of the data may include: when the size of the data is greater than a preset threshold, determining that the dump method is a first dump method corresponding to the online erasure coding technology; when the size of the data is less than or equal to the preset threshold, determining that the dump method is a second dump method corresponding to the non-online erasure coding technology.

[0057] Specifically, a preset threshold value for analyzing and processing the data volume is pre-set. It is conceivable that the specific numerical range or numerical value of the preset threshold value can be adjusted based on different application scenarios or application requirements. After obtaining the data volume, the data volume can be analyzed and compared with the preset threshold value. When the data volume is greater than the preset threshold value, it means that the data volume of the cached data in the memory is large at this time, and then it can be determined that the data length of the above-mentioned cached data can meet the data alignment requirements corresponding to the online erasure code technology. Therefore, it can be determined that the dump mode is the first dump mode corresponding to the online erasure code technology.

[0058] When the data size is less than or equal to the preset threshold, it means that the amount of cached data in the memory is small at this time, and it can be determined that the data length of the above-mentioned cached data cannot meet the data alignment requirements corresponding to the online erasure code technology. Therefore, it can be determined that the dump method is the second dump method corresponding to the non-online erasure code technology.

[0059] In some instances, the second dump mode corresponding to the non-online erasure coding technology includes at least one of the following: a first sub-dump mode corresponding to the online multiple-copy technology; an empty dump mode for identifying that cached data is kept stored in memory; and a second sub-dump mode corresponding to the offline erasure coding technology.

[0060] Step S203: performing a dump operation on the cache data based on the dump method.

[0061] After determining the dump mode for dumping the cache data, the determined dump mode can be used to dump the cache data. It is understandable that when the determined dump modes are different, dump files of different formats corresponding to the cache data can be obtained.

[0062] For example, when the dump mode is determined to be the first dump mode, the first dump mode can be used to perform an online dump operation on the cached data, and at this time, a first EC file corresponding to the cached data can be generated. When the dump mode is determined to be the first sub-dump mode corresponding to the online multiple-copy technology, the first sub-dump mode can be used to perform an online dump operation on the cached data, and at this time, a first Log file corresponding to the cached data can be generated; when the dump mode is determined to be the empty dump mode used to identify that the cached data is kept stored in the memory, no dump operation can be performed on the cached data; when the dump mode is determined to be the second sub-dump mode corresponding to the offline erasure code technology, the second sub-dump mode can be used to perform an offline dump operation on the cached data, and at this time, a second EC file corresponding to the cached data can be generated.

[0063] It should be noted that the size of the corresponding cached data in the memory may be different in different time periods or at different times, so the dumping methods for the above-mentioned cached data of different data sizes may be the same or different. For example, at time t1, the size of the cached data obtained in the memory is greater than the preset threshold, so the method used to dump the cached data is determined to be the online erasure coding technology; at time t2, the size of the cached data obtained in the memory is less than or equal to the preset threshold, so the method used to dump the cached data is determined to be the online multi-copy technology; at time t3, the size of the cached data obtained in the memory is less than or equal to the preset threshold, so the method used to dump the cached data is determined to be the offline erasure coding technology, etc.

[0064] The data processing method provided in this embodiment effectively realizes that in different application scenarios, different dump methods can be selected to perform dump operations on cached data, thereby effectively solving the problems of write traffic amplification, storage space amplification or consumption of an extra copy of read traffic when using multiple copy strategies or erasure coding technology for data processing operations in the prior art. In addition, the method does not limit the size of data written by users, thereby improving the scope of application of the method and the flexibility and reliability of use, so that the method can be adapted to a variety of application scenarios, further improving the practicality of the method.

[0065] Figure 3A flowchart of determining that the dump mode provided in the embodiment of the present application is a second dump mode corresponding to the non-online erasure coding technology; based on the above embodiment, refer to the attached Figure 3 As shown, since the second dump mode corresponds to multiple specific implementation modes (first sub-dump mode, empty dump mode, second dump mode), in order to ensure the accuracy and reliability of determining the second dump mode, this embodiment provides an implementation mode of determining the dump mode as the second dump mode corresponding to the non-online erasure code technology. Specifically, determining the dump mode as the second dump mode corresponding to the non-online erasure code technology in this embodiment may include:

[0066] Step S301: obtaining the time interval between the last dump operation and the last dump operation;

[0067] Step S302: when the time interval is greater than or equal to the preset time threshold, determining that the second dump mode is any one of the first sub-dump mode, the empty dump mode and the second sub-dump mode;

[0068] Step S303: when the time interval is less than the preset time threshold, it is determined that the second dump mode is the empty dump mode.

[0069] Among them, for the cached data in the memory, in order to improve the quality and efficiency of the dump operation on the cached data, the cached data in the memory can be dumped at a preset frequency instead of dumping the cached data in real time. For example: the cached data in the memory can be dumped at the following frequencies: 30s / time, 60s / time, 90s / time, etc., that is, the cached data in the memory can be dumped every 30s, 60s or 90s, so that the cached data in the memory can meet the alignment requirements corresponding to the online erasure coding method as much as possible, thereby realizing that most of the cached data in the memory can be dumped using the online erasure coding technology, which is beneficial to improving the quality and efficiency of data dump, and can also reduce the problem of extra read traffic required for offline dumping.

[0070] Since the cached data in the memory can be dumped at a preset frequency, the timer can be used to obtain the timestamp information of the cached data dump operation, that is, the last timestamp information corresponding to the last dump operation on the cached data can be obtained. After obtaining the last timestamp information, the time interval between the current moment and the last timestamp information can be determined.

[0071] After obtaining the above time interval, the above time interval and the preset time interval can be analyzed and processed, wherein the above preset time interval corresponds to the configured frequency for dumping the cached data in the memory, for example: when the configured dump operation frequency is 60s / time, the preset time interval can be 60s; when the configured dump operation frequency is 90s / time, the preset time interval can be 90s. It should be noted that the larger the preset time interval, that is, the lower the frequency of dumping the cached data in the memory, the data size corresponding to the cached data for the dump operation can more easily meet the data alignment requirements corresponding to the online erasure code technology.

[0072] When the comparison result between the above time interval and the preset time interval is that the time interval is greater than or equal to the preset time threshold, it means that the time for performing the dump operation on the cached data in the memory meets the preset frequency requirement, and it can be determined that the second dump mode at this time can be any one of the first sub-dump mode, the empty dump mode and the second sub-dump mode; preferably, it can be determined that the second dump mode is the first sub-dump mode. Correspondingly, when the comparison result between the above time interval and the preset time interval is that the time interval is less than the preset time threshold, it means that the time for performing the dump operation on the cached data in the memory does not meet the preset frequency requirement, and it can be determined that the second dump mode at this time can be the empty dump mode, that is, no dump operation is performed on the cached data in the memory.

[0073] Application scenario 1: When the user continuously writes four data segments (data segment a, data segment b, data segment c, and data segment d) to the memory, the amount of data written in the first three segments (data segment a, data segment b, data segment c) is relatively large, and the online erasure coding technology can be used to perform a dump operation on the first three segments of data, thereby generating EC files corresponding to data segment a, data segment b, and data segment c.

[0074] For the fourth data segment (data segment d) being written, the data volume is relatively small, and the time interval with the last dump operation can be counted. When the time interval is relatively short, no dump operation is required for data segment d; when the time interval is relatively long, the data segment d can be dumped using multiple copy technology, thereby generating log files d1-log file dn corresponding to data segment d.

[0075] Application scenario two: When the user continuously writes four data segments (data segment a, data segment b, data segment c and data segment d) to the memory, the amount of data written in the first three segments (data segment a, data segment b, data segment c) is relatively large, and then the online erasure coding technology can be used to perform dump operations on the first three segments of data respectively, so as to generate EC file a corresponding to data segment a, EC file b corresponding to data segment b, and EC file c corresponding to data segment c.

[0076] As for the fourth data segment (data segment d) being written, the data volume thereof is relatively small. No matter whether the time interval corresponding to the last dump operation is relatively long or not, no dump operation may be performed on the data segment d.

[0077] Application scenario three: When the user continuously writes four data segments (data segment a, data segment b, data segment c and data segment d) to the memory, the amount of data written in the first three segments (data segment a, data segment b, data segment c) is relatively large, and the online erasure coding technology can be used to perform a dump operation on the first three segments of data, thereby generating EC file a corresponding to data segment a, EC file b corresponding to data segment b, and EC file c corresponding to data segment c.

[0078] For the fourth data segment (data segment d) being written, the data volume is relatively small, and the time interval with the last dump operation can be counted. When the time interval is relatively short, no dump operation is required for data segment d; when the time interval is relatively long, the data segment d can be dumped offline using erasure coding technology, thereby generating an EC file d corresponding to the data segment d.

[0079] In this embodiment, by obtaining the time interval between the last dump operation and then analyzing and comparing the time interval with the preset time threshold, it is effectively achieved that different second dump modes can be selected at different time intervals to perform a dump operation on the cached data. Specifically, when the time interval is greater than or equal to the preset time threshold, the second dump mode is determined to be any one of the first sub-dump mode, the empty dump mode and the second sub-dump mode; when the time interval is less than the preset time threshold, the second dump mode is determined to be the empty dump mode. This not only ensures the quality and effect of the data dump operation, but also improves the flexibility and reliability of the method.

[0080] Figure 4 A schematic diagram of a process for performing a dump operation on cache data based on a dump method provided in an embodiment of the present application Figure 1 Based on the above embodiments, continue to refer to the attached Figure 4As shown, after determining that the second dump mode is the first sub-dump mode; this embodiment provides an implementation method for performing a dump operation on cache data based on a dump mode. Specifically, the dump operation on cache data based on a dump mode in this embodiment may include:

[0081] Step S401: performing a dump operation on cache data based on a first sub-dump mode to obtain multiple copy files corresponding to the cache data.

[0082] Step S402: Use offline erasure coding technology to perform a dump operation on the data in the multi-copy file to obtain an offline erasure coding file corresponding to the multi-copy file.

[0083] After determining that the second dump mode is the first sub-dump mode, the first sub-dump mode can be used to perform a dump operation on the cache data in the memory, thereby obtaining multiple copy files corresponding to the cache data.

[0084] Since multi-copy files occupy a larger space than erasure coded files, in order to reduce space occupancy, after obtaining the multi-copy files, you can also use offline erasure coding technology to dump the data in the multi-copy files, so that you can get the offline erasure coded files corresponding to the multi-copy files, that is, you can realize the offline dumping of multi-copy files into erasure coded files in the background.

[0085] It should be noted that when converting multiple copy files into offline erasure code files, an additional bit of read traffic is also wasted, that is, the offline traffic required for reading multiple copy files offline; however, for all cached data in the memory, only a small part of the cached data is dumped using multiple copy technology, and most of the cached data can be dumped using online erasure code technology. Therefore, for all cached data included in the memory, the read traffic required for data dump operations is still reduced.

[0086] In this embodiment, after determining that the second dump mode is the first sub-dump mode, a dump operation can be performed on the cached data based on the first sub-dump mode to obtain multiple copy files corresponding to the cached data, and then the offline erasure code technology is used to perform a dump operation on the data in the multiple copy files to obtain offline erasure code files corresponding to the multiple copy files, thereby effectively avoiding the situation where a large amount of space is occupied due to multiple copy files, so that the space occupied by the dump file corresponding to the cached data is relatively reduced, further improving the stability and reliability of the use of this method.

[0087] Figure 5 A schematic diagram of a process for performing a dump operation on cache data based on a dump method provided in an embodiment of the present application Figure 2 Based on the above embodiments, continue to refer to the attached Figure 5 As shown, after determining that the second dump mode is the second sub-dump mode; this embodiment provides another implementation method for performing a dump operation on cache data based on the dump mode. Specifically, the dump operation on cache data based on the dump mode in this embodiment may include:

[0088] Step S501: performing a dump operation on cached data based on a second sub-dump mode to obtain a first erasure code file corresponding to the cached data.

[0089] Step S502: Determine valid data corresponding to the first erasure code file.

[0090] Step S503: using offline erasure coding technology to perform a dump operation on valid data to obtain a second erasure code file.

[0091] After determining that the second dump mode is the second sub-dump mode, the second sub-dump mode can be used to perform a dump operation on the cached data, so that a first erasure code file corresponding to the cached data can be obtained. It should be noted that since the amount of cached data dumped using the second sub-dump mode is small, in order to meet the data alignment requirements corresponding to the erasure code technology, it is necessary to perform a padding operation on the cached data.

[0092] For example, the data alignment requirement corresponding to the erasure code technology is "the data volume size is 8", and the corresponding data volume size in the cache data is 6. At this time, in order to enable the erasure code technology to dump the cache data, it is necessary to perform a data padding operation on the cache data, that is, it is necessary to pad the cache data by 2, and the padded data is invalid data relative to the cache data. Then, the erasure code technology can be used to dump the cache data after the data padding operation, so as to generate a first erasure code file. At this time, the first erasure code file includes: valid data corresponding to the cache data, and invalid data corresponding to the data filling operation.

[0093] Since the first erasure code file includes: valid data corresponding to the cached data, and invalid data corresponding to the data filling operation, and in order to avoid the first erasure code file occupying a large space due to the presence of invalid data in the first erasure code file, after obtaining the first erasure code file, the first erasure code file can be analyzed and processed to determine the valid data corresponding to the first erasure code file.

[0094] Specifically, a standard data format is pre-set for performing data completion operations on cached data, and the data format corresponding to all data in the first erasure code file is identified. When the data format matches the standard data format, it can be determined that the data corresponding to the above data format is invalid data. When the data format does not match the standard data format, it can be determined that the data corresponding to the above data format is valid data.

[0095] After determining the valid data corresponding to the first erasure code file, the offline erasure code technology can be used to dump the valid data to obtain a second erasure code file. It can be understood that the data included in the second erasure code file corresponds to the cached data and does not include any invalid data, thereby effectively avoiding the problem of occupying a large space when performing storage operations on the second erasure code file.

[0096] In this embodiment, a dump operation is performed on cached data based on a second sub-dump method to obtain a first erasure code file corresponding to the cached data, and valid data corresponding to the first erasure code file is determined, and then an offline erasure code technology is used to perform a dump operation on the valid data to obtain a second erasure code file, thereby effectively avoiding the problem of occupying a large space when performing storage operations on the second erasure code file, and further improving the practicality of the method.

[0097] Figure 6 A schematic diagram of a process for performing a dump operation on cache data based on a dump method provided in an embodiment of the present application Figure 3 Based on the above embodiments, refer to the attached Figure 6 As shown, after determining that the dump mode is the first dump mode; the dump operation on the cache data based on the dump mode in this embodiment may include:

[0098] Step S601: Acquire data dump conditions corresponding to the first dump mode.

[0099] Among them, a data dump condition corresponding to the first dump mode is pre-set, and the data dump condition is used to limit the data length requirement corresponding to the first dump mode. The data dump condition can be: the amount of data corresponding to the first dump mode to be used is an integer multiple of the preset data amount. It can be understood that the amount of data corresponding to the first dump mode to be used is an integer multiple of the preset data amount, which can include: the amount of data corresponding to the first dump mode to be used is the same as the preset data amount, or the amount of data corresponding to the first dump mode to be used is a preset integer number of the preset data amount, and the preset integer number is greater than or equal to 2. Specifically, the pre-configured data dump condition can be stored in a preset area, and the data dump condition corresponding to the first dump mode can be obtained by accessing the preset area.

[0100] Step S602: Detect whether the cached data meets the data dump condition.

[0101] After obtaining the data dump conditions, the cached data and the data dump conditions may be analyzed and processed to detect whether the cached data satisfies the data dump conditions. In some instances, detecting whether the cached data satisfies the data dump conditions may include: obtaining a first data length of the cached data and a data alignment length corresponding to a first dump mode; and based on the data alignment length and the first data length, detecting whether the cached data satisfies the data dump conditions.

[0102] Specifically, based on the data alignment length and the first data length, detecting whether the cached data meets the data dump condition may include: when the first data length is an integer multiple of the data alignment length (including: the first data length is the same as the data alignment length, the first data length is at least 2 times the data alignment length), determining that the cached data meets the data dump condition; or, when the first data length is a non-integer multiple of the data alignment length (including: the first data length is different from the data alignment length, and the first data length is a non-integer multiple of the data alignment length), determining that the cached data does not meet the data dump condition.

[0103] For example, the cached data includes data A, data B and data C, and the data lengths corresponding to the above-mentioned cached data are data length a, data length b and data length c. After obtaining the data lengths, the above-mentioned data lengths can be analyzed and compared with the length threshold T corresponding to the data dump condition. When the analysis and comparison results are: data length a is the same as length threshold T (that is, data length c is 1 times of length threshold T), data length b is less than length threshold T, and data length c is 3 times of length threshold T, it can be determined that data A and data C meet the data dump condition, while data B does not meet the data dump condition.

[0104] Step S603: When the cache data meets the data dump condition, a dump operation is performed on the cache data.

[0105] Step S604: when the cache data does not meet the data dump condition, the cache data is adjusted, and a dump operation is performed on the adjusted cache data.

[0106] When the cache data meets the data dump condition, the cache data can be dumped using the first dump method, so that an EC file corresponding to the cache data can be generated. When the cache data does not meet the data dump condition, in order to be able to perform a timely dump operation on the cache data, the cache data can be adjusted so that the adjusted cache data meets the data dump condition, and then the adjusted cache data can be dumped.

[0107] In this embodiment, by obtaining the data dump condition corresponding to the first dump mode and detecting whether the cache data meets the data dump condition, when the cache data meets the data dump condition, the cache data is dumped; when the cache data does not meet the data dump condition, the cache data is adjusted and the adjusted cache data is dumped, thereby effectively achieving the stability and reliability of the dump operation on the cache data, and further improving the quality and efficiency of data analysis and processing.

[0108] Figure 7 A schematic diagram of a flow chart for adjusting cache data provided in an embodiment of the present application; based on the above embodiment, refer to the attached Figure 7 As shown, this embodiment provides an implementation method for adjusting cache data. Specifically, the adjustment of cache data in this embodiment may include:

[0109] Step S701: based on a data dump condition, obtaining at least one adjustment data corresponding to the cache data in the memory.

[0110] After the data dump condition is obtained, at least one adjustment data corresponding to the cached data may be obtained in the memory based on the data dump condition. In some examples, obtaining at least one adjustment data corresponding to the cached data in the memory based on the data dump condition may include: determining an adjustment data length for adjusting the cached data based on a data alignment length corresponding to the data dump condition and a first data length of the cached data; and obtaining at least one adjustment data corresponding to the cached data in the memory based on the adjustment data length.

[0111] Specifically, in different application scenarios, the data dump condition may correspond to different data alignment lengths. After determining the data dump condition, the data alignment length corresponding to the data dump condition may be determined. After obtaining the data alignment length corresponding to the data dump condition and the first data length of the cached data, the data alignment length and the first data length may be analyzed and processed to determine an adjusted data length for adjusting the cached data. In some instances, based on the data alignment length corresponding to the data dump condition and the first data length of the cached data, determining the adjusted data length for adjusting the cached data may include: determining the difference between the data alignment length and the first data length as the adjusted data length.

[0112] After the adjustment data length is obtained, the adjustment data length can be analyzed and processed to obtain at least one adjustment data corresponding to the cache data in the memory, thereby effectively ensuring the accuracy and reliability of obtaining the at least one adjustment data.

[0113] For example, the data alignment length corresponding to the data dump condition is 16. After the cache data is obtained, the cache data can be analyzed and processed to determine the first data length corresponding to the cache data. When the first data length is 10, the adjusted data length for adjusting the cache data can be determined based on the data alignment length and the first data length of the cache data. The adjusted data length is 16-10=6.

[0114] After the adjustment data length is obtained, at least one adjustment data corresponding to the cache data can be obtained in the memory. At this time, the sum of the data lengths corresponding to the at least one adjustment data is the adjustment data length 6.

[0115] Step S702: merge at least one adjusted data and cached data to obtain adjusted cached data, where the adjusted cached data meets the data dump condition.

[0116] After acquiring at least one adjustment data, the at least one adjustment data and the cache data may be merged to obtain adjusted cache data. It is understandable that the adjusted cache data may meet the data dump condition.

[0117] In some other examples, before the cache data in the memory is dumped, a compression operation may be performed on the cache data, and then the compressed cache data may be dumped.

[0118] For example, when the data alignment length corresponding to the data dump condition is 1024kB, if the first data length of the cache data that needs to be dumped is 600kB, then when the adjusted data length between the data alignment length and the first data length is 424kB, if the data compression rate is 50%, an adjusted data with a data size of 424kB / 50%=848kB can be extracted from the memory, that is, the size of the adjusted data stored in the memory without compression processing is 848kB.

[0119] After obtaining the adjustment data, the adjustment data can be compressed and the compressed adjustment data can be merged with the cache data to obtain the adjusted cache data. The data size of the adjusted cache data can meet the data dump conditions, and the adjusted cache data can be dumped.

[0120] In this embodiment, based on the data dump condition, at least one adjustment data corresponding to the cache data is obtained in the memory, and then the at least one adjustment data and the cache data are merged to obtain the adjusted cache data, thereby effectively adjusting the cache data so that the adjusted cache data meets the data dump condition, and then the first dump method can be used to perform a dump operation on the adjusted cache data, further ensuring the quality and efficiency of data processing.

[0121] For specific applications, refer to the attached Figure 8 As shown, taking the first dump mode as the online erasure code mode and the second dump mode as the online multiple copies mode as an example, this application embodiment provides a data processing method, which aims to solve the problem of space waste caused by the need to align data when applying the erasure code technology in the prior art, so that the method can be adapted to various scenarios. Specifically, the method may include the following steps:

[0122] Step 1: Write the data written by the user to the cache file in the form of a write-ahead log (WAL), cache the data in the memory to form cache data, and finally send feedback information to the user to inform the user of the result of the data writing operation.

[0123] It can be understood that when the data written by the user is successfully cached in the memory, a first feedback information can be sent to the user, and the first feedback information is used to notify the user that the data is successfully written; when the data written by the user is not successfully cached in the memory, a second feedback information can be sent to the user, and the second feedback information is used to notify the user that the data is not successfully written.

[0124] In this application embodiment, by writing data in the WAL mode, it is possible to effectively avoid the large data delay or failure of the data dump operation during the data dump operation, and the result of the data write operation can be fed back to the user in a timely manner, thereby ensuring the friendliness of the interactive operation with the user.

[0125] Step 2: Start a dump task and obtain the size of cached data in the memory based on the dump task.

[0126] Step 3: Analyze and compare the data size with the threshold. When the data size is greater than the threshold, use online erasure coding technology to dump the cached data to generate the first EC file.

[0127] Among them, when the cache data is dumped using online erasure coding technology, specifically, the cache data can be encoded with erasure coding, so that an EC file can be generated. In some instances, in order to ensure the quality and efficiency of the dump operation, it is possible to detect whether the cache data meets the dump conditions corresponding to the erasure coding technology. When the cache data meets the dump conditions, the cache data can be directly dumped; when the cache data does not meet the dump conditions, the cache data can be flexibly adjusted so that the adjusted data can meet the dump conditions, and then the adjusted data can be dumped.

[0128] Specifically, when performing flexible adjustment operations on cache data, the adjustment data used to adjust the cache data can be extracted in the memory, and then the adjustment data can be merged with the cache data, so that the data volume after the merge operation can be an integer multiple of the data alignment length corresponding to the EC file.

[0129] Step 4: When the data size is less than or equal to the threshold, the cache data is dumped using the online multi-copy technology to generate a log file.

[0130] When the cache data is dumped using the multi-copy technology, multiple dump files may be generated. Typically, the multi-copy technology may be 3-copy files, and three log files corresponding to the cache data may be generated.

[0131] In other examples, when the data size is less than or equal to the threshold, the time interval between the last dump operation and the last dump operation can be further obtained. When the time interval is greater than or equal to the time threshold, the cache data is dumped using the online multi-copy technology to generate a log file. When the time interval is less than the time threshold, no dump operation may be performed on the cache data, that is, the cache data can be kept stored in the memory.

[0132] Step 5: After obtaining the log file, you can dump the log file offline as an EC file.

[0133] Among them, since the storage space required by the Log file is larger than the storage space required by the EC file, in order to reduce the storage space occupied by the dump file, after the Log file is obtained, the Log file can be converted into an EC file offline.

[0134] The data processing method provided by the present application embodiment effectively realizes that when the amount of cached data in the memory is small, the cached data can be written into a Log file, and the Log file can be dumped as an EC file in the background offline. In other instances, the cached data can also be cached in the memory all the time, or can also be written into a new EC file separately and recycled by the background; when the amount of cached data is large, it means that the cached data at this time can meet the dumping requirements of the erasure code technology, and then the above cached data can be dumped as an EC file, so that most of the cached data in the memory can be dumped using the online erasure code technology, thereby reducing the waste of space.

[0135] It should be noted that when the cached data does not meet the dumping conditions of the erasure code technology, the cached data can be flexibly adjusted so that the adjusted cached data can meet the dumping conditions corresponding to the erasure code technology. This effectively solves the problem of an extra read traffic when using the background offline dump of EC files in the prior art. It also solves the problems of space waste and inflexible application scenarios caused by direct writing of user data to EC files in the prior art, and effectively mitigates the problems of write traffic amplification and storage space amplification, further improving the scope of application scenarios to which the method is applicable and effectively ensuring the practicability of the method.

[0136] Fig. 9 A schematic diagram of a data processing device provided in an embodiment of the present application; see Attachment Fig. 9 As shown, this embodiment provides a data processing device, which can execute the above Figure 2 Specifically, the data processing device may include: an acquisition module 11 , a determination module 12 and a processing module 13 .

[0137] An acquisition module 11 is used to acquire the amount of cached data in the memory;

[0138] A determination module 12 is used to determine a dump mode for performing a dump operation on cached data according to the amount of data, wherein the dump mode includes any one of the following: a first dump mode corresponding to an online erasure coding technology, and a second dump mode corresponding to a non-online erasure coding technology;

[0139] The processing module 13 is used to perform a dump operation on the cache data based on the dump mode.

[0140] In some instances, when the determination module 12 determines the dump mode for performing a dump operation on the cached data based on the data size, the determination module 12 can be used to execute: when the data size is greater than a preset threshold, the dump mode is determined to be a first dump mode corresponding to the online erasure code technology; when the data size is less than or equal to the preset threshold, the dump mode is determined to be a second dump mode corresponding to the non-online erasure code technology.

[0141] In some instances, the second dump mode corresponding to the non-online erasure coding technology includes at least one of the following: a first sub-dump mode corresponding to the online multiple-copy technology; an empty dump mode for identifying that cached data is kept stored in memory; and a second sub-dump mode corresponding to the offline erasure coding technology.

[0142] In some instances, when the determination module 12 determines that the dump mode is the second dump mode corresponding to the non-online erasure code technology, the determination module 12 can be used to execute: obtaining the time interval between the last dump operation; when the time interval is greater than or equal to a preset time threshold, determining that the second dump mode is any one of the first sub-dump mode, the empty dump mode, and the second sub-dump mode; when the time interval is less than the preset time threshold, determining that the second dump mode is the empty dump mode.

[0143] In some instances, after determining that the second dump mode is the first sub-dump mode; when the processing module 13 performs a dump operation on the cached data based on the dump mode, the processing module 13 can be used to execute: performing a dump operation on the cached data based on the first sub-dump mode to obtain multiple copy files corresponding to the cached data; using offline erasure code technology to perform a dump operation on the data in the multiple copy files to obtain offline erasure code files corresponding to the multiple copy files.

[0144] In some instances, after determining that the second dump mode is the second sub-dump mode; when the processing module 13 performs a dump operation on the cached data based on the dump mode, the processing module 13 can be used to execute: performing a dump operation on the cached data based on the second sub-dump mode to obtain a first erasure code file corresponding to the cached data; determining valid data corresponding to the first erasure code file; and using offline erasure code technology to perform a dump operation on the valid data to obtain a second erasure code file.

[0145] In some instances, after determining that the dump mode is the first dump mode; when the processing module 13 performs a dump operation on the cache data based on the dump mode, the processing module 13 can be used to execute: obtaining the data dump condition corresponding to the first dump mode; detecting whether the cache data meets the data dump condition; when the cache data meets the data dump condition, performing a dump operation on the cache data; when the cache data does not meet the data dump condition, adjusting the cache data, and performing a dump operation on the adjusted cache data.

[0146] In some instances, when the processing module 13 detects whether the cached data meets the data dump condition, the processing module 13 can be used to execute: obtaining a first data length of the cached data and a data alignment length corresponding to the first dump mode; based on the data alignment length and the first data length, detecting whether the cached data meets the data dump condition.

[0147] In some instances, when the processing module 13 detects whether the cached data meets the data dump condition based on the data alignment length and the first data length, the processing module 13 can be used to execute: when the first data length is the same as the data alignment length, or the first data length is an integer multiple of the data alignment length, determining that the cached data meets the data dump condition; or, when the first data length is different from the data alignment length and the first data length is a non-integer multiple of the data alignment length, determining that the cached data does not meet the data dump condition.

[0148] In some instances, when the processing module 13 adjusts the cache data, the processing module 13 can be used to execute: based on the data dump condition, obtaining at least one adjustment data corresponding to the cache data in the memory; merging the at least one adjustment data and the cache data to obtain adjusted cache data, and the adjusted cache data meets the data dump condition.

[0149] In some instances, when the processing module 13 obtains at least one adjustment data corresponding to the cached data in the memory based on the data dump condition, the processing module 13 can be used to execute: determining the adjustment data length for adjusting the cached data based on the data alignment length corresponding to the data dump condition and the first data length of the cached data; and obtaining at least one adjustment data corresponding to the cached data in the memory based on the adjustment data length.

[0150] In some instances, when the processing module 13 determines an adjustment data length for adjusting cache data based on the data alignment length corresponding to the data dump condition and the first data length of the cache data, the processing module 13 can be used to execute: determining the difference between the data alignment length and the first data length as the adjustment data length.

[0151] In some instances, when the acquisition module 11 acquires the size of cached data in the memory, the acquisition module 11 is used to execute: acquiring the cached data written into the memory by the user end in a write-ahead log manner; analyzing and processing the cached data to obtain the size of the cached data in the memory.

[0152] Fig. 9 The device shown can perform Figure 1-Figure 8 For the method of the embodiment shown in the figure, the part not described in detail in this embodiment can be referred to Figure 1-Figure 8 The implementation process and technical effects of this technical solution refer to Figure 1-Figure 8 The description in the illustrated embodiment will not be repeated here.

[0153] In one possible design, Fig. 9 The structure of the data processing device shown can be implemented as an electronic device, which can be a mobile phone, a tablet computer, a server, or other devices. Fig.10 As shown, the electronic device may include: a first processor 21 and a first memory 22. The first memory 22 is used to store the corresponding electronic device to execute the above Figure 1-Figure 8 The program of the data processing method provided in the illustrated embodiment, the first processor 21 is configured to execute the program stored in the first memory 22 .

[0154] The program includes one or more computer instructions, wherein when the one or more computer instructions are executed by the first processor 21, the following steps can be implemented:

[0155] Get the size of cached data in memory;

[0156] Determine, according to the amount of data, a dump mode for performing a dump operation on the cached data, the dump mode including any one of the following: a first dump mode corresponding to an online erasure coding technology, a second dump mode corresponding to a non-online erasure coding technology;

[0157] Dump the cache data based on the dump mode.

[0158] Furthermore, the first processor 21 is also used to execute the aforementioned Figure 1-Figure 8 All or part of the steps in the illustrated embodiments.

[0159] The structure of the electronic device may further include a first communication interface 23 for the electronic device to communicate with other devices or a communication network.

[0160] In addition, an embodiment of the present invention provides a computer storage medium for storing computer software instructions used by electronic devices, which includes instructions for executing the above Figure 1-Figure 7 The program involved in the data processing method in the method embodiment shown.

[0161] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0162] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0163] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0165] These computer program instructions may also be loaded onto a computer or other programmable device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0166] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0167] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0168] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, comprising: Get the size of cached data in memory; According to the data volume, determine a dump mode for performing a dump operation on the cached data, wherein the dump mode includes any one of the following: a first dump mode corresponding to an online erasure coding technology, a second dump mode corresponding to a non-online erasure coding technology, wherein the second dump mode corresponding to the non-online erasure coding technology includes at least one of the following: a first sub-dump mode corresponding to an online multiple-copy technology; an empty dump mode for identifying that the cached data is kept stored in a memory; a second sub-dump mode corresponding to an offline erasure coding technology; Performing a dump operation on the cache data based on the dump mode; When the dump mode is the second dump mode, determining that the dump mode is the second dump mode corresponding to the non-online erasure coding technology includes: The time interval between the acquisition and the last dump operation; When the time interval is greater than or equal to a preset time threshold, determining that the second dump mode is any one of the first sub-dump mode, the empty dump mode and the second sub-dump mode; When the time interval is less than a preset time threshold, it is determined that the second dump mode is the empty dump mode.

2. The method according to claim 1, determining a dump mode for performing a dump operation on the cache data according to the data amount, comprising: When the data volume is greater than a preset threshold, determining that the dump mode is a first dump mode corresponding to the online erasure coding technology; When the data volume is less than or equal to a preset threshold, it is determined that the dump mode is a second dump mode corresponding to the non-online erasure coding technology.

3. The method according to claim 1, after determining that the second dump mode is the first sub-dump mode; Performing a dump operation on the cache data based on the dump mode includes: Performing a dump operation on the cache data based on the first sub-dump mode to obtain multiple copy files corresponding to the cache data; The data in the multiple-copy file is dumped using offline erasure coding technology to obtain an offline erasure coding file corresponding to the multiple-copy file.

4. The method according to claim 1, after determining that the second dump mode is the second sub-dump mode; Performing a dump operation on the cache data based on the dump mode includes: Performing a dump operation on the cache data based on the second sub-dump mode to obtain a first erasure code file corresponding to the cache data; Determining valid data corresponding to the first erasure code file; The valid data is dumped using an offline erasure coding technology to obtain a second erasure coding file.

5. The method according to claim 2, after determining that the dump mode is the first dump mode; Performing a dump operation on the cache data based on the dump mode includes: Acquire a data dump condition corresponding to the first dump mode; Detecting whether the cached data meets the data dump condition; When the cache data meets the data dump condition, a dump operation is performed on the cache data; When the cache data does not satisfy the data dump condition, the cache data is adjusted, and a dump operation is performed on the adjusted cache data.

6. The method according to claim 5, detecting whether the cache data meets the data dump condition comprises: Acquire a first data length of the cache data and a data alignment length corresponding to the first dump mode; Based on the data alignment length and the first data length, it is detected whether the cache data meets a data dump condition.

7. The method according to claim 6, detecting whether the cache data meets the data dump condition based on the data alignment length and the first data length, comprises: When the first data length is an integer multiple of the data alignment length, determining that the cache data meets the data dump condition; or, When the first data length is a non-integer multiple of the data alignment length, it is determined that the cache data does not satisfy the data dump condition.

8. The method according to claim 5, adjusting the cache data comprises: Based on the data dump condition, acquiring at least one adjustment data corresponding to the cache data in the memory; The at least one adjusted data and the cache data are merged to obtain adjusted cache data, where the adjusted cache data meets the data dump condition.

9. The method according to claim 8, based on the data dump condition, acquiring at least one adjustment data corresponding to the cache data in the memory, comprising: Determining an adjustment data length for adjusting the cache data based on the data alignment length corresponding to the data dump condition and the first data length of the cache data; At least one adjustment data corresponding to the cache data is acquired in the memory based on the adjustment data length.

10. The method according to claim 8, determining an adjustment data length for adjusting the cache data based on the data alignment length corresponding to the data dump condition and the first data length of the cache data, comprising: The difference between the data alignment length and the first data length is determined as the adjusted data length.

11. The method according to any one of claims 1 to 10, wherein obtaining the amount of cached data in the memory comprises: Get the cache data written into the memory by the user end in the form of write-ahead log; The cache data is analyzed and processed to obtain the data volume of the cache data in the memory.

12. A data processing device, comprising: The acquisition module is used to obtain the data size of the cached data in the memory; A determination module, used to determine a dump mode for performing a dump operation on the cache data according to the data amount, wherein the dump mode includes any one of the following: a first dump mode corresponding to an online erasure coding technology, a second dump mode corresponding to a non-online erasure coding technology, wherein the second dump mode corresponding to the non-online erasure coding technology includes at least one of the following: a first sub-dump mode corresponding to an online multiple-copy technology; an empty dump mode for identifying that the cache data is kept stored in a memory; a second sub-dump mode corresponding to an offline erasure coding technology; A processing module, used for performing a dump operation on the cache data based on the dump mode; The determination module is also used to determine that the dump mode is the second dump mode corresponding to the non-online erasure code technology when the dump mode is the second dump mode, including: obtaining the time interval between the last dump operation; when the time interval is greater than or equal to a preset time threshold, determining that the second dump mode is any one of the first sub-dump mode, the empty dump mode and the second sub-dump mode; when the time interval is less than the preset time threshold, determining that the second dump mode is the empty dump mode.

13. An electronic device comprising: A memory, a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the data processing method as described in any one of claims 1 to 11.

Images