A data management method, device and electronic equipment

Abstract

Embodiments of the present application provide a data management method and device and electronic equipment, and relate to the technical field of data management, to automatically manage data in a database. In the method, data belonging to a target data group among a plurality of data is determined according to the creation time of each data in the plurality of data; the target data group includes one or more data with a creation time earlier than a target time; a first reduction ratio of target data in the target data group is determined according to the target time; the target data is data in the target data group with a lower access frequency than a first threshold; a regular part of the target data is determined; an irregular part of the target data is compressed; wherein the irregular part includes a part of the target data other than the regular part; and the proportion of the irregular part to the target data meets the first reduction ratio.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, and particularly relates to a data management method and device and electronic equipment. BACKGROUND

[0002] Cloud computing is composed of a series of resources that can be dynamically upgraded and virtualized, which are shared by all users of cloud computing and can be easily accessed through a network. And because users do not need to master the technology of cloud computing when using cloud computing, they can only rent the resources of cloud computing according to the needs of individuals or groups. Therefore, the use of cloud computing can improve the work efficiency of enterprises and make software development work faster and easier.

[0003] With the continuous development of public cloud computing platforms, more and more enterprises are participating in them, which also makes the historical data on the public cloud computing platform increasingly large. At present, the data stored in the network and the data stored in the distributed storage in the public cloud computing platform are stored in the database through indexing. But because the scale of historical data is very large, even if an index is created, a lot of manpower and material resources are still needed to filter the data when querying or modifying the database. At the same time, part of the historical data has little value but occupies a large amount of storage space, resulting in a large amount of storage space being wasted in the public cloud computing platform. How to automatically manage the data in the database is urgent to be solved. SUMMARY

[0004] The embodiments of the present application provide a data management method, device and electronic equipment to automatically manage the data in the database.

[0005] In a first aspect, the embodiments of the present application provide a data management method, comprising: determining data belonging to a target data group from a plurality of data according to the creation time of each data in the plurality of data; the target data group includes one or more data whose creation time is earlier than a target time; determining a first reduction ratio of target data in the target data group according to the target time; the target data is data in the target data group whose access frequency is lower than a first threshold; determining a regular part of the target data; compressing an irregular part of the target data; wherein the irregular part includes a part of the target data other than the regular part; and the proportion of the irregular part to the target data meets the first reduction ratio.

[0006] According to the scheme, the target data in the database is determined according to the creation time and the access frequency of the data, and the obsolete data with low access frequency in the database can be managed. The first reduction ratio of the target data is determined according to the target time, and the unconventional data of the target data is compressed according to the first reduction ratio, so that the more obsolete the data is, the greater the compression ratio is, thereby realizing the automatic management of the data in the database.

[0007] In a possible implementation, the regular part includes at least one of the following: an author of the target data, an abstract of the target data, a title of the target data, or a thumbnail of the target data.

[0008] According to the scheme, when the target data is compressed, the main content of the target data can be known according to the regular part of the data, without the need to find the complete content of the data. The time-consuming and resource waste caused by data decompression can be avoided.

[0009] In a possible implementation, before the determining of the data belonging to the target data group from the plurality of data according to the creation time of each data in the plurality of data, the method further includes: obtaining the creation time of each data in the plurality of data; generating a pyramid time framework according to the creation time of each data in the plurality of data; the pyramid time framework includes a plurality of time ranges; dividing the plurality of data into a plurality of data groups according to the time range to which the creation time belongs through the pyramid time framework; and the plurality of data groups include the target data group.

[0010] According to the scheme, since the upper part of the pyramid time framework can flow to the lower part with the growth of time, the data in the database always maintains the structure of the pyramid, and the data in the database can be conveniently managed.

[0011] In a possible implementation, the determining of the first reduction ratio of the target data in the target data group according to the target time includes: determining a function value of a time decay function according to the target time; the function value of the time decay function satisfies the following formula: wherein f(x) is the function value of the time decay function, x is a time difference between the target time and a current time, and k is a preset decay parameter. The first reduction ratio of each data in the target data group is determined according to the function value of the time decay function.

[0012] According to the scheme, the first reduction ratio is determined according to the time decay function, so that the ratio of the data amount of the target data group after compression to the original data amount is proportional to the function value of the time decay function of the data group. Therefore, the more obsolete the data is, the smaller the data amount is.

[0013] In a possible implementation, the second reduction ratio of the target data is determined according to the access frequency of the target data; the third reduction ratio of the target data is determined according to the first reduction ratio and the second reduction ratio; and the compression of the irregular part of the target data specifically includes that the proportion of the irregular part to the target data meets the third reduction ratio.

[0014] Based on the above scheme, since the first reduction ratio is used to represent the data obsolescence degree and the second reduction ratio is used to represent the data access frequency, the third reduction ratio of the target data is determined according to the first reduction ratio and the second reduction ratio, and the irregular part of the target data is compressed according to the third reduction ratio, so that the reduction ratio of the data is comprehensively determined according to the data obsolescence degree and the access frequency, and the compression of the irregular part of the target data according to the reduction ratio is more accurate.

[0015] In a possible implementation, the access situation of each data in the target data group is obtained by data sampling on each data in the target data group; the access situation includes one or more of the access amount of each data, the keyword matching situation of each data, and the usage rate of each data; and the access frequency of each data in the target data group is determined according to the access situation.

[0016] Based on the above scheme, the access frequency of each data can be determined by data sampling on each data. Moreover, the determined access frequency is more accurate by determining the access frequency of each data according to one or more of the access amount, the keyword matching situation, and the usage rate of the data.

[0017] In a possible implementation, after the compression of the irregular part of the target data, the method further includes: determining the access frequency of the target data by data sampling on the target data; and when the access frequency of the target data is greater than a second threshold, decompressing the irregular part of the target data to restore the target data.

[0018] Based on the above scheme, after the target data is compressed, the irregular part of the target data can be decompressed and the target data can be restored when the access frequency of the target data is higher than the second threshold, so that the data can be protected and data loss can be prevented.

[0019] In a second aspect, an embodiment of the present application provides a data management apparatus, comprising: a processing unit configured to perform the following processing: determining data belonging to a target data group from a plurality of data according to a creation time of each data in the plurality of data; the target data group comprising one or more data with a creation time earlier than a target time; determining a first reduction ratio of target data in the target data group according to the target time; the target data being data in the target data group with an access frequency lower than a first threshold; determining a regular part of the target data; compressing an irregular part of the target data; wherein the irregular part comprises a part of the target data other than the regular part; and the ratio of the irregular part to the target data conforms to the first reduction ratio.

[0020] In a possible implementation, the regular part comprises at least one of the following: an author of the target data, an abstract of the target data, a title of the target data, or a thumbnail of the target data.

[0021] In a possible implementation, before the processing unit determines the data belonging to the target data group from the plurality of data according to the creation time of each data in the plurality of data, the acquisition unit is configured to acquire the creation time of each data in the plurality of data; and the processing unit is configured to generate a pyramid time framework according to the creation time of each data in the plurality of data; the pyramid time framework comprises a plurality of time ranges; and the plurality of data is divided into a plurality of data groups according to the time range to which the creation time belongs through the pyramid time framework; and the plurality of data groups comprise the target data group.

[0022] In a possible implementation, when the processing unit determines the first reduction ratio of the target data in the target data group according to the target time, the processing unit is specifically configured to: determine a function value of a time decay function according to the target time; and the function value of the time decay function satisfies the following formula: wherein f(x) is the function value of the time decay function, x is a time difference between the target time and a current time, and k is a preset decay parameter; and the first reduction ratio of each data in the target data group is determined according to the function value of the time decay function.

[0023] In a possible implementation, the processing unit is further configured to: determine a second reduction ratio of the target data according to an access frequency of the target data; determine a third reduction ratio of the target data according to the first reduction ratio and the second reduction ratio; and when the processing unit compresses the irregular part of the target data, the processing unit is specifically configured to: the ratio of the irregular part to the target data conforms to the third reduction ratio.

[0024] In a possible implementation, the acquisition unit is configured to acquire the access situation of each data in the target data set by data sampling on each data in the target data set; the access situation comprises one or more of the access amount of each data, the keyword matching situation of each data, and the usage rate of each data; and the processing unit is configured to determine the access frequency of each data in the target data set according to the access situation.

[0025] In a possible implementation, after compressing the irregular part of the target data, the processing unit is further configured to: determine the access frequency of the target data by data sampling on the target data; and when the access frequency of the target data is greater than a second threshold, decompress the irregular part of the target data to restore the target data.

[0026] In a third aspect, an embodiment of the present application provides an electronic device, comprising:

[0027] a memory configured to store computer instructions;

[0028] a processor connected with the memory, configured to execute the computer instructions in the memory, and realize the method in any one of the first aspect when executing the computer instructions.

[0029] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, comprising:

[0030] The computer readable storage medium stores computer instructions, and when the computer instructions run on a computer, the computer instructions make the computer execute the method in any one of the first aspect.

[0031] The technical effects of each aspect in the second aspect to the fourth aspect and each aspect that can be achieved are refer to the technical effect description of the first aspect or the various possible schemes in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application.

[0033] Figure 1 A system structure schematic diagram to which the data management method provided by the embodiments of the present application is applicable;

[0034] Figure 2 One of the exemplary flowcharts of the data management method provided by the embodiments of the present application;

[0035] Figure 3AOne of the pyramid time frame schematic diagrams provided by the embodiments of the present application;

[0036] Figure 3B One of the pyramid time frame schematic diagrams provided by the embodiments of the present application;

[0037] Figure 4 The data packet schematic diagram provided by the embodiments of the present application;

[0038] Figure 5 One of the function curve diagrams of the time decay function provided by the embodiments of the present application;

[0039] Figure 6 One of the function curve diagrams of the time decay function provided by the embodiments of the present application;

[0040] Figure 7 The curve diagram of the first deletion ratio provided by the embodiments of the present application;

[0041] Figure 8 The schematic diagram of the corresponding relationship between the function value of the time decay function and the first deletion ratio provided by the embodiments of the present application;

[0042] Figure 9 The schematic diagram of the normal part and the abnormal part of the target data provided by the embodiments of the present application;

[0043] Figure 10 One of the exemplary flowcharts of the data management method provided by the embodiments of the present application;

[0044] Figure 11 One of the exemplary flowcharts of the data management method provided by the embodiments of the present application;

[0045] Figure 12 One of the exemplary flowcharts of the data management method provided by the embodiments of the present application;

[0046] Figure 13 One of the exemplary flowcharts of the data recovery method provided by the embodiments of the present application;

[0047] Figure 14 One of the exemplary flowcharts of the data recovery method provided by the embodiments of the present application;

[0048] Figure 15 One of the exemplary flowcharts of the data recovery method provided by the embodiments of the present application;

[0049] Figure 16 One of the schematic diagrams of the data management device provided by the embodiments of the present application;

[0050] Figure 17 One of the schematic diagrams of the data management device provided by the embodiments of the present application;

[0051] Figure 18 A structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0052] Figure 19 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, the professional terms related to the embodiments of the present application are introduced below.

[0054] (1) Pyramid time framework is a framework method based on the pyramid principle, which starts from a certain time in the past and counts to the latest time, and divides these times into different layers in the shape of a pyramid. With the growth of time, the upper layer of time flows to the lower layer, always maintaining the structure of a pyramid.

[0055] (2) Time decay function refers to a function whose value decreases to zero as time increases.

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the technical solutions of the present application, rather than all the embodiments. Based on the embodiments described in the present application document, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the technical solutions of the present application.

[0057] The terms "first" and "second" in the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprises" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units that are not listed, or can optionally include other steps or units inherent to the process, method, product or device. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0058] In addition, the term "and / or" in this paper is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents an "or" relationship between the associated objects without special explanation.

[0059] Due to the influence of large and various shared devices and large workload change range, the management task in the public cloud computing platform is very complex compared with the traditional system. And the number of machines for managing the public cloud computing platform is too large, and the related staff of the database can hardly intervene manually in all machines comprehensively and completely, so the ease of management is particularly important in the public cloud computing platform. In addition, as the data in the public cloud computing platform is more and more, some historical data has a very low access frequency, but still occupies a large storage space in the database of the public cloud computing platform. Then how to automatically manage the data in the database needs to be solved urgently.

[0060] Therefore, the present application provides a data management method and device and electronic equipment. The method comprises the following steps: dividing the data in the database into a plurality of data groups through a pyramid time framework. For the data belonging to the target data group, the deletion ratio of the data is determined according to the target time. And the data access frequency of each data is determined through the data sampling mechanism. Finally, the irregular part of the target data in the target data group with the access frequency lower than the first threshold is compressed according to the deletion ratio, so as to realize the automatic management of the data in the database.

[0061] Referring to Figure 1 The system structure schematic diagram applicable to the data management method provided by the embodiment of the present application can be a server 100, which comprises a memory 110, a processor 120 and a communication interface 130. The communication interface 130 can be used to obtain the creation time of each data in the database. The processor 120 is the control center of the server 100, which connects each part of the whole server 100 through various interfaces and lines, executes the software program or module stored in the memory 110, and calls the data stored in the memory 110 to execute various functions and process data of the server 100. Optionally, the processor 120 can comprise one or more processing units. The memory 110 can be a high-speed random access memory, and can also be a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device or other volatile solid-state memory device. It should be noted that the structure shown above is only an example, and the embodiment of the present application is not limited thereto. Figure 1 The structure shown above is only an example, and the embodiment of the present application is not limited thereto.

[0062] Referring to Figure 2 One of the exemplary flowcharts of the data management method provided by the embodiment of the present application can be applied to the server 100 as described above. The method can comprise the following flow: Figure 1

[0063] S201, the server determines the data belonging to the target data group from the plurality of data according to the creation time of each data in the plurality of data.

[0064] As​Figure 1 The server 100 shown can determine, by the processor 120, data belonging to a target data group from a plurality of data according to the creation time of each data. The plurality of data can be a plurality of data stored in a database of a public cloud computing platform. The target data group can include one or more data in the database with a creation time earlier than a target time.

[0065] In a possible implementation, before the server determines, from a plurality of data, data belonging to a target data group according to the creation time of each data, the server can obtain, by the communication interface 130, the creation time of each data from the plurality of data. According to the creation time of each data, a plurality of time ranges are determined, and a pyramid time framework is generated. The server can determine the time range to which the creation time of each data belongs, so as to divide the plurality of data in the database into a plurality of data groups according to the time range to which the creation time belongs by the generated pyramid time framework. The plurality of data groups include the target data group.

[0066] In one example, referring to Figure 3A One of the pyramid time frameworks provided by the embodiments of the present application is shown. As shown in Figure 3A In the pyramid time framework shown, the creation time of each data is divided into four time ranges, that is, the pyramid time framework is divided into four layers. The first layer is new data, which can include data with a creation time closest to the current time. For example, it can be data with a creation time within 1 year or data with a creation time within 1 month, etc. The second layer is relatively new data, which can include data with a creation time farther from the current time than the new data but closer to the obsolete data. For example, when the first layer includes data with a creation time within 1 year from the current time, the second layer can include data with a creation time between 1 year and 5 years from the current time. Or when the first layer includes data with a creation time within 1 month from the current time, the second layer can include data with a creation time between 1 month and 6 months from the current time, etc. The third layer is obsolete data, which can include data with a creation time farther from the current time than the relatively new data but closer to the very obsolete data. For example, when the second layer includes data with a creation time between 1 year and 5 years from the current time, the third layer can include data with a creation time between 5 years and 10 years from the current time. When the second layer includes data with a creation time between 1 month and 6 months from the current time, the third layer can include data with a creation time between 6 months and 12 months from the current time, etc. The fourth layer is very obsolete data, which can include data with a creation time farthest from the current time. When the third layer includes data with a creation time between 5 years and 10 years from the current time, the fourth layer can include data with a creation time more than 10 years from the current time. Or when the third layer includes data with a creation time between 6 months and 12 months from the current time, the fourth layer can include data with a creation time more than 12 months from the current time. It should be understood that Figure 3AThe pyramid time frame shown in the figure is exemplary only, and the number of layers of the pyramid time frame and the time ranges included in each layer are not limited in the present application.

[0067] In another example, referring to Figure 3B One of the pyramid time frame diagrams provided by the embodiments of the present application. The pyramid time frame can also include n+1 layers, such as Figure 3B In the pyramid time frame shown, the first layer can include data within 1 year from the current time, the second layer can include data between 1 year and 2 years from the current time,..., the nth layer can include data between (n-1) years and n years from the current time, and the (n+1) layer can include data more than n years from the current time, that is, data more than n years from the current time.

[0068] In one possible case, when the server divides multiple data in the database into multiple data groups through the pyramid time frame, referring to the data grouping diagram shown in Figure 4 Assuming Figure 3A The new data in the first layer refers to data with a creation time within 1 year from the current time. The newer data in the second layer refers to data with a creation time between 1 year and 5 years from the current time. The old data in the third layer refers to data with a creation time between 5 years and 10 years from the current time. The very old data in the fourth layer refers to data with a creation time more than 10 years ago. As shown in Figure 4 The server can divide the multiple data in the database into data group 1, data group 2, data group 3 and data group 4 through the generated pyramid time frame. Among them, data group 1 can include data in the database with a creation time within 1 year from the current time. Data group 2 can include data with a creation time between 1 year and 5 years from the current time. Data group 3 can include data with a creation time between 5 years and 10 years from the current time. Data group 4 can include data with a creation time more than 10 years ago.

[0069] In one example, the target data group can be one of the data groups in Figure 4 For example, the target data group can be data group 4, and the target data group can include data in the database with a creation time more than 10 years ago. Assuming the current time is 2022, the target time can be 2012. That is, the server can determine that the data in the database with a creation time earlier than 2012 is data belonging to the target data group.

[0070] In another example, the target data group can also include multiple data groups in Figure 4 For example, the target data group can include data group 1 and data group 2, and the target data group can include data in the database with a creation time within 1 year from the current time and data with a creation time between 1 year and 5 years from the current time. Assuming the current time is 2022, the target time can be 2021. That is, the server can determine that the data in the database with a creation time earlier than 2021 is data belonging to the target data group. Figure 4The target data group can include data in the database created 1 year ago, assuming the current time is 2022, then the target time can be 2021. That is, the server can determine that the data in the database created before 2021 is data belonging to the target data group.

[0071] At S202, the server determines a first reduction ratio of the target data in the target data group according to the target time.

[0072] Since the data belonging to the same data group can also include data with high access frequency and data with low access frequency, in order to avoid the problem that after the data with high access frequency is compressed, the user cannot find the complete content of the data in the cloud computing platform. The server can compress the target data in the target data group whose access frequency is lower than the first threshold value, and not compress the data whose access frequency is higher than the first threshold value. It should be appreciated that the first threshold value is preset according to actual conditions and experience, which is not limited in the present application.

[0073] In one possible implementation, the server can determine the first reduction ratio of each data group according to the target time of each data group in the database. And the first reduction ratio of the data group is used as the first reduction ratio of each data in the data group. That is, the server can determine the first reduction ratio of the target data group according to the target time of the target data group, and use the first reduction ratio of the target data group as the first reduction ratio of each data in the target data group. Therefore, the first reduction ratio of the target data is also the reduction ratio of the target data group. For example, assuming that the database includes data group 1, data group 2 and data group 3, and the target data group is data group 3. And the server determines the first reduction ratio of data 1 to be 0.1, the first reduction ratio of data group 2 to be 0.5, and the first reduction ratio of data group 3 to be 0.8 according to the target time of each data group in the database. Since the target data group is data group 3, the first reduction ratio of each data in data group 3 is 0.8, and the first reduction ratio of the target data is 0.8.

[0074] In another possible implementation, the server can determine the target data in the target data group whose access frequency is lower than the first threshold value according to the access frequency of each data. The first reduction ratio of the target data is determined according to the target time of the target data group to which the target data belongs.

[0075] In some examples, when determining the first reduction ratio of the target data group, the server can substitute the target time of the target data group into the time decay function to determine the function value of the time decay function, and determine the first reduction ratio of the target data in the target data group according to the function value. Wherein the function value of the time decay function satisfies formula (1):

[0076]

[0077] In the formula, f(x) is a function value of a time decay function, x is a time difference between a target time and a current time, and k is a preset decay parameter.

[0078] It should be understood that the unit of x in formula (1) can be the same as the unit of the time range to which each data group belongs in the pyramid time framework. For example, if the time range of the pyramid time framework is divided by years, then for a plurality of data groups divided according to the time framework, the unit of x in formula (1) is years. Assume that the data groups divided according to the pyramid time framework include data group 1, data group 2, and data group 3. Among them, data group 1 includes data within 1 year, data group 2 includes data between 1 year and 5 years from the current time, and data group 3 includes data more than 5 years from the current time. Then, the function value of the time decay function is determined according to the target time of data group 3, the value of x is 5, and the unit of x is years. If the time range of the pyramid time framework is divided by months, then for a plurality of data groups divided according to the time framework, the unit of x in formula (1) is months. Assume that the data groups divided according to the pyramid time framework include data group 1, data group 2, and data group 3. Among them, data group 1 includes data within 1 month, data group 2 includes data between 1 month and 5 months from the current time, and data group 3 includes data more than 5 months from the current time. Then, the function value of the time decay function is determined according to the target time of data group 3, the value of x is 5, and the unit of x is months.

[0079] Referring to Figure 5 One of the function curve graphs of the time decay function provided in the embodiments of the present application is shown in FIG. 1. Figure 5 In the embodiment, the decay parameter k in formula (1) is taken as 5 for example. Figure 5 In the embodiment, the x-axis represents the time difference between the target time and the current time, and the y-axis represents the function value of the time decay function. As can be seen from Figure 5 It can be seen that, under the condition that the decay parameter is unchanged, the function value of the time decay function gradually decreases to infinity to 0 as the time difference increases. The decay parameter is used to represent the decay speed of the time decay function.

[0080] Referring to Figure 6 One of the function curve graphs of the time decay function provided in the embodiments of the present application is shown in FIG. 1. Figure 6 In the embodiment, the x-axis represents the time difference between the target time and the current time, and the y-axis represents the function value of the time decay function. The dashed line represents the decay parameter of the time decay function is 10, and the solid line represents the decay parameter of the time decay function is 5. As can be seen from Figure 6It can be seen that the greater the decay parameter is, the faster the decay speed of the time decay function is. Therefore, the decay parameter can be set according to the actual situation such as the amount of data in the database, and can be 5, 8 or 10, etc. The present application does not limit this.

[0081] Since the function value of the time decay function gradually decreases as time increases, but for multiple data in the database, the data value of the data farther away from the current time can be lower. Therefore, in order to make the data with lower value in the database have a lower storage ratio, the first reduction ratio of the data farther away from the current time can be higher.

[0082] In one example, determining the first reduction ratio of the target data in the target data group according to the function value of the time decay function can be determined according to formula (2).

[0083]

[0084] In the formula, A is the first reduction ratio, f(x) is the function value of the time decay function, x is the time difference between the target time and the current time, and k is a preset decay parameter.

[0085] For example, when the function value of the time decay function of the target data group is 0.4, the first reduction ratio of the target data can be determined to be 0.6 according to formula (2).

[0086] Referring to Figure 7 The present application provides a curve graph of the first reduction ratio. Figure 7 The x-axis represents the time difference between the target time and the current time, and the y-axis represents the first reduction ratio. Taking 5 as an example of the decay parameter, it can be seen that the first reduction ratio gradually increases with the increase of time, and the first reduction ratio of the data farther away from the current time can be larger. It should be understood that Figure 7 The curve graph of the first reduction ratio shown is only exemplary and does not constitute a limitation on the relationship between the time difference x between the target time and the current time and the first reduction ratio A.

[0087] In another example, the server can determine the first reduction ratio of the target data in the target data group through the pre-stored corresponding relationship between the function value of the time decay function and the first reduction ratio. Referring to Figure 8 The present application provides a corresponding relationship between the function value of the time decay function and the first reduction ratio of the present application, and the corresponding relationship is shown in Figure 8In the correspondence between the function value of the time decay function and the first deletion ratio, the first deletion ratio corresponding to the function value of the time decay function greater than or equal to 0.5 can be 0. The first deletion ratio corresponding to the function value of the time decay function greater than or equal to 0.3 and less than 0.5 can be 0.5. The first deletion ratio corresponding to the function value of the time decay function less than 0.3 can be 0.8. When the function value of the time decay function of the target data group to which the target data belongs is 0.4, since 0.4 is greater than 0.3 and less than 0.5, the first deletion ratio of the target data is determined to be 0.5 through the correspondence between the function value of the time decay function and the first deletion ratio. Figure 8 The correspondence between the function value of the time decay function and the first deletion ratio can determine the first deletion ratio of the target data to be 0.5. It should be understood that, Figure 8 The correspondence between the function value of the time decay function and the first deletion ratio is only exemplary, and the correspondence can be pre-set and stored in the server according to actual conditions or experience, which is not limited in the present application.

[0088] In a possible implementation, when the server determines the target data in the target data group, the server can obtain the access conditions of each data in the target data group by sampling the data. Then, the access conditions of each data can be analyzed to determine the access frequency of each data, so as to determine the target data in the target data group whose access frequency is lower than the first threshold. The access conditions can include one or more of the access amount of each data, the keyword matching condition of each data, and the usage rate of each data.

[0089] Optionally, the server can obtain the access amount of each data in the target data group within a fixed time range, so that the data with high access amount has high access frequency, and the data with low access amount has low access frequency. For example, the server can obtain the access amount of each data in the target data group in the past 5 years. The access amount of data B is 10000 times, and the access amount of data C is 5000 times. Therefore, the access frequency of data B is higher than that of data C.

[0090] Optionally, the server can also obtain the keyword matching condition of each data when the user searches by using a keyword, and the keyword matching condition of each data can be represented by the keyword matching times of each data. Therefore, the data with high keyword matching times has high access frequency, and the data with low keyword matching times has low access frequency. For example, when the user searches by using a keyword, the user searches 10000 times, the keyword matching times of data B with the keyword input by the user is 800. The keyword matching times of data C with the keyword input by the user is 1000. Therefore, the access frequency of data C is higher than that of data B.

[0091] Optionally, the server can also obtain the usage rate of each piece of data. This allows it to determine that data with high usage rates is accessed frequently, and data with low usage rates is accessed less frequently. Usage rate can be determined through methods such as the number of downloads and forwards. For example, if the server finds that data B has been downloaded 1000 times and data C has been downloaded 5000 times in the past year, then the server can determine that data B has a higher usage rate than data C, and therefore accesses data B more frequently than data C.

[0092] In one possible implementation, the server can also determine the access frequency of each piece of data based on two or three of the following factors: access volume, keyword matching rate, and usage rate. For example, if data B has a higher access frequency than data C in any two of the following factors over the past year: access volume, keyword matching frequency, and usage rate, then data B's access frequency is higher than data C's.

[0093] Alternatively, the server can calculate a score and assign a weight to each of the metrics: access volume, keyword match count, and usage rate. Then, the scores can be weighted and summed to determine the access frequency score for each data point, thus identifying data with higher scores as having higher access frequencies.

[0094] It should be understood that the above-described method for determining the access frequency of each piece of data is merely exemplary and is not intended to limit the scope of this application.

[0095] S203, The server determines the regular portion of the target data.

[0096] The general portion of the target data may be the part of the target data that contains important information, and may include at least one of the following: the author of the target data, the abstract of the target data, the title of the target data, or a thumbnail of the target data.

[0097] Since database data types can include documents, images, videos, compressed files, etc., the general information section can also differ depending on the type of target data. For example, if the target data is a document, the general information section can include at least one of the following: document summary, outline, document author, and document title. If the target data is an image, the general information section can be a thumbnail of the image. If the target data is a video, the general information section can be a thumbnail of the video's cover image.

[0098] In a possible implementation, the server can determine the ratio of the data quantity of the regular part to the data quantity of the target data according to the first reduction ratio. For example, if the first reduction ratio is 0.8, the ratio of the data quantity of the regular part to the data quantity of the target data can be determined as 1-0.8=0.2.

[0099] In S204, the server compresses the irregular part of the target data.

[0100] The server can determine the irregular part of the target data according to the regular part determined in S203 and the first reduction ratio determined in S202. The irregular part of the target data is compressed at a high ratio, and the regular part of the target data is reserved, so that the data quantity of the compressed target data is proportional to the function value of the time decay function of the target data group to which the target data belongs. That is, the more obsolete the data is, the smaller the data quantity of the compressed data is. The irregular part includes the part of the target data other than the regular part, and the ratio of the irregular part to the target data meets the first reduction ratio.

[0101] Referring to Figure 9 The compression of the target data provided by the embodiments of the present application is shown in the schematic diagram. As shown in Figure 9 Before the target data is compressed, the target data includes a regular part 901 and an irregular part 902. After the server performs S204, the compressed target data only includes the regular part 901.

[0102] Based on the above scheme, the target data in the database is determined according to the creation time and the access frequency of the data, and the management of the obsolete data with low access frequency in the database can be realized. The first reduction ratio of the target data is determined according to the target time, and the irregular data of the target data is compressed according to the first reduction ratio, so that the more obsolete the data is, the larger the compression ratio is, and thus the automatic management of the data in the database is realized.

[0103] In a possible implementation, the server can further determine a second reduction ratio of the target data according to the access frequency of the target data. Then a third reduction ratio of the target data can be determined according to the first reduction ratio and the second reduction ratio. The regular part of the target data is determined according to the third reduction ratio, so that the irregular part of the target data is compressed, and the management of the target data is completed. The ratio of the irregular part of the target data to the target data conforms to the third reduction ratio. For example, if the first reduction ratio of the target data D is 0.8, and the second reduction ratio is also 0.8, because the first reduction ratio and the second reduction ratio are both too high, the third reduction ratio can be greater than the first reduction ratio and the second reduction ratio, for example, 0.9. If the first reduction ratio of the target data E is 0.8, and the second reduction ratio is 0.5, because the second reduction ratio is very small, it can be determined that the access frequency of the target data is very high, and therefore the third reduction ratio can be less than the first reduction ratio, for example, 0.6. The third reduction ratio can also be less than the second reduction ratio, for example, 0.3.

[0104] Optionally, the server can further pre-store a corresponding relationship between the third reduction ratio and the first reduction ratio and the second reduction ratio, and the third reduction ratio can be determined according to the corresponding relationship. The corresponding relationship can be set according to experience or actual situation, which is not limited in the present application.

[0105] Based on the above scheme, because the first reduction ratio is used to represent the data obsolescence degree, and the second reduction ratio is used to represent the access frequency of the data, the third reduction ratio of the target data is determined according to the first reduction ratio and the second reduction ratio, and the irregular part of the target data is compressed according to the third reduction ratio, so that the reduction ratio of the data can be determined comprehensively in combination with the data obsolescence degree and the access frequency, thereby making the compression of the irregular part of the target data more accurate according to the reduction ratio, and improving the user experience.

[0106] In the following, in order to enable a clearer understanding of the scheme provided by the embodiments of the present application, a data management method provided by the present application will be introduced in combination with specific embodiments.

[0107] Referring to Figures 10-12 An example flowchart of a data management method provided by the embodiments of the present application is shown in FIG. 1. The method can be applied to a server, and can specifically include the following steps:

[0108] S1001, generating a pyramid time framework.

[0109] The method for generating the pyramid time framework can refer to the method embodiment shown in FIG. 1, which will not be described here again. Figure 2

[0110] ​S1002, divide the plurality of data in the database into a plurality of data groups.

[0111] The server can divide the plurality of data in the database into a plurality of data groups through the pyramid time framework generated by S1001. The method of dividing the plurality of data in the database into a plurality of data groups can refer to the method embodiments shown in Figure 2 , which will not be repeated here.

[0112] S1003, determine a first deletion ratio of each data group according to a time decay function.

[0113] The method of determining the first deletion ratio can refer to the method embodiments shown in Figure 2 , which will not be repeated here.

[0114] S1004, obtain first data in the database.

[0115] The first data can be any data in the database.

[0116] S1005, determine whether the first data is data of a target data group.

[0117] If the first data is data of the target data group, execute S1006 as shown in Figure 10 . If the first data is not data of the target data group, execute S1010 as shown in Figure 11 .

[0118] The server can determine the data group to which the first data belongs according to the creation time of the first data, so as to determine whether the first data is data of the target data group. The method of determining whether the first data is data of the target data group can refer to the method embodiments shown in Figure 2 , which will not be repeated here.

[0119] S1006, determine the access frequency of the first data through a data sampling mechanism.

[0120] The method of determining the access frequency of the first data through the data sampling mechanism can refer to the method embodiments shown in Figure 2 , which will not be repeated here.

[0121] S1007, determine whether the access frequency of the first data is lower than a first threshold value.

[0122] When the access frequency of the first data is lower than the first threshold value, execute S1008 as shown in Figure 10 . When the access frequency of the first data is higher than or equal to the first threshold value, execute S1010 as shown in Figure 12 .

[0123] S1008, determining a regular part of the first data according to the first pruning ratio.

[0124] The method for determining the regular part of the first data according to the first pruning ratio can refer to the method embodiment shown in Figure 2 , which will not be described here again.

[0125] S1009, compressing the irregular part of the target data.

[0126] S1010, ending the processing.

[0127] In a possible case, after the irregular part of the target data is compressed by any of the above methods, the server can further determine the access frequency of the target data by data sampling on the target data. When the access frequency of the target data is greater than a second threshold, the compressed irregular part of the target data is decompressed to restore the target data. It should be understood that the second threshold is preset according to actual conditions or experience, which is not limited in the present application.

[0128] In another possible case, since the data in the database of the cloud computing platform is constantly added, the newly added data in the database can have a greater relevance with the compressed data. Therefore, the access frequency of the target data can also be determined according to the relevance of the newly added data in the database with the target data. For example, if the newly added data F in the database and the compressed data B can be retrieved through the same key, it can be determined that the data F has a strong relevance with the data B, and if multiple pieces of newly added data in the database have a strong relevance with the data B, it can be determined that the access frequency of the data B is high.

[0129] Based on the above method, refer to Figures 13-15 , which is one of the exemplary flowcharts of the data recovery method provided by the embodiments of the present application, and specifically includes:

[0130] S1301, adding the newly added data in the database to a data buffer pool.

[0131] In a possible implementation, since the compressed data has the characteristics of data obsolescence and low access frequency, if the relevance of the newly added data in the database with the compressed data is compared every time to determine the access frequency of the compressed data, it will cause long time consumption and waste of resources. Therefore, the newly added data in the database can be added to a preset data buffer pool, and when the data buffer pool is full, the relevance of all data in the data buffer pool with the compressed data is compared to determine the access frequency of the compressed data. It should be understood that the size of the data buffer pool can be preset according to actual conditions or experience, which is not limited in the present application.

[0132] S1302, judging whether the data buffer pool is full.

[0133] If the data buffer pool is full, S1303 is executed as shown in Figure 13 If the data buffer pool is not full, S1307 is executed as shown in Figure 14

[0134] S1303, the data in the data buffer pool is compared with the compressed data for relevance.

[0135] The server can determine the relevance of the data in the data buffer pool and the compressed data according to whether the data in the data buffer pool and the compressed data can be retrieved by the same key.

[0136] S1304, the access frequency of the compressed data is determined.

[0137] The server can determine the access frequency of each compressed data according to the relevance of the data in the data buffer pool and the compressed data determined in S1303.

[0138] S1305, it is determined whether the access frequency is greater than a second threshold value.

[0139] It is determined whether there is data in the compressed data whose access frequency is greater than the second threshold value. If there is data whose access frequency is greater than the second threshold value, S1306 is executed as shown in Figure 13 If there is no data whose access frequency is greater than the second threshold value, S1307 is executed as shown in Figure 15

[0140] S1306, the data whose access frequency is greater than the second threshold value is decompressed.

[0141] The storage location of the compressed irregular part of the data whose access frequency is greater than the second threshold value in the database is determined, the compressed irregular part is decompressed, and the decompressed data is restored to the original data format and placed in the cloud computing platform again.

[0142] S1307, the processing is ended.

[0143] Based on the same concept of the above method, see Figure 16 ​​A data management apparatus 1600 is provided for the embodiments of the present application. The data management apparatus 1600 is capable of performing each step in the above method, and thus will not be described in detail here. The data management apparatus 1600 can include a pyramid time frame generation module 1601, a time decay function module 1602, a data sampling module 1603, and a new data processing module 1604. The pyramid time frame generation module 1601 is configured to generate a pyramid time frame, and divide a plurality of data in a database into a plurality of data groups according to the pyramid time frame. The time decay function module 1602 is configured to calculate a function value of a time decay function of each data group to determine a first deletion ratio of each data group. The data sampling module 1603 is configured to determine an access frequency of each target in a target data group by a data sampling mechanism to determine target data in the target data group. The new data processing module 1604 is configured to, when new data is added to the database, associate the new data with compressed data to determine an access frequency of the compressed data, and decompress data with an access frequency greater than a second threshold.

[0144] Based on the same concept of the above method, see Figure 17 A data management apparatus 1700 is provided for the embodiments of the present application. The apparatus 1700 is capable of performing each step in the above method, and thus will not be described in detail here. The apparatus 1700 includes an acquisition unit 1701 and a processing unit 1702. In one scenario:

[0145] The processing unit 1702 is configured to perform the following processing: determining, according to a creation time of each data in a plurality of data, data belonging to a target data group in the plurality of data; the target data group including one or more data with a creation time earlier than a target time; determining a first deletion ratio of target data in the target data group according to the target time; the target data being data in the target data group with an access frequency lower than a first threshold; determining a regular part of the target data; compressing an irregular part of the target data; wherein the irregular part includes a part of the target data other than the regular part; and the proportion of the irregular part to the target data conforming to the first deletion ratio.

[0146] In one possible implementation, the regular part includes at least one of the following: an author of the target data, an abstract of the target data, a title of the target data, or a thumbnail of the target data.

[0147] In a possible implementation, before the processing unit 1702 determines the data in the target data group according to the creation time of each data in the plurality of data, the acquisition unit 1701 is configured to acquire the creation time of each data in the plurality of data; and the processing unit 1702 is further configured to generate a pyramid time framework according to the creation time of each data in the plurality of data; the pyramid time framework includes a plurality of time ranges; and the plurality of data are divided into a plurality of data groups according to the time range to which the creation time belongs by using the pyramid time framework; and the plurality of data groups include the target data group.

[0148] In a possible implementation, when the processing unit 1702 determines the first reduction ratio of the target data in the target data group according to the target time, the processing unit 1702 is specifically configured to determine the function value of a time decay function according to the target time; and the function value of the time decay function satisfies the following formula: wherein f(x) is the function value of the time decay function, x is the time difference between the target time and the current time, and k is a preset decay parameter; and the first reduction ratio of each data in the target data group is determined according to the function value of the time decay function.

[0149] In a possible implementation, the processing unit 1702 is further configured to determine the second reduction ratio of the target data according to the access frequency of the target data; and determine the third reduction ratio of the target data according to the first reduction ratio and the second reduction ratio. When the processing unit 1702 compresses the irregular part of the target data, the processing unit 1702 is specifically configured to: the proportion of the irregular part to the target data meets the third reduction ratio.

[0150] In a possible implementation, the processing unit 1702 is further configured to acquire the access situation of each data in the target data group by sampling the data of each data in the target data group; the access situation includes one or more of the access amount of the data, the keyword matching situation of the data, and the usage rate of the data; and the access frequency of each data in the target data group is determined according to the access situation.

[0151] In a possible implementation, after the processing unit 1702 compresses the irregular part of the target data, the processing unit 1702 is further configured to: determine the access frequency of the target data by sampling the data of the target data; and when the access frequency of the target data is greater than a second threshold, decompress the irregular part of the target data to restore the target data.

[0152] Based on the same concept of the above method, refer to Figure 18An electronic device structure schematic diagram provided by an embodiment of the present application is shown in FIG. 1. The electronic device includes a processor 1801 and a memory 1802. The memory 1802 is configured to store computer instructions. The processor 1801 is connected with the memory and is configured to execute the computer instructions in the memory, and implement the steps of any of the above methods when executing the computer instructions.

[0153] In a possible implementation, referring to Figure 19 An electronic device structure schematic diagram provided by an embodiment of the present application is shown in FIG. 1. The electronic device includes a processor 1801 and a memory 1802. The memory 1802 is configured to store computer instructions. The processor 1801 is connected with the memory and is configured to execute the computer instructions in the memory, and implement the steps of any of the above methods when executing the computer instructions. Figure 19 The electronic device 1900 is merely an example and should not impose any limitation on the functions and application scope of the embodiments of the present application. As shown in Figure 19 The components of electronic device 1900 may, for example, include one or more processors 1901, one or more storage media 1902, and a bus 1903 that connects the storage media 1902 to the processor(s) 1901. The processor(s) 1901 are configured to execute instructions and to process data stored in the storage media 1902. The storage media 1902 can include one or more non-transitory storage media, such as random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or Flash memory), or any other type of storage media. The storage media 1902 can also include a program or programs that implement any of the above methods, and / or any other desired program or programs. The bus 1903 can include any type of bus structure, including a memory bus or memory controller, a peripheral bus, a local bus, or any other suitable bus structure.

[0154] The electronic device 1900 can also communicate with one or more external devices 1904, such as a keyboard or a pointing device, using the I / O interface 1905. The I / O interface 1905 can also interface to various input / output devices, such as display, USB device, a modem, a network interface card, speakers, a camera, a microphone, a sensor, or other input / output devices. The I / O interface 1905 can also be used to communicate with one or more devices that enable a user to interact with the electronic device 1900, such as a keyboard, a pointing device, a microphone, a sensor, or other input devices. The electronic device 1900 can also include a communication interface 1906 that can be used to communicate with one or more other electronic devices. Figure 19As shown, the network adapter 1906 communicates with other modules of the electronic device 1900 over the bus 1903. It should be understood that although not shown, other hardware or software modules could be used in connection with the electronic device 1900. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0155] The embodiments of the present application further provide a computer readable storage medium, having computer instructions stored thereon, when the computer instructions are run on a computer, the computer is caused to execute the steps of any of the above methods.

[0156] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes: ROM, RAM, magnetic disk or optical disk and various storage media that can store program codes.

[0157] Although the specific embodiments of the present application are described above, those skilled in the art should understand that these are only illustrative, and the protection scope of the present application is defined by the appended claims. Those skilled in the art can make various changes or modifications to the embodiments without departing from the principles and essence of the present application, and these changes and modifications all fall within the protection scope of the present application. Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to the embodiments once they know the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0158] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A data management method characterized by, The method comprises: determining data belonging to a target data group from a plurality of data according to creation time of each data in the plurality of data; one or more data in the target data group have creation time earlier than a target time; determining a function value of a time decay function according to the target time; the function value of the time decay function satisfies the following formula: wherein, is a function value of the time decay function, is a time difference between the target time and the current time, k is a preset decay parameter; determining a first reduction ratio of each data in the target data group according to the function value of the time decay function; the target data is data in the target data group with access frequency lower than a first threshold; determining a regular part of the target data; wherein the regular part comprises at least one of the following: author of the target data, abstract of the target data, title of the target data or thumbnail of the target data; compressing an irregular part of the target data; wherein the irregular part comprises a part of the target data other than the regular part; the proportion of the irregular part to the target data meets the first reduction ratio.

2. The method of claim 1, wherein, Before the determining data belonging to a target data group from a plurality of data according to creation time of each data in the plurality of data, the method further comprises: obtaining creation time of each data in the plurality of data; generating a pyramid time framework according to the creation time of each data in the plurality of data; the pyramid time framework comprises a plurality of time ranges; dividing the plurality of data into a plurality of data groups according to the time range to which the creation time belongs through the pyramid time framework; the plurality of data groups comprise the target data group.

3. The method of claim 1, wherein, The method further comprises: determining a second reduction ratio of the target data according to access frequency of the target data; determining a third reduction ratio of the target data according to the first reduction ratio and the second reduction ratio; The method further comprises: The proportion of the irregular part to the target data meets the third reduction ratio.

4. The method of claim 1, wherein, The method further comprises: obtaining access situation of each data in the target data group by data sampling on each data in the target data group; the access situation comprises one or more of the following: access amount of the each data, keyword matching situation of the each data and usage rate of the each data; determining access frequency of each data in the target data group according to the access situation.

5. The method according to any of claims 1 to 4, characterized in that, After the compressing an irregular part of the target data, the method further comprises: determining access frequency of the target data by data sampling on the target data; when the access frequency of the target data is greater than a second threshold, decompressing the irregular part of the target data to restore the target data.

6. A data management apparatus characterized by comprising: The method comprises: a processing unit configured to determine data belonging to a target data group from a plurality of data according to creation time of each data in the plurality of data; one or more data in the target data group have creation time earlier than a target time; a function value of a time decay function is determined according to the target time; the function value of the time decay function satisfies the following formula: wherein, is a function value of the time decay function, is a time difference between the target time and the current time, k is a preset decay parameter; a first reduction ratio of each data in the target data group is determined according to the function value of the time decay function; the target data is data in the target data group whose access frequency is lower than a first threshold; a regular part of the target data is determined; an irregular part of the target data is compressed; wherein the irregular part includes a part of the target data other than the regular part; a ratio of the irregular part to the target data conforms to the first reduction ratio; wherein, the regular part includes at least one of the following: an author of the target data, an abstract of the target data, a title of the target data, or a thumbnail of the target data.

7. An electronic device, comprising: The method comprises: a memory configured to store computer instructions; A processor, connected with the memory, for executing computer instructions in the memory, and realizing the method as claimed in any one of claims 1 to 5 when executing the computer instructions.

8. A computer-readable storage medium, characterized in that, Comprising: The computer readable storage medium stores computer instructions, which, when run on a computer, cause the computer to execute the method as claimed in any one of claims 1 to 5.

Images