Data storage method, electronic device, and computer-readable storage medium

Abstract

The present application discloses a method for data storage, an electronic device, and a computer-readable storage medium, the method comprising: clustering a plurality of data to be stored by blood relationship to obtain a plurality of data sets; searching for a first data module in the plurality of data sets, and adding all the first data modules to a first target data family; generating a first identifier corresponding to each first data module in the first target data family; storing the correspondence between the first data module and the first identifier and a plurality of data to be stored, wherein when storing a plurality of data to be stored, in response to the data to be stored including the first data module, the first identifier corresponding to the first data module is used to replace the first data module in the data to be stored. The method of the present application can reduce the storage space of the data to be stored.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data storage method, electronic device and computer-readable storage medium. Background Art

[0002] In enterprises that provide multi-variety, small-batch, and personalized customization services, there will be a large amount of product data with high similarity. The main data content is consistent, but some data is quite different. In order to meet the needs of efficient and clear data storage, such data requires a large amount of storage space. As product types and customer business needs continue to expand, storage requirements are also increasing. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a data storage method, an electronic device and a computer-readable storage medium, which can reduce the storage space of the data to be stored.

[0004] To solve the above technical problems, the first aspect of the present application provides a method for data storage, the method comprising: clustering multiple data to be stored based on blood relationships to obtain multiple data sets, wherein the data to be stored includes at least one data unit, and the multiple data to be stored in the data set include main data and at least one customized data corresponding to the main data, and the customized data corresponding to the main data is generated based on the main data; searching for a first data module in the multiple data sets, and adding all the first data modules to a first target data family, wherein the first data module appears repeatedly in the multiple data sets, and the first data module includes at least one data unit, and the data units in the first data module exist in the same data to be stored at the same time; generating a first identifier corresponding to each first data module in the first target data family; storing the correspondence between the first data module and the first identifier and the multiple data to be stored, wherein, when storing the multiple data to be stored, in response to the data to be stored including the first data module, the first data module in the data to be stored is replaced with the first identifier corresponding to the first data module.

[0005] To solve the above technical problems, the second aspect of the present application provides an electronic device, which includes: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method described in the first aspect.

[0006] In order to solve the above technical problem, the third aspect of the present application provides a computer-readable storage medium on which program data is stored. When the program data is executed by a processor, the method described in the first aspect is implemented.

[0007] Different from the prior art, the present application first clusters the data to be stored according to blood relationship to obtain multiple data sets, and then searches for the first data module in the multiple data sets, which helps to improve the efficiency of searching in different data sets; and further generates a first identifier corresponding to the first data module found, stores the correspondence between the first data module and the first identifier, and finally replaces the first data module with the first identifier when storing the data to be stored. Through the above process, it can be seen that the present application replaces the repeated first data module with the first identifier, which can release the space originally used to store the first data module, and only needs to provide a smaller storage space to store the first identifier, thereby reducing the storage space of all data to be stored. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0009] Figure 1 It is a flowchart of an implementation method of the data storage method of the present application;

[0010] Figure 2 yes Figure 1 A schematic flow chart of an implementation method of step S200;

[0011] Figure 3 yes Figure 1 A schematic diagram of a flow chart of an implementation method before step S400;

[0012] Figure 4 yes Figure 1 A schematic diagram of another embodiment of the process before step S400;

[0013] Figure 5 yes Figure 1 A schematic diagram of a flow chart of another implementation method before step S400;

[0014] Figure 6 yes Figure 1 A schematic diagram of a flow chart of an implementation method after step S400;

[0015] Figure 7 yes Figure 6 A schematic flow chart of an implementation method of step S700;

[0016] Figure 8 It is a structural schematic diagram of an implementation method of a storage device of the present application;

[0017] Fig. 9 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0018] Fig.10 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different implementation methods can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0021] See also Figure 1 The present application provides a data storage method, the method comprising:

[0022] S100: Clustering multiple data to be stored based on blood relationship to obtain multiple data sets, wherein the data to be stored includes at least one data unit, and the multiple data to be stored in the data set include main data and at least one customized data corresponding to the main data, and the customized data corresponding to the main data is generated based on the main data.

[0023] Specifically, before clustering, multiple data to be stored include multiple master data and multiple customized data. The customized data are all generated from a certain master data, and the customized data generated from the master data has a blood relationship with the master data, and has no blood relationship with other master data. One master data can generate at least one customized data, that is, one customized data has a blood relationship with only one master data, and one master data can have a blood relationship with multiple customized data. The present application clusters multiple data to be stored based on blood relationship, and aggregates the master data and customized data with blood relationship into one data set, that is, there is no blood relationship between the data of the multiple data sets after clustering, and the blood relationship only exists between the master data and customized data in the same set, and a data set only includes one master data. At the same time, both the master data and the customized data include at least one data unit, and the data unit refers to the smallest unit obtained by deconstructing the data to be stored.

[0024] In one embodiment, a plurality of data to be stored are clustered according to blood relationship to obtain N data sets, which are respectively recorded as data set D1, data set D2, ..., D n , ..., data set D N , where n∈N. The data set D n Master data included and M customized data, where the M customized data are respectively recorded as customized data , Customized data , ..., Customized data , ..., Customized data , where m∈M, where the number of customized data in different data sets can be different or the same, depending on the blood relationship between the main data and the customized data.

[0025] S200: Search for a first data module in multiple data sets, and add all first data modules to a first target data family, wherein the first data module appears repeatedly in multiple data sets, and the first data module includes at least one data unit, and the data units in the first data module exist simultaneously in the same data to be stored.

[0026] Specifically, the first data module may have only one data unit or more than two data units, wherein when the first data module has more than two data units, all data units in the first data module are from the same data to be stored, that is, from the same master data or the same customized data. The first data module refers to a completely identical data module that can be found in the data to be stored in at least two different data sets, that is, the first data module appears repeatedly in the data to be stored in different data sets. Of course, the first data module may appear in some data sets or in all data sets.

[0027] Furthermore, by searching in multiple data sets, all first data modules found are different from each other, and all first data modules are added to a first target data family, which is used to accommodate all found first data modules.

[0028] For a better understanding, the search process of the first data module is described in detail below:

[0029] First, each data to be stored in each data set is deconstructed according to the smallest unit to obtain at least one data unit corresponding to each data to be stored.

[0030] For each data to be stored in each data set, at least one data unit corresponding to the data to be stored is combined to obtain at least one data module corresponding to the data to be stored. The data units included in the data module can be arranged adjacently or non-adjacently in the data to be stored, that is, when combining, it is not necessary to combine several adjacent data units. In order to reduce the amount of processing, if the data module after combining a certain data unit is uncommon, the data module is discarded.

[0031] Search for a repeated data module in all data modules corresponding to all data sets, and define the data module as the first data module, that is, the number of occurrences of the first data module in all data modules is greater than a threshold (the threshold may be 2, 3, 4 or a greater value). It should be noted that if two data modules include the same data units but have different arrangement orders of the data units, the two data modules are also considered to be the same data module.

[0032] S300: Generate a first identifier corresponding to each first data module in the first target data family.

[0033] Specifically, after finding the first data module and adding the first data module to the first target data family, a unique first identifier is generated accordingly, and the first identifier and the first data module form a mapping relationship, thereby ensuring a one-to-one correspondence between the first data module and the first identifier, that is, the unique first data module can be found through the unique first identifier.

[0034] S400: Store the correspondence between the first data module and the first identifier and multiple data to be stored, wherein when storing multiple data to be stored, in response to the data to be stored including the first data module, replace the first data module in the data to be stored with the first identifier corresponding to the first data module.

[0035] Specifically, when storing multiple data to be stored, the first data module in the data to be stored is directly replaced with the first identifier. That is to say, the storage space originally occupied by the first data module in the data to be stored is released, and only the space for storing the first identifier is needed. Since the present application also stores the correspondence between the first data module and the first identifier, the corresponding first data module can be directly called through the first identifier. At the same time, the first data module appears repeatedly in multiple data sets. It can be seen that the same first data module is replaced by the first identifier multiple times.

[0036] Different from the prior art, the present application first clusters the data to be stored according to blood relationship to obtain multiple data sets, and then searches for the first data module in the multiple data sets, which helps to improve the efficiency of searching in different data sets; and further generates a first identifier corresponding to the first data module found, stores the correspondence between the first data module and the first identifier, and finally replaces the first data module with the first identifier when storing the data to be stored. Through the above process, it can be seen that the present application replaces the repeated first data module with the first identifier, which can release the space originally used to store the first data module, and only needs to provide a smaller storage space to store the first identifier, thereby reducing the storage space of all data to be stored.

[0037] See also Figure 2 In one embodiment, the step S200 includes:

[0038] S210: For each data set, search for a second data module in the main data and custom data in the data set, and use the found second data module to construct a first data family corresponding to the data set, wherein the second data module appears repeatedly in the data set, and the number of data units included in the second data module is greater than or equal to a first number threshold, and the data units in the second data module exist simultaneously in the same data to be stored.

[0039] Specifically, the second data module may have only one data unit or more than two data units, wherein when the second data module has more than two data units, all data units in the second data module are from the same master data or the same custom data. The second data module refers to a completely identical data module that can be found in the master data and at least one custom data in the same data set, that is, the second data module is repeated in the master data and at least one custom data in the same data set. At the same time, setting the first quantity threshold can avoid the data volume of the first data module being too small. By searching for the larger second data module, on the one hand, the efficiency of searching the second data module can be improved. On the other hand, considering that data with a smaller data volume occupies less storage space, the optimization of the storage space for the data to be stored is not obvious.

[0040] Furthermore, by searching the main data and the customized data in the plurality of data sets, all the second data modules found are different from each other, and all the second data modules are added to the first data family, which is used to accommodate all the found second data modules.

[0041] For a better understanding, the following takes a data set as an example to introduce the specific process of constructing the first data family corresponding to the data set:

[0042] First, each data to be stored in the data set is deconstructed according to the smallest unit, including the main data and all customized data in each data set, to obtain at least one data unit corresponding to each main data and at least one data unit corresponding to each customized data.

[0043] For each master data, at least one data unit corresponding to the master data is combined according to a first quantity threshold to obtain at least one data module corresponding to the master data, wherein the number of data units included in the data module is greater than or equal to the first quantity threshold, and the data units included in the data module can be arranged adjacently or non-adjacently in the master data, that is, when combining, it is not necessary to combine several adjacent data units. Similarly, for each custom data, at least one data unit corresponding to the custom data is combined according to a first quantity threshold to obtain at least one data module corresponding to the custom data, wherein the number of data units included in the data module is greater than or equal to the first quantity threshold, and the data units included in the data module can be arranged adjacently or non-adjacently in the custom data, that is, when combining, it is not necessary to combine several adjacent data units. In order to reduce the amount of processing, if a data module after combining a certain data unit is uncommon, the data module is discarded.

[0044] After obtaining at least one data module corresponding to each master data and at least one data module corresponding to each customized data, the data modules that appear repeatedly are searched in the corresponding master data and customized data in the data set, and the found data modules are defined as the second data modules, that is, the number of occurrences of the second data module in all data modules is greater than a threshold value (the threshold value can be 2, 3, 4 or a larger value). It should be noted that if the data units included in the two data modules are the same, but the arrangement order of the data units is different, the two data modules are also considered to be the same data module.

[0045] S220: Search for a third data module in a first data family corresponding to a plurality of data sets, and determine the found third data module as a first data module, wherein the third data module appears repeatedly in the first data family, and the number of data units included in the third data module is greater than or equal to a second number threshold, the second number threshold is less than or equal to the first number threshold, and the data units in the third data module exist simultaneously in the same second data module.

[0046] Specifically, the third data module may have only one data unit or more than two data units, wherein when the third data module has more than two data units, all data units in the third data module are from the same second data module. The third data module refers to a completely identical data module that can be found in multiple second data modules in the first data family, that is, the third data module is repeated in multiple second data modules in the first data family. At the same time, setting the second quantity threshold can also avoid the data volume of the third data module being too small. By searching for a larger third data module, on the one hand, the efficiency of searching for the third data module can be improved. On the other hand, considering that data with a smaller data volume occupies less storage space, the optimization of the storage space for the data to be stored is not obvious. In addition, the second quantity threshold can be equal to the first quantity threshold, or it can be less than the first quantity threshold, that is, the data volume of the third data module can be equal to the data volume of the second data module, or it can be less than the data volume of the second data module. The specific selection can be selected according to the actual situation, and this application does not make any restrictions.

[0047] Furthermore, by searching in a plurality of second data modules in the first data family, all third data modules found are different from each other, and all third data modules are determined as first data modules.

[0048] For a better understanding, the specific process of searching for the third data module in multiple data sets is introduced:

[0049] First, all second data modules in the first data family are deconstructed according to the smallest unit to obtain at least one data unit corresponding to each second data module.

[0050] For each second data module, at least one data unit corresponding to the second data module is combined according to the second quantity threshold to obtain at least one data module corresponding to the second data module, wherein the number of data units included in the data module is greater than or equal to the second quantity threshold, and the data units included in the data module can be arranged adjacently or non-adjacently in the second data module, that is, when combining, it is not necessary to combine several adjacent data units. In order to reduce the amount of quantity processing, if a data module after combining a certain data unit is uncommon, the data module is discarded.

[0051] After obtaining at least one data module corresponding to each second data module, a repeated data module is searched in each second data module, and the found data module is defined as a third data module, that is, the number of occurrences of the third data module in all data modules is greater than a threshold value (the threshold value may be 2, 3, 4 or a greater value). It should be noted that if two data modules include the same data units but have different arrangement orders of the data units, the two data modules are also considered to be the same data module.

[0052] S230: Add all first data modules to the first target data family.

[0053] Specifically, it is equivalent to adding all the third data modules mentioned above to the first target data family.

[0054] From the above implementation, it can be seen that the second data module that is repeated in the main data and the customized data is first searched in each data set, and then all the second data modules found in all data sets form a first data family, and then the third data module that is repeated is searched in all the second data modules in the first data family, and all the third data modules are determined as first data modules. On the one hand, the customized data is generated by the main data, and there is a high similarity between the customized data and the main data. Therefore, it is easy to find the second data module between the main data and the customized data, and then by searching for the third data module in all the second data modules, the overall search efficiency can be improved; on the other hand, the similarity between the customized data and the main data is very high, so that it is ensured that the third data module finally found is not only repeated in the main data and part of the customized data in the data set, but also repeated between different data sets. In this way, the number of replacements of all the third data modules (i.e., the first data modules) in the subsequent storage of the data to be stored is increased, and the storage space of the data to be stored is further reduced.

[0055] Of course, in some other implementations, the first data module may only appear repeatedly between main data or between customized data in multiple data sets.

[0056] See also Figure 3 , before the above step S400, further comprising:

[0057] S311: Calculate a first data volume, wherein the first data volume is the sum of the storage data volume of a first data module in a plurality of data to be stored replacing the first identifier corresponding to the first data module with the storage data volume of all first data modules in the first target data family.

[0058] Specifically, the first data module is replaced by the first identifier in the plurality of data to be stored, and then all the first data modules in the first target data family are stored separately. The data information of the replaced data to be stored is the same as that of the data to be stored without replacement, but the storage data volume of the two is different. The overall data volume after replacement can be defined as the first data volume. Of course, when calculating the first data volume here, it is not actually necessary to perform the storage action. This is just for the convenience of explanation.

[0059] S312: Determine whether the first data amount is the minimum data amount.

[0060] Specifically, in this embodiment, there is no limitation on the method for determining whether it is the minimum data amount, and it can be solved by using the minimum value of the above-mentioned first data amount as the objective function through an algorithm such as a genetic algorithm, and calculating the minimum value multiple times. Of course, a preset data amount can also be set as the minimum data amount, and as long as the preset data amount is reached, it can be considered as the minimum data amount.

[0061] S313: In response to the first data amount being the minimum data amount, executing the step of storing the correspondence between the first data module and the first identifier and the plurality of data to be stored.

[0062] Specifically, the above-mentioned storage step is executed when it is determined that the first data amount is the minimum data amount. That is to say, even if the first data module is obtained through search, storage cannot be performed directly. Calculation is required to ensure that the calculated first data amount is the minimum data amount, and the storage step is executed when the required storage space is minimized.

[0063] S314: Otherwise, update at least one of the first quantity threshold or the second quantity threshold, and return to the step of searching for the first data module in multiple data sets and adding all the first data modules to the first target data family until the first data amount is the minimum data amount.

[0064] Specifically, when it is determined that the first data volume is not the minimum data volume, at least one of the first quantity threshold in step S210 and the second quantity threshold in step S220 is updated, and steps S210, S220 and S230 are re-executed to search for a new first data module. It can be understood that the first quantity threshold will affect the data volume of the second data module and the third data module, and the second quantity threshold will affect the data volume of the third data module. The first data volume can be changed by updating until the first data volume is the minimum data volume.

[0065] In one embodiment, see Figure 4 , before the above step S400, further comprising:

[0066] S320: Search for a repeated fourth data module in the data to be stored in each data set that does not contain the second data module; wherein the fourth data module includes data units whose number is greater than or equal to a third number threshold, the third number threshold is less than or equal to the first number threshold, and the data units in the fourth data module exist simultaneously in the same data to be stored.

[0067] Specifically, the fourth data module may have only one data unit or more than two data units, wherein when the fourth data module has more than two data units, all data units in the fourth data module are from the same data to be stored. The fourth data module refers to a completely identical data module found in the data to be stored that does not contain the second data module. Setting the third quantity threshold at the same time can also avoid the data volume of the fourth data module being too small. By searching for the larger fourth data module, on the one hand, the efficiency of searching the fourth data module can be improved. On the other hand, considering that the data with a smaller data volume occupies less storage space, the optimization of the storage space for the data to be stored is not obvious. In addition, the third quantity threshold can be equal to the first quantity threshold, or it can be less than the first quantity threshold, that is, the data volume of the fourth data module can be equal to the data volume of the second data module, or it can be less than the data volume of the second data module. The specific selection can be selected according to the actual situation, and this application does not limit it.

[0068] For a better understanding, the specific process of searching for the fourth data module in multiple data sets is introduced:

[0069] First, the data to be stored in each data set that does not contain the second data module is deconstructed according to the smallest unit to obtain at least one data unit corresponding to each data to be stored that does not contain the second data module.

[0070] For each data to be stored that does not contain the second data module, at least one data unit corresponding to the data to be stored that does not contain the second data module is combined according to the third quantity threshold to obtain at least one data module corresponding to the data to be stored that does not contain the second data module, wherein the number of data units included in the data module is greater than or equal to the third quantity threshold, and the data units included in the data module can be arranged adjacently or non-adjacently in the data to be stored, that is, when combining, it is not necessary to combine several adjacent data units. In order to reduce the amount of processing, if a data module after combining a certain data unit is uncommon, the data module is discarded.

[0071] After obtaining at least one data module corresponding to each data to be stored that does not contain the second data module, a repeated data module is searched in each data to be stored that does not contain the second data module, and the found data module is defined as a fourth data module, that is, the number of occurrences of the fourth data module in all data modules is greater than a threshold value (the threshold value may be 2, 3, 4 or a greater value). It should be noted that if the data units included in two data modules are the same, but the arrangement order of the data units is different, the two data modules are also considered to be the same data module.

[0072] S330: Construct a second target data family using all fourth data modules.

[0073] Specifically, by adding all fourth data modules found in the data to be stored that does not contain the second data module to the second target data family, the second target data family is used to accommodate all the found fourth data modules.

[0074] S340: Generate a second identifier corresponding to each fourth data module in the second target data family.

[0075] Specifically, after finding the fourth data module and adding the fourth data module to the second target data family, a unique second identifier is generated accordingly, and the second identifier and the fourth data module form a mapping relationship, thereby ensuring a one-to-one correspondence between the fourth data module and the second identifier, that is, the unique fourth data module can be found through the unique second identifier.

[0076] Further reading Figure 4 , the above step S400 further includes:

[0077] S350: Store the correspondence between the first data module and the first identifier, the correspondence between the fourth data module and the second identifier, and multiple data to be stored, wherein when storing multiple data to be stored, in response to the data to be stored including the first data module, the first identifier corresponding to the first data module is used to replace the first data module in the data to be stored, and in response to the data to be stored including the fourth data module, the second identifier corresponding to the fourth data module is used to replace the fourth data module in the data to be stored.

[0078] Specifically, when storing multiple data to be stored, the first data module in the data to be stored is directly replaced with the first identifier, that is, the storage space of the original first data module in the data to be stored is released, and only the space for storing the first identifier is needed. Similarly, the fourth data module in the data to be stored is directly replaced with the second identifier, that is, the storage space originally occupied by the fourth data module in the data to be stored is released. Since the present application also stores the correspondence between the first data module and the first identifier and the correspondence between the fourth data module and the second identifier, the corresponding first data module can be directly called through the first identifier and the corresponding fourth data module can be directly called through the second identifier. At the same time, the first data module and the fourth data module appear repeatedly in multiple data sets. It can be seen that the same first data module is replaced multiple times by the first identifier and the same fourth data module is replaced multiple times by the second identifier.

[0079] The difference between the above implementation and the implementation of replacing the first data module with only the first identifier in the previous text is that the first data module is a data module that appears repeatedly in the main data and the customized data and exists in multiple data sets. It considers the data module that appears repeatedly in the overall data to be stored. The first data module has universality, and this implementation also considers adding a fourth data module. The fourth data module is a repeated data module found in the data to be stored other than the first data module, that is, it avoids the same part of the data between the main data and the customized data, and searches for repeated data modules for the different parts. Although the fourth data module does not have universality, it has the characteristics of differentiation and customization, and can find the differentiated, customized and repeated data modules from the data to be stored. The storage space of the data to be stored is reduced by replacing the fourth data module with the second identifier. More specifically, the characteristics of universality are applicable to the needs of most or even all customers, while the characteristics of differentiation and customization are applicable to the customized needs of some customers.

[0080] In one embodiment, see Figure 5 , before step S400, further comprising:

[0081] S341: Calculate the second data volume, wherein the second data volume is the storage data volume of a first data module in the data to be stored replaced by a first identifier corresponding to the first data module, the storage data volume of a fourth data module in the data to be stored replaced by a second identifier corresponding to the fourth data module, and the sum of the storage data volume of all first data modules in the first target data family and the storage data volume of all fourth data modules in the second target data family.

[0082] Specifically, among the multiple data to be stored, the first data module is replaced by the first identifier, and the fourth data module is replaced by the second identifier, and then all the first data modules in the first target data family and all the fourth data modules in the second target data family are stored separately. The data information of the replaced data to be stored is the same as that of the data to be stored without replacement, but the storage data volume of the two is different. The overall data volume after replacement can be defined as the second data volume. Of course, when calculating the second data volume here, it is not actually necessary to perform the storage action. This is just for the convenience of explanation.

[0083] S342: Determine whether the second data amount is the minimum data amount.

[0084] Specifically, in this embodiment, there is no limitation on the method for determining whether it is the minimum data amount, and it can be solved by using the minimum value of the second data amount as the objective function through an algorithm such as a genetic algorithm, and calculating the minimum value multiple times. Of course, a preset data amount can also be set as the minimum data amount, and as long as the preset data amount is reached, it can be considered as the minimum data amount.

[0085] S343: In response to the second data amount being the minimum data amount, executing the step of storing the correspondence between the first data module and the first identifier, the correspondence between the fourth data module and the second identifier, and the plurality of data to be stored.

[0086] Specifically, the above-mentioned storage step is executed when it is determined that the second data amount is the minimum data amount. That is to say, even if the first data module and the fourth data module are obtained through searching, storage cannot be performed directly. Calculation is required to ensure that the calculated second data amount is the minimum data amount, and the storage step is performed when the required storage space is minimized.

[0087] S344: Otherwise, update at least one of the first quantity threshold, the second quantity threshold or the third quantity threshold, and return to the step of searching for the first data module in multiple data sets and adding all the first data modules to the first target data family until the second data amount is the minimum data amount.

[0088] Specifically, when it is determined that the second data volume is not the minimum data volume, at least one of the first quantity threshold in step S210, the second quantity threshold in step S220, and the third quantity threshold in step S320 is updated, and steps S210, S220, and S230 are re-executed to search for a new first data module, and steps S320 and S330 are re-executed to search for a new fourth data module. It can be understood that the first quantity threshold will affect the data volume of the second data module, the third data module, and the fourth data module, the second quantity threshold will affect the data volume of the third data module, and the third quantity threshold will affect the data volume of the fourth data module. The second data volume can be changed by updating until the second data volume is the minimum data volume.

[0089] In an application scenario, for the convenience of explanation, the N data sets mentioned above are referenced, and the N data sets are respectively recorded as data set D1, data set D2, ..., D n , ..., data set D N , where n∈N. The data set D n Master data included and M customized data, where the M customized data are respectively recorded as customized data , Customized data , ..., Customized data , ..., Customized data , where m∈M, and define the master data The amount of data is , define custom data The amount of data is Without replacing the first data module by the first identifier and without replacing the fourth data module by the second identifier, the amount of data to be stored is ; By replacing the first data module with the first identifier and replacing the fourth data module with the second identifier, the total amount of data to be stored is ;in, is the data volume of the i-th first data module in the first target data family, is the data volume of the jth fourth data module in the second target data family, , when the first data module does not exist in the master data, The value is 0, when the first data module exists in the master data, The value is 1. , when the fourth data module does not exist in the custom data, The value is 0, when the first data module exists in the master data, The value is 1, a is the number of the first target data family, b is the number of the second target data family, and the number of the family refers to the total number of data modules in the corresponding target data family. It can be seen from the above formula that the repeated first data module and the fourth data module only need to be stored once, and all the repeated first data modules in the main data are replaced, and all the repeated fourth data modules in the customized data are replaced, which can save a lot of storage space.

[0090] In another application scenario, all the repeated first data modules and fourth data modules in the customized data may be replaced.

[0091] Of course, it can be understood that when replacing the main data and the customized data, the specific selection can be at least one of the first data module and the fourth data module, and partial replacement or complete replacement can be selected.

[0092] In one embodiment, see Figure 6 , after the above step S400, further comprising:

[0093] S500: Acquire new data.

[0094] Specifically, the newly added data is relative to the data to be stored. The newly added data is the data added after step 400. It should be noted that the data to be stored at this time has actually been stored in step 400, not literally to be stored. The data to be stored at this time is already in a stored state, and the newly added data is in a to-be-stored state at this time. In addition, the newly added data is different from the main data and customized data in the data to be stored.

[0095] S600: Determine whether there is a blood relationship between the newly added data and the data to be stored, wherein the newly added data includes at least one data unit.

[0096] Specifically, the blood relationship means that the customized data is generated by a certain master data, and thus the customized data has a blood relationship with the corresponding master data that generates it. When judging whether the newly added data has a blood relationship, it is more specifically to judge whether the newly added data is master data or customized data. If it is master data, it has no blood relationship with the master data in the data to be stored. If it is customized data, it has a blood relationship with a certain master data in the data to be stored.

[0097] S700: storing the newly added data according to the blood relationship between the newly added data and the data to be stored.

[0098] Specifically, different storage methods may be selected according to whether there is a blood relationship between the newly added data and the data to be stored.

[0099] In one application scenario, the first data module is searched for repeated occurrences in the newly added data that has no blood relationship, and then the first data module in the newly added data is replaced with the first identifier; the fourth data module is searched for repeated occurrences in the newly added data that has a blood relationship, and then the fourth data module in the newly added data is replaced with the second identifier.

[0100] In another application scenario, the first data module is searched for repeated occurrences in the newly added data that has no blood relationship, and then the first data module in the newly added data is replaced with a first identifier; the first data module and the fourth data module are searched for repeated occurrences in the newly added data that has a blood relationship, and then the first data module in the newly added data is replaced with the first identifier and the fourth data module in the newly added data is replaced with the second identifier.

[0101] In one embodiment, see Figure 7 The above step S700 includes:

[0102] S710: In response to the fact that there is no blood relationship between the newly added data and the data to be stored, searching for a fifth data module in the newly added data, wherein the fifth data module exists in both the first data family and the newly added data.

[0103] Specifically, there is no blood relationship between the newly added data and the data to be stored, that is, the newly added data is the main data, and at this time, the data modules of the first data family can be matched to find out the data modules that appear repeatedly in the newly added data. The fifth data module is equivalent to part of the third data module or all the third data modules in the first data family, and these third data modules exist in the newly added data.

[0104] S720: Calculate the third data volume, where the third data volume is the storage data volume of multiple data to be stored after replacing the first data module in the data to be stored with the first identifier corresponding to the first data module, the storage data volume after replacing the fourth data module in the data to be stored with the second identifier corresponding to the fourth data module, the storage data volume after replacing the fifth data module in the newly added data with the first identifier corresponding to the fifth data module, and the sum of the storage data volumes of all first data modules in the first target data family and the storage data volumes of all fourth data modules in the second target data family.

[0105] Specifically, the fifth data module is in the first target data family. The fifth data module can be part of the first data module or all of the first data modules, which is subject to the search result. Therefore, the fifth data module also corresponds to the first identifier. The first data module is replaced by the first identifier in multiple data to be stored, the fourth data module is replaced by the second identifier, and the fifth data module is replaced by the first identifier in the newly added data. Then, all the first data modules (including the fifth data module) in the first target data family and all the fourth data modules in the second target data family are stored separately. The data information of the replaced data to be stored and the newly added data is the same as that of the data to be stored and the newly added data that are not replaced, but the storage data volume of the two is different. The overall data volume after replacement can be defined as the third data volume. Of course, when calculating the third data volume here, it is not actually necessary to perform the storage action. This is just for the convenience of explanation.

[0106] S730: Determine whether the third data amount is the minimum data amount.

[0107] Specifically, in this embodiment, there is no limitation on the method for determining whether it is the minimum data amount, and it can be solved by using the minimum value of the third data amount as the objective function through an algorithm such as a genetic algorithm, and calculating the minimum value multiple times. Of course, a preset data amount can also be set as the minimum data amount, and as long as the preset data amount is reached, it can be considered as the minimum data amount.

[0108] S740: In response to the third data amount being the minimum data amount, the correspondence between the first data module and the first identifier, the correspondence between the fourth data module and the second identifier, multiple data to be stored, and newly added data are stored; wherein, when storing multiple data to be stored and newly added data, in response to the data to be stored including the first data module, the first identifier corresponding to the first data module is used to replace the first data module in the data to be stored; in response to the data to be stored including the fourth data module, the second identifier corresponding to the fourth data module is used to replace the fourth data module in the data to be stored; and in response to the newly added data including the fifth data module, the first identifier corresponding to the fifth data module is used to replace the fifth data module in the data to be stored.

[0109] Specifically, the above-mentioned storage step is executed when it is determined that the third data amount is the minimum data amount. That is to say, even if the fifth data module is obtained through search, storage cannot be performed directly. Calculation is required to ensure that the calculated fifth data amount is the minimum data amount, and the storage step is performed when the required storage space is minimized.

[0110] S750: Otherwise, update at least one of the first quantity threshold, the second quantity threshold or the third quantity threshold, and return to the step of searching for the first data module in multiple data sets and adding all the first data modules to the first target data family until the third data amount is the minimum data amount.

[0111] Specifically, when it is determined that the third data volume is not the minimum data volume, at least one of the first quantity threshold in step S210, the second quantity threshold in step S220, and the third quantity threshold in step S320 is updated, and steps S210, S220, and S230 are re-executed to find a new first data module, steps S320 and S330 are re-executed to find a new fourth data module, and step S710 is re-executed to find a new fifth data module. It can be understood that the first quantity threshold will affect the data volume of the second data module, the third data module, the fourth data module, and the fifth data module, the second quantity threshold will affect the data volume of the third data module and the fifth data module, and the third quantity threshold will affect the data volume of the fourth data module. The third data volume can be changed by updating until the third data volume is the minimum data volume.

[0112] It can be seen from the above implementation that when there is no blood relationship between the newly added data and the data to be stored, by searching for the repeated fifth data module, the fifth data module exists in both the newly added data and the first data family. Although there is no blood relationship between the newly added data and the data to be stored, the newly added data and the data to be stored will still have the same data module. By further searching for the fifth data module, the storage space for the new data can be reduced.

[0113] In one implementation, the above step S700 includes:

[0114] In response to a blood relationship between the newly added data and the data to be stored, the newly added data is stored. When storing the newly added data, in response to the newly added data including a first data module, the first data module in the newly added data is replaced with a first identifier corresponding to the first data module, and in response to the newly added data including a fourth data module, the fourth data module in the newly added data is replaced with a second identifier corresponding to the fourth data module.

[0115] Specifically, when there is a blood relationship between the newly added data and the data to be stored, that is, the similarity between the newly added data and a certain main data in the data to be stored is high, therefore, when storing the newly added data, the repeated first data module and the fourth data module can be searched in the newly added data and replaced by the corresponding identifiers, which can fully reduce the storage space occupied by the new data.

[0116] Of course, in some other embodiments, in response to the presence of a blood relationship between the newly added data and the data to be stored, the newly added data is stored. When storing the newly added data, in response to the newly added data including a first data module, the first data module in the newly added data is replaced with a first identifier corresponding to the first data module, or in response to the newly added data including a fourth data module, the fourth data module in the newly added data is replaced with a second identifier corresponding to the fourth data module.

[0117] In one implementation, the above step S700 includes:

[0118] In response to the existence of a blood relationship between the newly added data and the data to be stored, and the second data modules in the first data family corresponding to the first main data in the multiple data to be stored all exist in the newly added data, the newly added data is stored, wherein the sixth data module exists in both the first main data and the first target data family. When storing the newly added data, in response to the new data including the sixth data module, the sixth data module in the newly added data is replaced with the first identifier corresponding to the sixth data module.

[0119] Specifically, when there is a blood relationship between the newly added data and the data to be stored, the second data modules in the first data family corresponding to the first master data can all be found in the newly added data, that is, the newly added data at this time at least contains the second data module in the first data family corresponding to the first master data, and the first data module corresponding to the first master data can be defined as the sixth data module, that is, the sixth data module exists in the first data module in the first master data, that is, the sixth data module exists in both the first master data and the first target data family. When the newly added data is stored, the first identifier corresponding to the sixth data module can be directly used to replace the sixth data module in the newly added data, without matching in all the first data modules in the first target data family, thereby improving the storage efficiency.

[0120] See also Figure 8 , Figure 8 It is a schematic diagram of the structure of an implementation of a storage device of the present application. The storage device 300 includes a clustering module 310, a search module 320, a generation module 330 and a storage module 340.

[0121] The clustering module 310 is used to cluster the multiple data to be stored based on blood relationship to obtain multiple data sets.

[0122] The search module 320 is used to search for the first data module in multiple data sets and add all the first data modules to the first target data family.

[0123] The generating module 330 is used to generate a first identifier corresponding to each first data module in the first target data family.

[0124] The storage module 340 is used to store the correspondence between the first data module and the first identifier and a plurality of data to be stored.

[0125] Among them, the storage device 300 executes the method steps in any of the above-mentioned implementation modes when working. The detailed method steps can be found in the above-mentioned related content and will not be repeated here.

[0126] The storage device 300 may be a device having data processing and storage functions.

[0127] See also Fig. 9 , Fig. 9 It is a structural diagram of an embodiment of an electronic device of the present application, wherein the electronic device 40 includes a memory 401 and a processor 402 coupled to each other, wherein the memory 401 stores program data (not shown), and the processor 402 calls the program data to implement the method in any of the above embodiments. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0128] See also Fig.10 , Fig.10 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 50 stores program data 500. When the program data 500 is executed by a processor, the method in any of the above embodiments is implemented. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0129] Among them, the computer-readable storage medium 50 can specifically be a device that can store the program data 500, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or it can also be a server that stores the program data 500. The server can send the stored program data 500 to other devices for execution, or it can also execute the stored program data 500 by itself.

[0130] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for data storage, characterized in that: The method comprises: Clustering a plurality of data to be stored according to a blood relationship to obtain a plurality of data sets, wherein the data to be stored includes at least one data unit, and the plurality of data to be stored in the data set includes master data and at least one customized data corresponding to the master data, and the customized data corresponding to the master data is generated based on the master data, wherein the blood relationship means that the customized data is generated by a certain master data, and thus the customized data has a blood relationship with the corresponding master data generating the customized data; Searching for a first data module in the plurality of data sets, and adding all the first data modules to a first target data family, wherein the first data module appears repeatedly in the plurality of data sets, and the first data module includes at least one data unit, and the data units in the first data module exist simultaneously in the same data to be stored; Generate a first identifier corresponding to each of the first data modules in the first target data family; The correspondence between the first data module and the first identifier and multiple data to be stored are stored, wherein when storing multiple data to be stored, in response to the data to be stored including the first data module, the first identifier corresponding to the first data module is used to replace the first data module in the data to be stored.

2. The method according to claim 1, characterized in that The step of searching for a first data module in the plurality of data sets and adding all the first data modules to a first target data family includes: For each of the data sets, searching for a second data module in the main data and the customized data in the data set, and using the found second data module to construct a first data family corresponding to the data set, wherein the second data module appears repeatedly in the data set, and the number of the data units included in the second data module is greater than or equal to a first number threshold, and the data units in the second data module exist simultaneously in the same data to be stored; In the first data families corresponding to the plurality of data sets, searching for a third data module, and determining the found third data module as the first data module, wherein the third data module appears repeatedly in the first data family, and the number of the data units included in the third data module is greater than or equal to a second number threshold, the second number threshold is less than or equal to the first number threshold, and the data units in the third data module exist in the same second data module at the same time; All of the first data modules are added to a first target data family.

3. The method according to claim 2, characterized in that Before storing the correspondence between the first data module and the first identifier and the plurality of data to be stored, the method further includes: Calculating a first data volume, wherein the first data volume is the sum of storage data volumes of a plurality of the data to be stored by replacing the first data module in the data to be stored with the first identifier corresponding to the first data module and storage data volumes of all the first data modules in the first target data family; determining whether the first data amount is a minimum data amount; In response to the first data amount being the minimum data amount, executing the step of storing the correspondence between the first data module and the first identifier and the plurality of data to be stored; Otherwise, update at least one of the first quantity threshold or the second quantity threshold, and return to the step of searching for the first data module in the multiple data sets and adding all the first data modules to the first target data family until the first data amount is the minimum data amount.

4. The method according to claim 2, characterized in that: Before storing the correspondence between the first data module and the first identifier and the plurality of data to be stored, the method further includes: Searching for a recurring fourth data module in the data to be stored that does not contain the second data module in each of the data sets; wherein the number of data units included in the fourth data module is greater than or equal to a third number threshold, the third number threshold is less than or equal to the first number threshold, and the data units in the fourth data module exist simultaneously in the same data to be stored; constructing a second target data family using all of the fourth data modules; Generate a second identifier corresponding to each of the fourth data modules in the second target data family; and storing the correspondence between the first data module and the first identifier and a plurality of the data to be stored, wherein when storing the plurality of the data to be stored, in response to the data to be stored including the first data module, replacing the first data module in the data to be stored with the first identifier corresponding to the first data module, specifically comprises: The correspondence between the first data module and the first identifier, the correspondence between the fourth data module and the second identifier, and multiple data to be stored are stored, wherein when storing multiple data to be stored, in response to the data to be stored including the first data module, the first identifier corresponding to the first data module is used to replace the first data module in the data to be stored, and in response to the data to be stored including the fourth data module, the second identifier corresponding to the fourth data module is used to replace the fourth data module in the data to be stored.

5. The method according to claim 4, characterized in that Before storing the correspondence between the first data module and the first identifier and the plurality of data to be stored, the method further includes: Calculate a second data volume, wherein the second data volume is a sum of a storage data volume of a plurality of the data to be stored by replacing the first data module in the data to be stored with the first identifier corresponding to the first data module, a storage data volume of a fourth data module in the data to be stored by replacing the second identifier corresponding to the fourth data module, and a storage data volume of all the first data modules in the first target data family and a storage data volume of all the fourth data modules in the second target data family; determining whether the second data amount is a minimum data amount; In response to the second data amount being the minimum data amount, executing the step of storing the correspondence between the first data module and the first identifier, the correspondence between the fourth data module and the second identifier, and the plurality of data to be stored; Otherwise, update at least one of the first quantity threshold, the second quantity threshold or the third quantity threshold, and return to execute the step of searching for the first data module in multiple data sets and adding all of the first data modules to the first target data family until the second data amount is the minimum data amount.

6. The method according to claim 4, characterized in that After the step of storing the correspondence between the first data module and the first identifier and the plurality of data to be stored, the method further includes: Get new data; Determining whether there is a blood relationship between the newly added data and the data to be stored, wherein the newly added data includes at least one data unit; The newly added data is stored according to the blood relationship between the newly added data and the data to be stored.

7. The method according to claim 6, characterized in that The step of storing the newly added data according to the blood relationship between the newly added data and the data to be stored includes: In response to the absence of a blood relationship between the newly added data and the data to be stored, searching for a fifth data module in the newly added data, wherein the fifth data module exists in both the first data family and the newly added data; Calculate a third data volume, wherein the third data volume is the storage data volume of multiple data to be stored after replacing the first data module in the data to be stored with the first identifier corresponding to the first data module, the storage data volume after replacing the fourth data module in the data to be stored with the second identifier corresponding to the fourth data module, the storage data volume after replacing the fifth data module in the newly added data with the first identifier corresponding to the fifth data module, and the sum of the storage data volumes of all the first data modules in the first target data family and the storage data volumes of all the fourth data modules in the second target data family; determining whether the third data amount is a minimum data amount; In response to the third data amount being the minimum data amount, the correspondence between the first data module and the first identifier, the correspondence between the fourth data module and the second identifier, the plurality of data to be stored, and the newly added data are stored; wherein, when storing the plurality of data to be stored and the newly added data, in response to the data to be stored including the first data module, the first identifier corresponding to the first data module is used to replace the first data module in the data to be stored; in response to the data to be stored including the fourth data module, the second identifier corresponding to the fourth data module is used to replace the fourth data module in the data to be stored; and in response to the newly added data including the fifth data module, the first identifier corresponding to the fifth data module is used to replace the fifth data module in the data to be stored; Otherwise, update at least one of the first quantity threshold, the second quantity threshold or the third quantity threshold, and return to execute the step of searching for the first data module in multiple data sets and adding all of the first data modules to the first target data family until the third data amount is the minimum data amount.

8. The method according to claim 6, characterized in that The step of storing the newly added data according to the blood relationship between the newly added data and the data to be stored includes: In response to a blood relationship between the newly added data and the data to be stored, the newly added data is stored, wherein, when storing the newly added data, in response to the newly added data including the first data module, the first identifier corresponding to the first data module is used to replace the first data module in the newly added data, and in response to the newly added data including the fourth data module, the second identifier corresponding to the fourth data module is used to replace the fourth data module in the newly added data.

9. The method according to claim 8, characterized in that The step of storing the newly added data according to the blood relationship between the newly added data and the data to be stored includes: In response to the existence of a blood relationship between the newly added data and the data to be stored, and the second data modules in the first data family corresponding to the first master data in multiple data to be stored all exist in the newly added data, the newly added data is stored, wherein the sixth data module exists in both the first master data and the first target data family. When storing the newly added data, in response to the newly added data including the sixth data module, the first identifier corresponding to the sixth data module is used to replace the sixth data module in the newly added data.

10. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the data storage method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the steps in the method according to any one of claims 1 to 9.

Images