Data storage method and device, equipment and medium

Abstract

The invention discloses a data storage method and device, equipment and a medium, is applied to the technical field of big data, and is used for solving the problem that in the prior art, more information cannot be stored under the same storage capacity, so that the big data processing capacity is poor. The method specifically comprises the following steps: acquiring to-be-stored data of each entry in a merging period from a cache database; performing similarity matching on the to-be-stored data, and grouping the to-be-stored data according to a similarity matching result to obtain a plurality of to-be-merged data groups; and merging the to-be-stored data of all entries contained in each to-be-merged data group into target to-be-stored data of one entry, and storing the target to-be-stored data in the merged database. Thus, by merging the to-be-stored data of all the entries contained in the to-be-merged data set into the target to-be-stored data of one entry, more data can be stored under the same storage capacity, and the processing capacity of big data is improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data storage method, device, equipment and medium. Background Art

[0002] In the field of big data storage, it has evolved from the initial HDFS and HBase based on Hadoop technology to the use of different technologies according to different application scenarios. For example, in terms of structured data storage, there are MPP databases, and in terms of object-relational data storage, graph databases have been developed. Now, with the in-depth and extensive application of the Internet of Things, time series databases have been born. However, the above databases are only designed to adapt to data characteristics and application scenarios, and do not consider larger-scale expansion to accommodate more data. With the advent of the digital age, the scale of data has exploded. With the exponential growth of data volume and the hundreds of millions or even billions of incremental data every day, data storage faces the challenge of storing a large amount of data.

[0003] At present, there are two main ways to deal with a large amount of data storage: the first is to deal with it by continuously expanding capacity and adding equipment; the second is a fixed storage period, and data that exceeds the storage period will be discarded. These two methods are generally used in combination, that is, to discard historical data and maintain moderate capacity expansion. However, with the increasing popularity of mobile networks and the Internet of Things, data is growing exponentially. For the method of fixed storage period, the storage period will only become shorter and shorter when the storage capacity remains unchanged, and finally it cannot meet the most basic application needs. In desperation, it has to expand capacity; for enterprises that mainly respond to capacity expansion, the expansion period will become shorter and shorter, and in the end, they have to shorten the storage period. The root cause of all these problems is that the amount of data that needs to be stored is too large. Therefore, how to store more information with the same storage capacity is an urgent problem to be solved. Summary of the invention

[0004] The present application provides a data storage method, device, equipment and medium to solve the problem in the prior art that more information cannot be stored under the same storage capacity, resulting in poor big data processing capabilities.

[0005] The technical solutions provided by this application are as follows:

[0006] In one aspect, the present application provides a data storage method, comprising:

[0007] Obtain the data to be stored for each entry in the merge cycle from the cache database;

[0008] Perform similarity matching on the data to be stored of each entry to obtain a similarity matching result;

[0009] The data to be stored of each entry is grouped according to the similarity matching result to obtain multiple data groups to be merged;

[0010] For each data group to be merged, merge the data to be stored of all entries included in the data group to be merged into target data to be stored of one entry, the target data to be stored includes a common part and a merged part, the common part is composed of the common part of the original data to be stored of all entries included in the group, and the merged part is composed of the different parts of the original data to be stored of all entries included in the group;

[0011] The target data to be stored of each data group to be merged is stored in the merged database.

[0012] Optionally, similarity matching is performed on the data to be stored of each entry to obtain a similarity matching result, including:

[0013] Determine the similarity between the data to be stored of each entry in the merging period and the data to be stored of other entries in the merging period according to the data type and / or application scenario of the data to be stored of each entry;

[0014] The similarity between the data to be stored of each entry in the merging period and the data to be stored of other entries in the merging period is taken as the similarity matching result.

[0015] Optionally, for each data group to be merged, the data to be stored of all entries contained in the data group to be merged are merged into target data to be stored of one entry, including:

[0016] cyclically performing grouping iterative merging processing on the to-be-stored data of all entries included in the to-be-merged data group until the total number of merged sub-results is less than a preset first number threshold;

[0017] Merge each merge sub-result to obtain the target data to be stored for an entry;

[0018] Among them, the group iterative merging processing includes: grouping the data to be merged according to a preset first quantity threshold to obtain multiple data groups to be merged; merging the data in each data group to be merged to obtain merged sub-results; wherein, when the group iterative merging processing is executed for the first time, the data to be merged is the data to be stored of all entries contained in the data group to be merged; when the group iterative merging processing is not executed for the first time, the data to be merged is the merged sub-results obtained by the previous execution of the group iterative merging processing.

[0019] Optionally, for each data group to be merged, after merging the data to be stored of all entries contained in the data group to be merged into the target data to be stored of one entry, the method further includes:

[0020] According to the target data to be stored for each entry, an index field and an index range of the target data to be stored are established;

[0021] The index fields and index ranges are associated with the target data to be stored and stored in the merged database.

[0022] Optionally, before obtaining the to-be-stored data of each entry in the merging cycle from the cache database, the following step is further included:

[0023] Receive the original storage data of each item input from the outside in real time;

[0024] According to the preset data processing rules, the data format of the original stored data is converted to obtain the data to be stored for each item;

[0025] The data to be stored of each entry is stored in the cache database.

[0026] Optionally, after storing the data to be stored of each entry in the cache database, the process further includes:

[0027] Preprocessing the data to be stored in each entry in the cache database according to a preset cycle to obtain a preprocessing result of the data to be stored; wherein the preprocessing includes statistical processing and / or key value determination processing;

[0028] The data to be stored of each entry and the pre-processing results of the data to be stored are stored in the database before merging.

[0029] In another aspect, the present application provides a data storage device, comprising:

[0030] A first data acquisition unit, used to acquire the to-be-stored data of each entry in a merging period from the cache database;

[0031] A matching unit, used for performing similarity matching on the data to be stored of each entry to obtain a similarity matching result;

[0032] A grouping unit, used to group the data to be stored of each entry according to the similarity matching result to obtain multiple data groups to be merged;

[0033] a merging unit, for merging, for each data group to be merged, the data to be stored of all entries included in the data group to be merged into target data to be stored of one entry, wherein the target data to be stored includes a common part and a merged part, wherein the common part is composed of the common part of the original data to be stored of all entries included in the group, and the merged part is composed of the different parts of the original data to be stored of all entries included in the group;

[0034] The storage unit is used to store the target to-be-stored data of each to-be-merged data group in the merged database.

[0035] Optionally, the matching unit is specifically used for:

[0036] Determine the similarity between the data to be stored of each entry in the merging period and the data to be stored of other entries in the merging period according to the data type and / or application scenario of the data to be stored of each entry;

[0037] The similarity between the data to be stored of each entry in the merging period and the data to be stored of other entries in the merging period is taken as the similarity matching result.

[0038] On the other hand, the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the data storage method provided by the present application is implemented when the processor executes the computer program.

[0039] On the other hand, the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the data storage method provided by the present application is implemented.

[0040] The beneficial effects of this application are as follows:

[0041] In this application, a cache database is set up to store the data to be stored of each item, and the data to be stored is obtained from the cache database according to the merge cycle. The merging of the data to be stored and the storage of the data in the cache are independent of each other, avoiding the impact of data merging on the data storage process; the merging process is performed by presetting the merging cycle, which can reduce the impact on the overall performance of the system. By merging the data to be stored of all items contained in the data group to be merged into the target data to be stored of one item, more data can be stored under the same storage capacity, thereby improving the processing capability of the resulting big data.

[0042] Other features and advantages of the present application will be described in the subsequent description, and in part, will become apparent from the description, or will be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0044] Figure 1 This is a schematic diagram of the overall framework of the data storage method in the embodiment of the present application;

[0045] Figure 2A schematic diagram of a specific process of a method for storing data in a data cache in an embodiment of the present application;

[0046] Figure 3 This is a schematic diagram of a specific process of a data storage method of a pre-merger database in an embodiment of the present application;

[0047] Figure 4 This is a functional structure diagram of a data storage device in an embodiment of the present application;

[0048] Figure 5 Schematic diagram of the hardware structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and beneficial effects of this application clearer, the technical solution in the embodiment of this application will be clearly and completely described below in conjunction with the drawings in the embodiment of this application. Obviously, the described embodiment is only a part of the embodiment of this application, not all the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] At present, there are two main ways to store big data: one is to expand capacity, that is, to add storage devices; the other is to set a fixed storage period and automatically delete expired data. Usually, these two methods are used in combination to manage storage by moderately expanding capacity and discarding old data. However, with the popularization of mobile networks and the Internet of Things, the amount of data has increased dramatically. If the storage capacity remains unchanged, the fixed storage period will continue to shorten, making it difficult to meet application needs, resulting in the need for capacity expansion. Enterprises that rely on capacity expansion will also have to shorten the capacity expansion period due to the large amount of data, and ultimately still need to reduce the storage period. The root cause of these problems is the huge amount of data. Therefore, how to store more information with limited storage capacity has become a problem that needs to be solved urgently.

[0051] To this end, in an embodiment of the present application, the data to be stored of each entry within the merge period is obtained from the cache database; similarity matching is performed on the data to be stored of each entry to obtain a similarity matching result; the original data to be stored of each entry is grouped according to the similarity matching result to obtain multiple data groups to be merged; for each data group to be merged, the data to be stored of all entries contained in the data group to be merged are merged into the target data to be stored of one entry, the target data to be stored includes the same part and the merged part, the same part is composed of the same part of the original data to be stored of all entries contained in the group, and the merged part is composed of the different parts of the original data to be stored of all entries contained in the group; the target data to be stored of each data group to be merged is stored in the merged database. In this way, by setting the cache database to store the data to be stored of each entry, the data to be stored is obtained from the cache database according to the merge period, the merging of the data to be stored and the storage of the data in the cache are independent of each other, avoiding the influence of data merging on the data storage process; the merging process is performed by presetting the merging period, which can reduce the influence on the overall performance of the system. By merging the data to be stored of all items included in the data group to be merged into the target data to be stored of one item, more data can be stored under the same storage capacity, thereby improving the processing capability of large data.

[0052] After introducing the application scenarios and design concepts of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described in detail below.

[0053] The present application embodiment provides a data storage method, which is applied to a server. Figure 1 As shown, the general process of the data storage method provided in the embodiment of the present application is as follows:

[0054] Step 101: Obtain the data to be stored of each entry in the merging cycle from the cache database.

[0055] In practical applications, the data to be stored is the data that needs to be stored in a mobile network, the Internet of Things, or other information transmission networks with a unified data format. In the data to be stored in the mobile network, the Internet of Things, or other information transmission networks, in addition to completely repeated data, there is a large amount of similar data. Specifically, the characteristics of the data to be stored are that the values ​​of most data columns are the same, and only the values ​​of one or two columns are different. For example, in a mobile network environment, the user's communication records, location information, device status, and other data may contain a large amount of similar data. For example, the location information of the same user at different time points may have only slight changes. The data collected by the Internet of Things devices often have similarities. For example, the readings of multiple temperature sensors at different time points may have only slight differences. In order to avoid the impact of data merging progress on data storage, the data to be stored is processed after the data to be stored is obtained for each entry, and the data to be stored is stored in the cache database. According to the preset merging cycle, data is obtained from the cache database and the relevant processes for data storage are executed. Among them, the merging cycle can be set according to actual needs.

[0056] Step 102: Perform similarity matching on the data to be stored of each entry to obtain a similarity matching result.

[0057] In practical applications, similarity matching is to identify and quantify the similarity between the data to be stored of different items based on the data type and data characteristics, and provide a good data basis for subsequent data integration. In specific implementation, similarity matching is performed on the data to be stored of each item to obtain the similarity matching result, which can be adopted but not limited to the following methods:

[0058] First, according to the data type and / or data feature of the data to be stored in each entry, the similarity between the data to be stored in each entry in the merging period and the data to be stored in other entries in the merging period is determined.

[0059] Then, the similarity between the data to be stored of each entry in the merging period and the data to be stored of other entries in the merging period is taken as the similarity matching result.

[0060] Specifically, data features refer to column features of data. The similarity calculation method can adopt a nominal attribute similarity calculation method, a binary attribute similarity calculation method or an ordinal attribute similarity calculation method. The data type similarity can be calculated by adopting the selected similarity calculation method according to the data type, and the data feature similarity can be calculated by adopting the selected similarity calculation method according to the data feature. According to the data type or data feature of the data to be stored of each entry, the data type similarity or data feature similarity between the data to be stored of each entry in the merging period and the data to be stored of other entries in the merging period can be determined. According to the data type and data feature of the data to be stored of each entry, the data type similarity and data feature similarity between the data to be stored of each entry in the merging period and the data to be stored of other entries in the merging period can be determined. According to the preset weight coefficients corresponding to the data type similarity and data feature similarity, a weighted average calculation method is adopted to calculate the similarity between the data to be stored of each entry and the data to be stored of other entries in the merging period.

[0061] Step 103: grouping the data to be stored of each entry according to the similarity matching result to obtain multiple data groups to be merged.

[0062] In practical applications, a similarity threshold is preset, and the original data to be stored of each entry is grouped to obtain multiple data groups to be merged according to the similarity matching results and the preset similarity threshold. Among them, the similarity threshold is used to determine the data to be stored that is merged into the same group. The threshold can be a fixed similarity value, or it can be dynamically determined according to the distribution of the data. Specifically, entries with a similarity greater than the similarity threshold can be selected to be merged into the same group. Alternatively, the two entries with the highest similarity can be found first to be merged, and then the merging range can be gradually expanded until the similarity of all entries is lower than the similarity threshold or reaches a predetermined number of clusters. In addition, hierarchical clustering or K-means clustering algorithm can also be used to group the data to be stored of each entry according to the similarity matching results.

[0063] In specific implementation, the data to be stored of each item is grouped according to the similarity matching result. According to the similarity matching result and the preset similarity threshold, multiple data groups to be merged are obtained, and non-merged data groups may also be obtained. Among them, the similarity thresholds of the data to be stored of each item in the non-merged data group and the data in the multiple data groups to be merged are all lower than the preset similarity threshold. The data similarity of each item in the non-merged data group is low, and merging will result in additional occupation of resources. Therefore, for the non-merged data group, it will be directly output to the merged database for storage.

[0064] Step 104: For each data group to be merged, the data to be stored of all entries contained in the data group to be merged are merged into target data to be stored for one entry, and the target data to be stored includes an identical part and a merged part, the identical part is composed of the identical part of the original data to be stored of all entries contained in the group, and the merged part is composed of the different parts of the original data to be stored of all entries contained in the group.

[0065] In practical applications, the merge here refers to lossless merge or lossy merge. The same part of the target data to be stored can be composed of the same column data in the original data to be stored of all entries contained in the group, and the merged part is composed of different column data in the original data to be stored of all entries contained in the group. For example, Table 1 is the data before merging, and Table 2 is the data after merging. It can be seen from Tables 1 and 2 that the values ​​of C5 and C6 in the three data are different and need to be merged. In specific implementation, the merged part includes a merged name-value pair list, and the multiple merged name-value pairs listed in the merged name-value pair list can be separated by a specific delimiter (such as a comma) when stored, or the merged data items can be stored in JSON format.

[0066] Table 1 Data before merging

[0067] Column Name Column 1 Column 2 Column 3 Column 4 Column 5 Column 6 Data 1 V11 V12 V13 V14 V15 V16 Data 2 V11 V12 V13 V14 V25 V26 Data 3 V11 V12 V13 V14 V35 V36

[0068] Table 2 Merged data

[0069]

[0070] In specific implementation, for each data group to be merged, the data to be stored of all entries contained in the data group to be merged are merged into the target data to be stored of one entry, which can be implemented in but not limited to the following ways:

[0071] cyclically performing grouping iterative merging processing on the to-be-stored data of all entries included in the to-be-merged data group until the total number of merged sub-results is less than a preset first number threshold;

[0072] Merge each merge sub-result to obtain the target data to be stored for an entry;

[0073] Among them, the group iterative merging processing includes: grouping the data to be merged according to a preset first quantity threshold to obtain multiple data groups to be merged; merging the data in each data group to be merged to obtain merged sub-results; wherein, when the group iterative merging processing is executed for the first time, the data to be merged is the data to be stored of all entries contained in the data group to be merged; when the group iterative merging processing is not executed for the first time, the data to be merged is the merged sub-results obtained by the previous execution of the group iterative merging processing.

[0074] In practical applications, the first quantity threshold corresponds to the maximum amount of merged data corresponding to the merging capability. The amount of data in the merging cycle is often hundreds of millions or even billions of items. Even if it is processed in a distributed memory computing cluster, the merging may not be completed in one go. At this time, the data to be stored of all items included in the data group to be merged can be cyclically grouped and iterated to merge, that is, according to the first quantity threshold, the data to be stored of all items included in the data group to be merged are grouped, and each group of data to be stored is merged to obtain a merged sub-result. Then, the merged sub-results are grouped according to the first quantity threshold, and each group of merged sub-results is merged to obtain the current merged sub-result corresponding to the merged sub-result obtained by the last merge. The grouping and merging process is cyclically executed until the number of merged sub-results obtained is less than the first quantity threshold. At this time, the merged sub-results are directly merged to obtain a target data to be stored of an item, wherein the target data to be stored is obtained after all the data to be stored in the data group to be merged are completely merged.

[0075] Step 105: Store the target data to be stored of each data group to be merged in the merged database.

[0076] In practical applications, after obtaining the target data to be stored for each data group to be merged, it is stored in the merged database. In order to reduce the occupancy of the cache, after storing the target data to be stored in the merged database, the data to be stored of each entry in the merge cycle in the cache database can be deleted.

[0077] In a possible implementation, in order to optimize the query path and reduce the query time, after merging the to-be-stored data of all items contained in the to-be-merged data group into one item of target to-be-stored data for each to-be-merged data group, an index field and index range corresponding to the target to-be-stored data may be established. The specific establishment process may be in the following manner but is not limited to:

[0078] First, according to the target data to be stored for each entry, an index field and an index range of the target data to be stored are established;

[0079] Then, the index field and index range are associated with the target data to be stored and stored in the merged database.

[0080] In practical applications, the index fields and index ranges are mainly set for the merge parallel. According to actual needs, various different indexing methods can be adopted for the merge parallel. For example: The first indexing method is to index the entire merge parallel as an index column. Index column name: merge parallel; index values: {{column 5: V15, column 6: V16}, {column 5: V25, column 6: V26}, {column 5: V35, column 6: V36}}. The second indexing method is to create indexes for multiple columns of the merge parallel separately. For example, in Table 2, the merge parallel can be indexed as follows: Index column name: column 5, index values: V15, V25, V35; Index column name: column 6, index values: V16, V26, V36. The third indexing method is for columns that require range queries. Based on the first and second indexing methods above, a range index can be additionally added. For example, in the data of Table 2, if range queries are required for column 5, the following indexes are added (assuming: V15 < V25 < V35): Index column name: starting value of column 5, index value: V15; Index column name: ending value of column 5, index value: V35.

[0081] In a possible implementation manner, referring to Figure 2 as shown, before obtaining the data to be stored for each entry during the merge period from the cache database, a data warehousing method for the data cache library is also provided. The data warehousing method includes:

[0082] Step 201: Receive the original storage data of each entry input in real time from the outside.

[0083] Step 202: According to the preset data processing rules, perform data format conversion on the original storage data to obtain the data to be stored for each entry.

[0084] Step 203: Store the data to be stored for each entry in the cache database.

[0085] In practical applications, the data processing rules include the rules for data format conversion. Since the data sources of the original storage data of each entry input in real time are different and the data formats are different, according to the preset data processing rules, perform data format conversion on the original storage data to obtain the data to be stored for each entry, so as to achieve the unification of data formats. During the data format conversion process, the original storage data can be cleaned, such as removing useless fields, processing missing values or abnormal values, etc., to ensure the integrity and accuracy of the original storage data. At the same time, through data format conversion, the standardization of the data to be stored can be achieved, thereby improving the consistency and comparability of the data to be stored. In addition, for the convenience of data merging, the cache library stores the data to be stored for each entry in batches according to the merge period. In this way, all the data within the merge batch can be directly obtained during merging, and the data of this batch can be directly deleted after merging, avoiding large-scale data traversal and search.

[0086] It is worth mentioning that in order to minimize the computing resources consumed by data format conversion, the cache library is generally considered to be a distributed file system such as HDFS. If there is enough memory, it can also be considered in the form of memory or memory + HDFS.

[0087] In one possible implementation, see Figure 3 As shown, after storing the data to be stored of each entry in the cache database, the data is also stored in the database before merging, including but not limited to:

[0088] Step 301: preprocessing the data to be stored of each entry in the cache database according to a preset cycle to obtain a preprocessing result of the data to be stored; wherein the preprocessing includes statistical processing and / or key value determination processing.

[0089] Step 302: Store the data to be stored and the pre-processing results of the data to be stored of each entry in the database before merging.

[0090] In practical applications, statistical processing refers to basic statistical analysis of the data to be stored, such as calculating the average, standard deviation, maximum, minimum, etc., to understand the distribution characteristics of the data to be stored. Key value determination processing is to determine the primary key or unique identifier of the data to be stored to ensure the uniqueness and traceability of the data. It may also be necessary to generate or assign additional key values ​​to the data to be stored, such as partition keys, index keys, etc., to optimize the storage and query performance of the data to be stored. Storing the data to be stored of each item and the pre-processing results of the data to be stored in the database before merging can play a role in data backup.

[0091] It is worth mentioning that the data storage periods of the database before merging, the cache database and the database after merging are different, and the data storage periods of the database before merging, the cache database and the database after merging increase in sequence.

[0092] In a possible implementation, the server may also receive a data query request input from an external source, perform a query in the merged database according to the data query request, obtain a corresponding query result, and the server feeds the query result back to an external device. When the server performs a query in the merged database according to the data query request and fails to obtain the corresponding query result, the server may also perform a query in the pre-merger database according to the data query request. The query request includes a query field and / or a query condition, the query field includes at least one of the corresponding column of the query, the data name, and other query information, the query condition may be a filtering criterion for at least one column, and the filtering criterion may be implemented by various comparison operators and logical operators. When a query is performed in the merged database and the query field included in the query request is a data name, it can be restored based on the merged part and the same part corresponding to the data name.

[0093] Based on the above embodiments, the present application provides a data storage device. Figure 4 As shown, the data storage device 400 provided in the embodiment of the present application at least includes:

[0094] A first data acquisition unit 401 is used to acquire the to-be-stored data of each entry in a merging cycle from a cache database;

[0095] A matching unit 402 is used to perform similarity matching on the data to be stored of each entry to obtain a similarity matching result;

[0096] A grouping unit 403 is used to group the data to be stored of each entry according to the similarity matching result to obtain multiple data groups to be merged;

[0097] A merging unit 404 is used to merge the to-be-merged data of all entries in the to-be-merged data group into target to-be-stored data of one entry for each to-be-merged data group, wherein the target to-be-stored data includes a common part and a merged part, wherein the common part is composed of the common part of the original to-be-stored data of all entries in the group, and the merged part is composed of the different parts of the original to-be-stored data of all entries in the group;

[0098] The storage unit 405 is used to store the target to-be-stored data of each to-be-merged data group in the merged database.

[0099] In a possible implementation manner, the matching unit 402 in the data storage device 400 provided in the embodiment of the present application is specifically used for:

[0100] Determine the similarity between the data to be stored of each entry in the merging period and the data to be stored of other entries in the merging period according to the data type and / or application scenario of the data to be stored of each entry;

[0101] The similarity between the data to be stored of each entry in the merging period and the data to be stored of other entries in the merging period is taken as the similarity matching result.

[0102] In a possible implementation manner, the merging unit 404 in the data storage device 400 provided in the embodiment of the present application is specifically used for:

[0103] For each data group to be merged, the data to be stored of all entries contained in the data group to be merged are merged into the target data to be stored of one entry, including:

[0104] cyclically performing grouping iterative merging processing on the to-be-stored data of all entries included in the to-be-merged data group until the total number of merged sub-results is less than a preset first number threshold;

[0105] Merge each merge sub-result to obtain the target data to be stored for an entry;

[0106] Among them, the group iterative merging processing includes: grouping the data to be merged according to a preset first quantity threshold to obtain multiple data groups to be merged; merging the data in each data group to be merged to obtain merged sub-results; wherein, when the group iterative merging processing is executed for the first time, the data to be merged is the data to be stored of all entries contained in the data group to be merged; when the group iterative merging processing is not executed for the first time, the data to be merged is the merged sub-results obtained by the previous execution of the group iterative merging processing.

[0107] In a possible implementation manner, the data storage device 400 provided in the embodiment of the present application further includes: an index establishment unit 406; the index establishment unit 406 is used to:

[0108] According to the target data to be stored for each entry, an index field and an index range of the target data to be stored are established;

[0109] The index fields and index ranges are associated with the target data to be stored and stored in the merged database.

[0110] In a possible implementation manner, the data storage device 400 provided in the embodiment of the present application further includes: a first data processing unit 407; the first data processing unit 407 is used to:

[0111] Receive the original storage data of each item input from the outside in real time;

[0112] According to the preset data processing rules, the data format of the original stored data is converted to obtain the data to be stored for each item;

[0113] The data to be stored of each entry is stored in the cache database.

[0114] In a possible implementation manner, the data storage device 400 provided in the embodiment of the present application further includes: a second data processing unit 408; the second data processing unit 408 is used to:

[0115] Preprocessing the data to be stored in each entry in the cache database according to a preset cycle to obtain a preprocessing result of the data to be stored; wherein the preprocessing includes statistical processing and / or key value determination processing;

[0116] The data to be stored of each entry and the pre-processing results of the data to be stored are stored in the database before merging.

[0117] It should be noted that the principle of solving the technical problem by the data storage device 400 provided in the embodiment of the present application is similar to the data storage method provided in the embodiment of the present application. Therefore, the implementation of the data storage device 400 provided in the embodiment of the present application can refer to the implementation of the data storage method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0118] After introducing the data storage method and device provided in the embodiments of the present application, the electronic device provided in the embodiments of the present application is briefly introduced next.

[0119] See also Figure 5 As shown, the electronic device 500 provided in the embodiment of the present application includes at least: a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program, the data storage method provided in the embodiment of the present application is implemented.

[0120] It should be noted that Figure 5 The electronic device 500 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0121] The electronic device 500 provided in the embodiment of the present application may further include a bus 503 connecting different components (including the processor 501 and the memory 502). The bus 503 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0122] The memory 502 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 5021 and / or a cache memory 5022 , and may further include a read-only memory (ROM) 5023 .

[0123] The memory 502 may also include a program tool 5025 having a set (at least one) of program modules 5024, wherein the program modules 5024 include but are not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each of which or some combination may include the implementation of a network environment.

[0124] The electronic device 500 may also communicate with one or more external devices 504 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 500 (e.g., mobile phones, computers, etc.), and / or communicate with any device that enables the electronic device 500 to communicate with one or more other electronic devices 500 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 505. Furthermore, the electronic device 500 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 506. Figure 5 As shown, the network adapter 506 communicates with other modules of the electronic device 500 via the bus 503. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0125] The computer-readable storage medium provided in the embodiment of the present application is introduced below. The computer-readable storage medium provided in the embodiment of the present application stores computer instructions, and when the computer instructions are executed by the processor, the data storage method provided in the embodiment of the present application is implemented. Specifically, the computer instructions can be built-in or installed in the electronic device 500, so that the electronic device 500 can implement the data storage method provided in the embodiment of the present application by executing the built-in or installed computer instructions.

[0126] In addition, the data storage method provided in the embodiment of the present application can also be implemented as a program product, which includes a program code. When the program product can be run on the electronic device 500, the program code is used to enable the electronic device 500 to execute the data storage method provided in the embodiment of the present application.

[0127] The program product provided in the embodiments of the present application may adopt any combination of one or more readable media, wherein the readable medium may be a readable signal medium or a readable storage medium, and the readable storage medium may be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0128] The program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on a computing device. However, the program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0129] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0130] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0131] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0132] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data storage method, characterized in that: include: Obtain the data to be stored for each entry in the merge cycle from the cache database; Perform similarity matching on the data to be stored of each entry to obtain a similarity matching result; The data to be stored of each entry is grouped according to the similarity matching result to obtain multiple data groups to be merged; For each data group to be merged, merge the data to be stored of all entries included in the data group to be merged into target data to be stored of one entry, wherein the target data to be stored includes a common part and a merged part, wherein the common part is composed of the common part of the original data to be stored of all entries included in the group, and the merged part is composed of the different parts of the original data to be stored of all entries included in the group; The target data to be stored of each data group to be merged is stored in the merged database.

2. The data storage method according to claim 1, characterized in that: Perform similarity matching on the data to be stored of each entry to obtain similarity matching results, including: Determine the similarity between the data to be stored of each entry in the merging period and the data to be stored of other entries in the merging period according to the data type and / or application scenario of the data to be stored of each entry; The similarity between the to-be-stored data of each entry in the merging period and the to-be-stored data of other entries in the merging period is taken as the similarity matching result.

3. The data storage method according to claim 1, characterized in that: For each data group to be merged, the data to be stored of all entries contained in the data group to be merged are merged into one entry of target data to be stored, including: cyclically performing grouping iterative merging processing on the to-be-stored data of all entries included in the to-be-merged data group until the total number of merged sub-results is less than a preset first number threshold; Merge each merge sub-result to obtain the target data to be stored for an entry; Among them, the group iterative merging processing includes: grouping the data to be merged according to a preset first quantity threshold to obtain multiple data groups to be merged; merging the data in each data group to be merged to obtain merged sub-results; wherein, when the group iterative merging processing is executed for the first time, the data to be merged is the data to be stored of all entries contained in the data group to be merged; when the group iterative merging processing is not executed for the first time, the data to be merged is the merged sub-results obtained by the previous execution of the group iterative merging processing.

4. The data storage method according to any one of claims 1 to 3, characterized in that: After merging all the data to be stored in each data group to be merged into one target data to be stored, the method further includes: According to the target data to be stored for each entry, an index field and an index range of the target data to be stored are established; The index field and the index range are associated with the target data to be stored and stored in the merged database.

5. The data storage method according to claim 4, characterized in that: Before obtaining the data to be stored for each entry in the merge cycle from the cache database, the following steps are also included: Receive the original storage data of each item input from the outside in real time; According to the preset data processing rules, the data format of the original stored data is converted to obtain the data to be stored for each item; The data to be stored of each entry is stored in the cache database.

6. The data storage method according to claim 5, characterized in that: After storing the data to be stored of each entry in the cache database, the following steps are also included: Preprocessing the data to be stored of each entry in the cache database according to a preset cycle to obtain a preprocessing result of the data to be stored; wherein the preprocessing includes statistical processing and / or key value determination processing; The data to be stored of each entry and the pre-processing results of the data to be stored are stored in the database before merging.

7. A data storage device, characterized in that: include: A first data acquisition unit, used to acquire the to-be-stored data of each entry in a merging period from the cache database; A matching unit, used for performing similarity matching on the data to be stored of each entry to obtain a similarity matching result; A grouping unit, used to group the data to be stored of each entry according to the similarity matching result to obtain multiple data groups to be merged; a merging unit, for merging, for each data group to be merged, the data to be stored of all entries included in the data group to be merged into target data to be stored of one entry, wherein the target data to be stored includes a common part and a merged part, wherein the common part is composed of the common part of the original data to be stored of all entries included in the group, and the merged part is composed of the different parts of the original data to be stored of all entries included in the group; The storage unit is used to store the target to-be-stored data of each to-be-merged data group in the merged database.

8. The data storage device according to claim 7, characterized in that The matching unit is specifically used for: Determine the similarity between the data to be stored of each entry in the merging period and the data to be stored of other entries in the merging period according to the data type and / or application scenario of the data to be stored of each entry; The similarity between the to-be-stored data of each entry in the merging period and the to-be-stored data of other entries in the merging period is taken as the similarity matching result.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data storage method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the data storage method according to any one of claims 1 to 6 is implemented.

Images