Data storage method and device, electronic device, and storage medium
By identifying the field types in the dataset and determining the storage strategy, matching storage data is generated, redundancy problems in big data storage are solved, storage efficiency and flexibility are improved, and resource consumption is reduced.
Patent Information
- Application Number
- CN202210894962.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-27
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-07-27
AI Technical Summary
Big data is stored in distributed file systems. Data bloat due to multi-copy strategy, excessive redundant information is consumed, and a large amount of storage resources is affected, which affects data utilization and system performance.
By identifying the field types in the dataset, determining the corresponding storage strategy, generating storage data matching the field types, avoiding redundant storage, and adopting directory, key-value or offset quantization storage methods.
Save storage space, improve storage space utilization of distributed file systems, reduce data deletion frequency, reduce resource consumption, and improve data storage flexibility.
Smart Images

Figure CN115269539B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data processing technology, and in particular to a data storage method and device, an electronic device, and a storage medium. Background Art
[0002] The intersection and integration of information technology and economic society has led to a rapid growth of data. Big data is increasingly having an important impact on global production, circulation, distribution, consumption activities, economic operation mechanisms, social lifestyles, and national governance capabilities. Currently, big data storage is based on distributed file systems. Distributed file systems generally use cheap servers and generally adopt a multi-copy strategy to prevent data loss and improve parallel processing capabilities.
[0003] However, with the rapid growth of data volume, the adoption of a multi-copy strategy also leads to rapid data expansion. Due to the long data bit length or too much redundant information in the data, a large amount of storage resources will be consumed when storing large-scale data. In addition, in order to ensure the normal operation of the system, historical data needs to be deleted regularly. The large consumption of storage resources shortens the data retention cycle and affects the utilization of data, thereby affecting the performance of the distributed file system. Summary of the invention
[0004] To solve the above technical problems, the embodiments of the present application provide a data storage method and device, an electronic device, a computer-readable storage medium, and a computer program product.
[0005] According to one aspect of an embodiment of the present application, a data storage method is provided, comprising: selecting data to be processed from a data set, and identifying multiple field data contained in the data to be processed to obtain a field type corresponding to each field data; determining a storage strategy corresponding to each field data according to the field type corresponding to each field data; generating storage data matching the field type of each field data according to the storage strategy corresponding to each field data; and storing the storage data corresponding to each field data.
[0006] According to one aspect of an embodiment of the present application, a data storage device includes: an identification unit, which is used to select data to be processed from a data set, and identify multiple field data contained in the data to be processed to obtain a field type corresponding to each field data; a matching unit, which is used to determine the storage strategy corresponding to each field data according to the field type corresponding to each field data; a processing unit, which is used to generate storage data that matches the field type of each field data according to the storage strategy corresponding to each field data; and an execution unit, which is used to store the storage data corresponding to each field data.
[0007] In another exemplary embodiment, the storage strategy corresponding to each field data is determined based on the field type corresponding to each field data, including: if the field data corresponds to a non-special character field type, determining that the field data corresponds to a first storage strategy; if the field data corresponds to a special character field type, a time field type, or an overlong field type used to characterize that the field length exceeds a preset length threshold, determining that the field data corresponds to a second storage strategy; wherein the first storage strategy is a strategy that does not require conversion of the field data, and the second storage strategy is a strategy that requires conversion of the field data.
[0008] In another exemplary embodiment, determining the storage strategy corresponding to each field data according to the field type corresponding to each field data includes: obtaining the field cardinality corresponding to each field data; respectively detecting the relationship between the field cardinality corresponding to each field data and a preset cardinality threshold to obtain the detection result corresponding to each field data; determining the storage strategy corresponding to each field data according to the detection result corresponding to each field data and the field type corresponding to each field data.
[0009] In another exemplary embodiment, the storage strategy corresponding to each field data is determined based on the detection result corresponding to each field data and the field type corresponding to each field data, including: if the field cardinality represented by the detection result corresponding to the field data is less than or equal to the preset cardinality threshold, and the field data is a non-special character field type, then the field data is determined to correspond to a first storage strategy; if the field cardinality represented by the detection result corresponding to the field data is less than or equal to the preset cardinality threshold, and the field data is a special character field type or an overlong field type used to represent a field length exceeding a preset length threshold, then the field data is determined to correspond to a second storage strategy; if the field cardinality represented by the detection result corresponding to the field data is less than, equal to or greater than the preset cardinality threshold, and the field data is a time field type, then the field data is determined to correspond to a second storage strategy; wherein the first storage strategy is a strategy that does not require conversion of the field data, and the second storage strategy is a strategy that requires conversion of the field data.
[0010] In another exemplary embodiment, the storage strategy includes a first storage strategy that does not require conversion of field data, or a second storage strategy that requires conversion of field data; the storage data matching the field type of each field data is generated according to the storage strategy corresponding to each field data, including: if the field data corresponds to the first storage strategy, and the field data is a non-special character field type, the field content contained in the field data is used as the storage data matching the field data of the non-special character field type; if the field data corresponds to the second storage strategy, and the field data is a special character field type or an extra-long field type used to characterize that the field length exceeds a preset length threshold, the field content contained in the field data is converted into content in the form of a key-value pair, and the key element in the key-value pair content is used as the storage data matching the field data of the special character field type or the extra-long field type; if the field data corresponds to the second storage strategy, and the field data is a time field type, the offset time is calculated based on the preset base time and the target time contained in the field data, and the offset time is used as the storage data matching the field data of the time field type.
[0011] In another exemplary embodiment, storing the storage data corresponding to the various field data includes: using the storage data corresponding to the various field data as a directory name; storing the directory name in a designated directory; wherein the designated directory is obtained by dividing according to designated type data.
[0012] In another exemplary embodiment, the method of selecting the data to be processed from the data set includes: sorting the data contained in the data set in ascending order according to the field cardinality to obtain a data sequence; iteratively calculating the data in sequence according to the order of the data sequence to obtain a result value; comparing the result value with a storage value of a storage space, and if the result value is less than the storage value, selecting the data from the data set whose result value is greater than or equal to the storage value as the data to be processed.
[0013] According to one aspect of an embodiment of the present application, an electronic device includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the data storage method as described above.
[0014] According to one aspect of an embodiment of the present application, a computer-readable storage medium stores computer-readable instructions thereon. When the computer-readable instructions are executed by a processor of a computer, the computer executes the data storage method as described above.
[0015] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the data storage method provided in the above various optional embodiments.
[0016] In the technical solution provided in the embodiments of the present application, by identifying the field types corresponding to the multiple field data contained in the data to be processed, and according to the field types corresponding to the various field data, determining the storage strategies corresponding to the various field data, generating storage data matching the field types of the various field data, and then storing the storage data corresponding to the multiple field data contained in the data to be processed; in this way, on the one hand, directly storing all the field data contents is avoided, the storage of redundant data is reduced, storage space is saved, and the utilization rate of the storage space of the distributed file system is improved; on the other hand, more data can be stored, which can reduce the frequency of data deletion to a certain extent, and save resource consumption such as human resource consumption or system resource consumption; on the other hand, the flexibility of data storage is improved, which is suitable for many application scenarios.
[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0019] Figure 1 It is a schematic diagram of an implementation environment involved in this application;
[0020] Figure 2 is a flow chart of a data storage method shown in an exemplary embodiment of the present application;
[0021] Figure 3 yes Figure 2 A flowchart of the step of selecting data to be processed from a data set in step S201 in the illustrated embodiment in an exemplary embodiment;
[0022] Figure 4 Yes Yes Figure 2Step S202 in the illustrated embodiment is a flowchart of determining the storage strategy corresponding to each field data by field type in an exemplary embodiment;
[0023] Figure 5 Yes Yes Figure 2 Step S202 in the illustrated embodiment is a flowchart of determining a storage strategy corresponding to each field data by configuring a preset cardinality threshold and field type in an exemplary embodiment;
[0024] Figure 6 yes Figure 2 After the illustrated embodiment determines that the storage strategy corresponding to the field data is the first storage strategy, steps S203 to S204 are a flowchart in an exemplary embodiment;
[0025] Figure 7 yes Figure 2 After the illustrated embodiment determines that the storage strategy corresponding to the field data is the second storage strategy, and the field data is a special character field type or an overlong field type, steps S203 to S204 are a flowchart in an exemplary embodiment;
[0026] Figure 8 yes Figure 2 In the illustrated embodiment, after determining that the storage strategy corresponding to the field data is the second storage strategy, and the field data is of the time field type, steps S203 to S204 are a flowchart in an exemplary embodiment;
[0027] Fig. 9 It is a structural diagram of a data storage system and a distributed file system for implementing the data storage method of the present application, shown in an exemplary embodiment of the present application;
[0028] Fig.10 is a block diagram of a data storage device shown in an exemplary embodiment of the present application;
[0029] Fig.11 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION
[0030] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.
[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0032] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0033] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.
[0034] Among related technologies, the intersection and integration of information technology and economic society has triggered a rapid growth of data. Data has become a basic strategic resource for the country. Big data is increasingly having an important impact on global production, circulation, distribution, consumption activities, economic operation mechanisms, social lifestyles, and national governance capabilities. Big data usually refers to data sets that are huge in number and difficult to collect, process, and analyze, and also refers to data that is stored for a long time in traditional infrastructure. Big data storage is the persistence of these data sets in computers. Big data storage technology is another key data in the field of big data. It usually uses distributed storage instead of centralized storage, and replaces expensive machines with cheaper machines, which greatly reduces the cost of massive storage.
[0035] Currently, big data storage is implemented based on distributed file systems, such as HDFS (Distributed File System, a distributed file system designed to run on general-purpose hardware), GFS (Google File System, a scalable distributed file system designed to implement large-scale distributed data-intensive applications), etc. Distributed file systems generally use cheap servers. In order to prevent data loss and improve parallel processing capabilities, a multi-copy strategy is generally adopted. For example, CDH (Cloudera's Distribution Including Apache Hadoop, a data center management tool) generally defaults to 3 copies, that is, replication = 3. With the explosive growth of data scale, the storage demand based on the multi-copy strategy has also exploded. Storage is a bottomless pit of investment, and the quality of storage capacity directly affects the development of big data related businesses.
[0036] With the rapid growth of data volume, the use of a multi-copy strategy also leads to rapid data expansion. Due to the long data bit length or too much redundant information in the data, such as data containing a large amount of time-related information, a large amount of storage resources are required when storing large-scale data due to the excessive redundant information in each piece of data. At the same time, in order to ensure the normal operation of the distributed file system, historical data needs to be deleted regularly. Therefore, the large consumption of storage resources will shorten the data retention cycle, affect the utilization rate of data in scenarios where historical data is used for mining and analysis or reprocessing, and thus affect the performance of the distributed file system.
[0037] The embodiments of the present application propose a data storage method and device, an electronic device, and a computer-readable storage medium, which mainly relate to big data storage technology. These embodiments will be described in detail below.
[0038] First see Figure 1 , Figure 1 1 is a schematic diagram of an implementation environment involved in the present application. The implementation environment includes a distributed file system 10 and a server 20, and the distributed file system 10 and the server 20 communicate with each other through a wired or wireless network.
[0039] The server 20 is used to select the data to be processed from the data set, and identify multiple field data contained in the data to be processed, obtain the field type corresponding to each field data, and then determine the storage strategy corresponding to each field data according to the field type corresponding to each field data, and finally generate storage data matching the field type of each field data according to the storage strategy corresponding to each field data, and store the storage data corresponding to each field data in the distributed file system 10. Compared with the data storage scheme of the prior art, the data storage method provided by the present implementation environment can reduce the storage of redundant data to save storage space, thereby improving the utilization rate of the storage space of the distributed file system.
[0040] It should be noted that server 20 can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. There is no restriction here.
[0041] Figure 2 is a flowchart of a data storage method shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown in Figure 1 The server 20 in the embodiment shown is specifically implemented. In other implementation environments, the method may be implemented by devices in other implementation environments, and this embodiment does not limit this.
[0042] like Figure 2 As shown, in an exemplary embodiment, the data storage method may include steps S201 to S204, which are described in detail as follows:
[0043] Step S201 : selecting data to be processed from a data set, and identifying a plurality of field data contained in the data to be processed to obtain a field type corresponding to each field data.
[0044] In this embodiment of the present application, an initial data set is first obtained from a distributed file system, including various field data stored in a traditional storage method of directly storing data content on the distributed file system. After obtaining the data set, the data to be processed is selected therefrom and the data storage method provided by the present application is applied to process the data to be processed. That is, after obtaining the data to be processed, the various field data contained in the data to be processed are identified, and the field type corresponding to each field data is identified.
[0045] Step S202: determining a storage strategy corresponding to each field data according to the field type corresponding to each field data.
[0046] This application presets multiple storage strategies, and there are different storage strategies for field data of different field types. Therefore, after identifying the field type corresponding to each field data, it is possible to determine the storage strategy corresponding to each field data contained in the data to be processed according to the correspondence between the preset field type and the storage strategy, so as to store each field data in a manner corresponding to each storage strategy.
[0047] Step S203: generating storage data matching the field type of each field data according to the storage strategy corresponding to each field data.
[0048] After determining the storage strategy corresponding to each field data, storage data matching the field type is generated based on the field data according to different strategy modes of various storage strategies and the field type of the field data for storage.
[0049] Step S204, storing the storage data corresponding to each field data.
[0050] After obtaining the storage data according to the storage strategy corresponding to each field data, it is only necessary to store the storage data, thereby avoiding the situation where the complete storage of the field data content occupies storage resources.
[0051] As can be seen from the above, in the method provided in this embodiment, after selecting the data to be processed from the data set, the field types corresponding to the multiple field data contained in the data to be processed are identified, and according to the field types corresponding to the various field data, the storage strategies corresponding to the various field data are determined, and storage data matching the field types of the various field data are generated, and then the storage data corresponding to the multiple field data contained in the data to be processed are stored, avoiding the direct storage of all the field data content, thus reducing the storage of redundant data to save storage space, thereby improving the retrieval efficiency of the distributed file system and the utilization of storage space. On the other hand, more data can be stored, which can reduce the frequency of data deletion to a certain extent, not only improving the utilization of historical data, but also saving resource consumption such as human resource consumption or system resource consumption; on the other hand, the flexibility of data storage is improved, which is suitable for many application scenarios.
[0052] See also Figure 3 , Figure 3 yes Figure 2 The flowchart of selecting the data to be processed from the data set in step S201 in the embodiment shown in an exemplary embodiment. Figure 3 As shown, step S201 may include steps S301 to S303, and the data to be processed are selected from the data set through the above steps, which are described in detail as follows:
[0053] Step S301, sorting the data contained in the data set in ascending order of field cardinality to obtain a data sequence.
[0054] It should be noted that, in this embodiment, after obtaining the initial data set from the distributed file system, the block file size of the applied distributed file system is first read (for example, in CDH, the block file size is 256M, and the block file size of HDFS usually defaults to 128M), and the block file size value is used as the storage value of the storage space, and the preset cardinality threshold, preset length threshold, time field number threshold, preset base time and empirical coefficient (greater than 1) applied to the storage method provided in this application are also configured.
[0055] In addition, statistical analysis or estimation will be performed on each field data in the data set. The analysis results are obtained by statistical analysis of old data and estimation of new data in combination with their lineage (based on data traceability). The analysis results mainly include the field distinction, field length, field group (inclusion relationship) between fields, field type, field cardinality, and average daily file size (total size of pre-processed data storage) of each data. Field types can include non-special character field types, special character field types, time field types, and overlong field types. Among them, the time field type represents field data whose time field number exceeds the time field number threshold, and the overlong field type represents field data whose field length exceeds the preset length threshold.
[0056] Then, the field cardinality of each data contained in the data set is obtained from the analysis results of the data set, and each data is sorted in the order of the field cardinality value from small to large to obtain the data sequence of the data set.
[0057] In addition, in another embodiment of the present application, when configuring the preset cardinality threshold, the preset length threshold, and the empirical coefficient, a mandatory mark (such as a field required by the division of authority and domain in management) will also be configured. If the data set contains data corresponding to the mandatory mark, the data will be placed at the first position of the data sequence as mandatory data, taking precedence over other data in the data set that does not correspond to the mandatory mark. The field type or field content corresponding to the mandatory field mark can be set according to the requirements, and the present application is not limited to a specific field data.
[0058] Step S302, iteratively calculate each data in sequence according to the order of the data sequence to obtain a result value.
[0059] After the data sequence is obtained, each data is iteratively calculated in turn according to the order of the data sequence to obtain the result value. According to the result value, the size of the block file that may be generated when the data involved in the calculation is stored can be analyzed.
[0060] For example, the iterative calculation of the data at the first position of the data sequence can be expressed as: daily average file size / field cardinality n*empirical coefficient, where field cardinality n represents the field cardinality of the data at the first position of the data sequence. Correspondingly, the result value of the iterative calculation of the data sequence is calculated as: result value = block file size = product of daily average file size / field cardinality * empirical coefficient, where the product of field cardinality represents the product of the field cardinality of each data selected for iterative calculation in order. An empirical coefficient greater than 1 can avoid the generation of small files.
[0061] Step S303, comparing the result value with the storage value of the storage space, if the result value is less than the storage value, then selecting each data with the result value greater than or equal to the storage value from the data set as the data to be processed.
[0062] In this embodiment, each time a result value is calculated, the result value is compared with the storage value of the read storage space. If the result value is smaller than the storage value, it means that the comparison result between the result value and the storage value meets the critical state of the block file size: the block file size is smaller than the block file size of the distributed file system, and the block file size corresponding to the selected data to be processed needs to be greater than or equal to the block file size of the distributed file system, so as to avoid the generation of small files and increase the system data redundancy. Then the last data of the result value obtained by the iterative calculation does not meet this condition.
[0063] Therefore, when the comparison result value is less than the stored value, the iterative calculation of each data in the data sequence is stopped, and each data whose result value has been iteratively calculated and whose result value is greater than or equal to the stored value is selected from the data set as the data to be processed.
[0064] In another embodiment, when each data in the data sequence is iteratively calculated according to the sorting order, if when iterating to a data, it is identified that there is a combination relationship between it and the data of the previous iteration to form a data combination, then when calculating, only the field with the largest field cardinality value in the data combination needs to be iterated. For example, the data in a data combination respectively correspond to the province code, the city code, and the district and county code, and only the cardinality of the district and county code can be iterated first. If the calculation result value satisfies the condition that the result value is greater than or equal to the stored value, all the data in the data combination are selected as the data to be processed; if not, the field with the second largest field cardinality value in the data combination is iteratively calculated, and so on, until the result value satisfies the condition that the result value is greater than or equal to the stored value, or the minimum field cardinality value is still not satisfied, then the iterative calculation is stopped and all the data in the data combination are not used as the data to be processed.
[0065] This embodiment sorts each data in the data set according to the field cardinality and then performs iterative calculations, and selects the data to be processed and applies it to the data storage method provided in this application. This not only greatly reduces the storage of redundant data to save storage space, but also does not increase the operating cost of the server because all data in the data set are processed using the data storage method provided in this application. While ensuring the system operating performance and cost, the utilization rate of the distributed file system storage space is improved.
[0066] See also Figure 4 , Figure 4 yes Figure 2 The step S202 in the embodiment shown is a flow chart in an exemplary embodiment. Figure 4 As shown, step S202 may include step S401 to step S402. Through the above steps, the storage strategy corresponding to each field data is determined according to the field type, which is described in detail as follows:
[0067] Step S401: If the field data corresponds to a non-special character field type, it is determined that the field data corresponds to a first storage strategy.
[0068] When performing statistical analysis on each data in a data set, the field types obtained in the analysis results may include non-special character field types, special character field types, time field types, and extra-long field types, where the extra-long field type is characterized by field data whose field length exceeds the preset length threshold of the implementation configuration.
[0069] If the field type of the field data is identified as a non-special character field type, then compared to the special character field type, time field type and extra-long field type, the non-special character field type does not have redundant special characters, more time data information or extra-long field length. Therefore, the present application corresponds the field data of the non-special character field type to the first storage strategy that does not require conversion of the field data.
[0070] Step S402: If the field data corresponds to a special character field type, a time field type, or an overlong field type used to indicate that the field length exceeds a preset length threshold, it is determined that the field data corresponds to a second storage strategy.
[0071] Special characters, time information and over-length systems will increase the redundancy of data according to their length. Therefore, for field data of special character field types, time field types or over-long field types used to characterize field lengths exceeding a preset length threshold, it is determined that the corresponding second storage strategy is a storage strategy that requires conversion of the field data. The second storage strategy converts time field data into storage data with low redundancy and corresponding to the field data.
[0072] The above embodiment determines the storage strategy corresponding to each field data contained in the data to be processed by the field type. In another exemplary embodiment of the present application, the storage strategy corresponding to each field data can also be determined by configuring a preset cardinality threshold and field type, such as Figure 5 As shown, Figure 5 yes Figure 2 The flowchart of step S202 in the illustrated embodiment in an exemplary embodiment may include steps S501 to S504, which are described in detail as follows:
[0073] Step S501, obtaining the field cardinality corresponding to each field data.
[0074] Step S502 , respectively detecting the relationship between the field cardinality corresponding to each field data and the preset cardinality threshold, and obtaining the detection result corresponding to each field data.
[0075] From the analysis results obtained by statistically analyzing each data in the data set, the field cardinality corresponding to each field data contained in the data to be processed is obtained, and the pre-configured preset cardinality threshold is obtained. Then, the relationship between the field cardinality corresponding to each field data and the preset cardinality threshold is detected to obtain the detection result corresponding to each field data. The detection result can be characterized as a situation where the field cardinality is less than the preset cardinality threshold, or a situation where the field cardinality is greater than or equal to the preset cardinality threshold.
[0076] Step S503: if the detection result corresponding to the field data indicates that the field cardinality is less than or equal to a preset cardinality threshold, and the field data is a non-special character field type, it is determined that the field data corresponds to the first storage strategy.
[0077] The smaller the field cardinality, the smaller the field distinction, and the redundancy will be relatively low. Therefore, in this embodiment, by configuring a preset cardinality threshold, field data with a field cardinality less than or equal to the preset cardinality threshold and a non-special character field type is stored using a first storage strategy that does not require conversion of the field data, thereby not increasing the redundancy of the stored data and improving storage efficiency.
[0078] Step S504, if the field data corresponds to the detection result indicating that the field cardinality is less than or equal to the preset cardinality threshold, and the field data is a special character field type or an overlong field type used to indicate that the field length exceeds the preset length threshold, it is determined that the field data corresponds to the second storage strategy.
[0079] The field content of the field data of the special character field type includes special characters, which is used to represent an overlong field type whose field length exceeds a preset length threshold, etc. The field content of the corresponding field data may include special characters or are field types that will make the field data highly redundant.
[0080] Based on the judgment of the field data corresponding to the first storage strategy, if the detection result represents that the field cardinality is greater than the preset cardinality threshold, it means that the redundancy of the field data will be relatively high, and the field data of the special character field type and the over-long field type itself has a high data redundancy, so there is no need to convert this type of data into storage data for storage. Therefore, the field data whose detection result represents that the field cardinality is less than or equal to the preset cardinality threshold and whose field type is the special character field type or the over-long field type is determined to correspond to the second storage strategy that requires the field data to be converted.
[0081] Step S505: if the field data corresponds to a detection result indicating that the field cardinality is less than or equal to or greater than a preset cardinality threshold, and the field data is a time field type, it is determined that the field data corresponds to a second storage strategy.
[0082] The field data of the time field type contains field content that exceeds the time field number threshold. Regardless of whether the field cardinality of the field data of the time field type is less than or equal to the preset cardinality threshold or greater than the preset cardinality threshold, the field data of the time field type with more date fields needs to be converted into storage data for storage. Therefore, the field data whose field cardinality is less than or equal to or greater than the preset cardinality threshold and whose field type is the time field type represented by the detection result is determined to correspond to the second storage strategy that requires the conversion of the field data.
[0083] As can be seen from the above, in the method provided in this embodiment, the storage strategy corresponding to each field data contained in the data to be processed is determined by the field type, or the storage strategy corresponding to each field data is determined by the configured preset cardinality threshold and field type, and each field data contained in the data to be processed that does not need to be converted corresponds to the first storage strategy, and the field data that needs to be converted corresponds to the second storage strategy. In this way, when determining the storage strategy corresponding to each field data contained in the data to be processed, the field data is first divided according to the field type and field cardinality related to the redundancy of the field, and then the storage data corresponding to the field data is stored through matching different storage strategies, which not only further reduces the storage of redundant data and saves storage space, but also ensures that the data storage method provided by the present application does not increase the operating cost too much for the data sorting, processing and re-storage of the distributed file system, and reduces the impact on the system operation performance.
[0084] In the storage method provided in this application, the storage method corresponding to the first storage strategy can be directory storage, the storage method corresponding to the second storage strategy can be key-value storage, and the storage method corresponding to the second storage strategy can also be offset quantization storage. If it is determined that the storage strategy corresponding to the field data is the first storage strategy, such as Figure 6 As shown, Figure 6 yes Figure 2 In an exemplary embodiment, steps S203 to S204 of the illustrated embodiment may be steps S601 to S604, which are described in detail as follows:
[0085] Step S601: if the field data is a non-special character field type, the field content contained in the field data is used as storage data matching the field data of the non-special character field type.
[0086] When it is determined that the storage strategy corresponding to the field data is the first storage strategy, and the field type is a non-special character field type, the field data can be processed before storage. The method provided by the present application is to generate matching storage data according to the storage strategy corresponding to the field data. Under the preset first storage strategy, the storage data is the field content of the field data that naturally matches the field data, that is, the field content of the field data is obtained, and the field content is used as the storage data for storage.
[0087] Step S602: Use the storage data corresponding to each field data as a directory name, and store the directory name in a designated directory.
[0088] In this embodiment, the designated directory is obtained by partitioning according to the designated type of data, such as a partition directory divided by time. The first storage strategy applies directory storage, and the storage data of the field content is used as the name of the directory. The directory name formed by each storage data is stored in the partition directory of the Hive table using a multi-level directory storage method. The partition standard is to divide the designated type of data of the designated directory, which can be based on days or hours, and there is no restriction here. After obtaining the partition directory storing the storage data, the obtained partition directory is stored in the distributed file system to complete the data storage, thereby reducing data redundancy and saving storage space.
[0089] In another embodiment of the present application, in the case of a non-Hive warehouse, the directory structure can be converted into a file for naming, that is, the field content is used as the file name, and the file name is used as the storage data for storage, thereby obtaining multiple storage files of the field data corresponding to the first storage strategy, and the obtained storage files are stored in the distributed file system to complete the data storage.
[0090] If it is determined that the storage strategy corresponding to the field data is the second storage strategy, and the field data is a special character field type or an overlong field type used to indicate that the field length exceeds the preset length threshold, such as Figure 7 As shown, Figure 7 yes Figure 2 In an exemplary embodiment, steps S203 to S204 of the illustrated embodiment may be steps S701 to S702, which are described in detail as follows:
[0091] Step S701: convert the field content contained in the field data into content in the form of key-value pairs.
[0092] The key-value pair is a combination form used when storing in a database, and the key in the key-value pair is followed by the corresponding value. In this embodiment of the present application, when it is determined that the storage strategy corresponding to the field data is the second storage strategy, and the field type is a special character field type or an overlong field type used to characterize that the field length exceeds a preset length threshold, the field content contained in the field data is converted into content in the form of a key-value pair, wherein the field content of the field data is used as the value element of the key-value pair, and a key element is generated corresponding to the field content.
[0093] For example, the conversion of the field content contained in the field data into the content in the form of key-value pairs can be achieved by dictionaryizing the field data. Specifically, a data dictionary is constructed to write the field content of the field data into the data dictionary, and a dictionary code corresponding to the field data is generated. The dictionary code is used as the key element, and the field content corresponding to the dictionary code is used as the value element to form a key-value pair. If there is a new field data corresponding to the second storage strategy, a new dictionary code is added to correspond to it. The dictionary encoding rule is configurable (for example, it is determined in combination with the field cardinality size, with the principle of being uniquely distinguishable and having the shortest length).
[0094] Step S702: Use the key element in the key-value pair content as storage data that matches the field data of the special character field type or the overlong field type used to represent that the field length exceeds a preset length threshold, and store the storage data.
[0095] After obtaining the content in the form of key-value pairs through the above steps, the key element in the key-value pair is used as the storage data matching the field data corresponding to the first storage strategy, and the key element as the storage data is stored in the distributed file system to complete the data storage. In this way, when the field data is needed later, the stored key element can be mapped to its real field content, thereby reducing the redundancy of data storage.
[0096] In addition, in another exemplary embodiment of the present application, Figure 7 In steps S701 to S702 of the illustrated embodiment, after converting the field content contained in the field data into content in the form of key-value pairs and obtaining the corresponding storage data as key elements, the embodiment can further store the key elements and the field data in the form of key-value pairs through the directory storage method in the first storage strategy, use the storage data as key elements as directory names, and store the directory names in the designated directory. In this way, the storage of redundant data is further reduced, and storage space is saved.
[0097] If it is determined that the storage strategy corresponding to the field data is the second storage strategy, and the field data is a time field type, such as Figure 8 As shown, Figure 8 yes Figure 2 In an exemplary embodiment, steps S203 to S204 of the illustrated embodiment may be steps S801 to S802, which are described in detail as follows:
[0098] Step S801, calculating the offset time according to the preset reference time and the target time contained in the field data.
[0099] After determining that the storage strategy corresponding to the field data is the second storage strategy and the field type is a time field type, the field data can be processed before storage. First, the target time contained in the field data and the preset reference time for the configuration are obtained, and then the offset time between the target time and the preset reference time is calculated. Among them, the configuration of the preset reference time can be based on the finest granularity time in the partition, or it can be set according to the needs, and no specific limitation is made here.
[0100] Step S802: Use the offset time as storage data that matches the field data of the time field type, and store the storage data.
[0101] After the offset time is obtained, it is used as storage data that matches the field data of the time field type, and the storage data of the offset time is stored in the distributed file system to complete the data storage to reduce data redundancy.
[0102] It can be seen from this that the technical solution of the above embodiment of the present application, after determining the storage strategy corresponding to each field data, generates storage data as field content, key elements of key-value pairs or offset time through the storage methods corresponding to each storage strategy, and then stores the obtained storage data into the distributed file system. In this way, by storing the storage data corresponding to multiple field data contained in the data to be processed, the storage of redundant data is reduced to save storage space, thereby improving the utilization rate of the storage space of the distributed file system.
[0103] See also Fig. 9 In an exemplary embodiment, a data storage system 910 is deployed, such as Fig. 9 The data storage system 910 shown is connected to the distributed file system 920, and the data storage method provided in the present application is implemented through the data storage system 910 and the distributed file system 920.
[0104] The data storage system 910 includes a data reading module 911, a statistics module 912, a configuration module 913, a selection module 914, a conversion module 915, and a storage processing module 916. The data reading module 911 obtains old data from the distributed file system 920, obtains new data from a data source 930 deployed outside the distributed file system 920, forms a data set, and reads the block file size of the distributed file system 920; in addition, the data reading module 911 also obtains the statistical module 912 to perform statistical analysis or estimation on each field data in the data set to obtain the analysis result, as well as the configuration information such as the preset cardinality threshold, preset length threshold, time field number threshold, preset reference time, and experience coefficient configured by the configuration module 913.
[0105] The data reading module 911 transmits the acquired data set, analysis results, configuration information, etc. to the selection module 914. The selection module 914 sorts the data contained in the data set in ascending order according to the field cardinality to obtain a data sequence, and iteratively calculates each data in turn according to the order of the data sequence to obtain a result value. Finally, based on the comparison result between the result value and the block file size of the distributed file system 920, each data whose result value is greater than or equal to the block file size, i.e., the storage value, is selected from the data set as the data to be processed.
[0106] The selection module 914 will also determine the storage strategies corresponding to the multiple field data included in the data to be processed. If it is determined from the data to be processed that the field data corresponds to the second storage strategy and is a special character field type or an extra-long field type, then after the determination, the selection module 914 will transmit the field to the data conversion module 915, and the conversion module 915 will convert the field content contained in the field data into a key-value pair format, and use the key element in the key-value pair format as the matching storage data.
[0107] If the selection module 914 determines from the data to be processed that the field data corresponds to the first storage strategy and is a non-special character field type, or determines from the data to be processed that the field data corresponds to the second storage strategy and is a time field type, then after determination, the selection module 914 will transfer the field to the warehousing processing module 916, and the warehousing processing module 916 will use the field content contained in the field data as storage data that matches the field data of the non-special character field type; calculate the offset time based on the preset base time and the target time contained in the field data, and use the offset time as storage data that matches the time field type.
[0108] The selection module 914 transmits the data to be processed and the storage data generated by each corresponding storage strategy to the storage processing module 916 , and the storage processing module 916 stores the storage data corresponding to each field data into the distributed file system 920 .
[0109] Thus, in this embodiment, the data storage method provided by the present application is implemented through the data storage system 910 and the distributed file system 920, thereby avoiding the direct storage of all field data contents. In this way, the storage of redundant data is reduced to save storage space, thereby improving the utilization rate of the distributed file system storage space; on the other hand, more data can be stored, which can reduce the frequency of data deletion to a certain extent, saving resource consumption such as human resource consumption or system resource consumption; on the other hand, the flexibility of data storage is improved, which is suitable for many application scenarios.
[0110] Fig.10 FIG. 1 is a block diagram of a data storage device 1000 according to an exemplary embodiment of the present application. Fig.10 As shown, the device comprises:
[0111] The identification unit 1001 is used to select the data to be processed from the data set, and identify multiple field data contained in the data to be processed to obtain the field type corresponding to each field data; the matching unit 1002 is used to determine the storage strategy corresponding to each field data according to the field type corresponding to each field data; the processing unit 1003 is used to generate storage data matching the field type of each field data according to the storage strategy corresponding to each field data; the execution unit 1004 is used to generate storage data matching the field type of each field data according to the storage strategy corresponding to each field data.
[0112] The device applies the data storage method provided by the present application, identifies the field types corresponding to multiple field data contained in the data to be processed through the identification unit 1001, determines the storage strategy corresponding to each field data according to the field type corresponding to each field data, and the processing unit 1003 generates storage data matching the field type of each field data. Then, the storage data corresponding to the multiple field data contained in the data to be processed is stored through the execution unit 1004, avoiding the direct storage of all the field data content, thereby reducing the storage of redundant data, saving storage space, and thus improving the utilization rate of the storage space of the distributed file system.
[0113] In another exemplary embodiment, the device also includes a selection unit, which is used to sort the data contained in the data set in order from small to large according to the field cardinality to obtain a data sequence; iteratively calculate each data in turn according to the order of the data sequence to obtain a result value; compare the result value with the storage value of the storage space, and if the result value is less than the storage value, select each data whose result value is greater than or equal to the storage value from the data set as the data to be processed.
[0114] In another exemplary embodiment, the matching unit 1002 is further used to determine that the field data corresponds to a first storage strategy if the field data corresponds to a non-special character field type; if the field data corresponds to a special character field type, a time field type, or an overlong field type used to characterize that the field length exceeds a preset length threshold, determine that the field data corresponds to a second storage strategy; wherein the first storage strategy is a strategy that does not require conversion of the field data, and the second storage strategy is a strategy that requires conversion of the field data;
[0115] In another exemplary embodiment, the matching unit 1002 is also used to determine that the field data corresponds to a first storage strategy if the field cardinality represented by the field corresponding to the detection result is less than or equal to a preset cardinality threshold, and the field data is a non-special character field type; if the field cardinality represented by the field corresponding to the detection result is less than or equal to the preset cardinality threshold, and the field data is a special character field type or an overlong field type used to represent a field length exceeding a preset length threshold, then determine that the field data corresponds to a second storage strategy; if the field cardinality represented by the field corresponding to the detection result is less than or equal to or greater than the preset cardinality threshold, and the field data is a time field type, then determine that the field data corresponds to a second storage strategy; wherein the first storage strategy is a strategy that does not require conversion of the field data, and the second storage strategy is a strategy that requires conversion of the field data.
[0116] In another exemplary embodiment, the processing unit 1003 is also used to, if the field data corresponds to the first storage strategy and the field data is a non-special character field type, use the field content contained in the field data as storage data that matches the field data of the non-special character field type; if the field data corresponds to the second storage strategy and the field data is a special character field type or an extra-long field type used to characterize that the field length exceeds a preset length threshold, convert the field content contained in the field data into content in the form of a key-value pair, and use the key element in the content in the key-value pair as storage data that matches the field data of the special character field type or the extra-long field type; if the field data corresponds to the second storage strategy and the field data is a time field type, calculate the offset time based on the preset base time and the target time contained in the field data, and use the offset time as storage data that matches the field data of the time field type.
[0117] In another exemplary embodiment, the execution unit 1004 is further used to use the storage data corresponding to each field data as a directory name; store the directory name in a specified directory; wherein the specified directory is obtained by dividing according to the specified type of data.
[0118] It should be noted that the data storage device provided in the above embodiment and the data storage method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the data storage device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.
[0119] An embodiment of the present application also provides an electronic device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the electronic device implements the data storage method provided in the above-mentioned embodiments.
[0120] Fig.11 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Fig.11 The computer system 1100 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0121] like Fig.11 As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1108 to the random access memory (RAM) 1103, such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in the RAM 1103. The CPU 1101, ROM 1102 and RAM 1103 are connected to each other through a bus 1104. The input / output (I / O) interface 1105 is also connected to the bus 1104.
[0122] The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed, so that a computer program read therefrom is installed into the storage section 1108 as needed.
[0123] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1109, and / or installed from a removable medium 1111. When the computer program is executed by a central processing unit (CPU) 1101, various functions defined in the system of the present application are executed.
[0124] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (Erasable Programmable Read Only Memory, EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0125] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0126] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0127] Another aspect of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the data storage method as described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.
[0128] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the data storage method provided in each of the above embodiments.
[0129] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A data storage method, characterized in that: include: Selecting data to be processed from the data set, and identifying multiple field data contained in the data to be processed to obtain the field type corresponding to each field data; Determine, according to the field type corresponding to each field data, a storage strategy corresponding to each field data, wherein the storage strategy includes a first storage strategy that does not require conversion of the field data, or a second storage strategy that requires conversion of the field data; If the field data corresponds to the first storage strategy, and the field data is a non-special character field type, the field content contained in the field data is used as storage data matching the field data of the non-special character field type; If the field data corresponds to the second storage strategy, and the field data is a special character field type or an overlong field type used to represent that the field length exceeds a preset length threshold, the field content contained in the field data is converted into content in the form of a key-value pair, and the key element in the content in the form of the key-value pair is used as the storage data matching the field data of the special character field type or the overlong field type; If the field data corresponds to the second storage strategy and the field data is of a time field type, an offset time is calculated according to a preset reference time and a target time contained in the field data, and the offset time is used as storage data matching the field data of the time field type; The storage data corresponding to each field data is stored.
2. The method according to claim 1, characterized in that The step of determining the storage strategy corresponding to each field data according to the field type corresponding to each field data includes: If the field data corresponds to a non-special character field type, determining that the field data corresponds to a first storage strategy; If the field data corresponds to a special character field type, a time field type, or an overlong field type used to characterize a field length that exceeds a preset length threshold, it is determined that the field data corresponds to a second storage strategy; wherein the first storage strategy is a strategy that does not require conversion of the field data, and the second storage strategy is a strategy that requires conversion of the field data.
3. The method according to claim 1, characterized in that The step of determining the storage strategy corresponding to each field data according to the field type corresponding to each field data includes: Obtain the field cardinality corresponding to each field data; Relationships between field cardinalities corresponding to the respective field data and preset cardinality thresholds are detected respectively, to obtain detection results corresponding to the respective field data; The storage strategy corresponding to each field data is determined according to the detection result corresponding to each field data and the field type corresponding to each field data.
4. The method according to claim 3, characterized in that The step of determining the storage strategy corresponding to each field data according to the detection result corresponding to each field data and the field type corresponding to each field data includes: If the field data corresponds to a detection result representing a field cardinality that is less than or equal to the preset cardinality threshold, and the field data is a non-special character field type, determining that the field data corresponds to a first storage strategy; If the field cardinality represented by the detection result corresponding to the field data is less than or equal to the preset cardinality threshold, and the field data is a special character field type or an overlong field type used to represent that the field length exceeds the preset length threshold, it is determined that the field data corresponds to the second storage strategy; If the field data corresponding to the detection result represents a field cardinality that is less than, equal to or greater than the preset cardinality threshold, and the field data is a time field type, it is determined that the field data corresponds to a second storage strategy; wherein the first storage strategy is a strategy that does not require conversion of the field data, and the second storage strategy is a strategy that requires conversion of the field data.
5. The method according to any one of claims 1 to 4, characterized in that The storing of the storage data corresponding to each field data includes: The storage data corresponding to each field data is used as the directory name; The directory name is stored in a designated directory; wherein the designated directory is obtained by dividing data according to designated types.
6. The method according to any one of claims 1 to 4, characterized in that The step of selecting the data to be processed from the data set includes: Sort the data contained in the data set in ascending order according to the cardinality of the field to obtain a data sequence; Iteratively calculating each of the data in sequence according to the order of the data sequence to obtain a result value; The result value is compared with the storage value of the storage space. If the result value is less than the storage value, each data whose result value is greater than or equal to the storage value is selected from the data set as the data to be processed.
7. A data storage device, characterized in that: include: An identification unit, used for selecting data to be processed from the data set, and identifying a plurality of field data contained in the data to be processed to obtain a field type corresponding to each field data; A matching unit, configured to determine a storage strategy corresponding to each field data according to a field type corresponding to each field data, wherein the storage strategy includes a first storage strategy that does not require conversion of the field data, or a second storage strategy that requires conversion of the field data; a processing unit, configured to, if the field data corresponds to the first storage strategy and the field data is a non-special character field type, use the field content contained in the field data as storage data matching the field data of the non-special character field type; If the field data corresponds to the second storage strategy, and the field data is a special character field type or an overlong field type used to represent that the field length exceeds a preset length threshold, the field content contained in the field data is converted into content in the form of a key-value pair, and the key element in the content in the form of the key-value pair is used as the storage data matching the field data of the special character field type or the overlong field type; If the field data corresponds to the second storage strategy and the field data is of a time field type, an offset time is calculated according to a preset reference time and a target time contained in the field data, and the offset time is used as storage data matching the field data of the time field type; The execution unit is used to store the storage data corresponding to each field data.
8. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the electronic device to implement the data storage method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the data storage method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Data processing method, device, equipment and medium
CN111639938A
Data processing and creating method and device for distributed database and electronic equipment
CN112395366A
Data cleaning method and device
CN113064885A
Log file processing method and device
CN113468613A
MongoDB middleware, data storage method and data migration method
CN113849458A
