Data storage method, device, equipment and computer readable storage medium

Abstract

This application provides a data storage method, apparatus, device, and computer-readable storage medium. The method includes: acquiring at least one piece of initial data to be stored; storing the at least one piece of initial data in a first-level database, each piece of initial data including data from multiple different fields; decomposing the at least one piece of initial data according to the different fields, and marking each field in the decomposed at least one piece of initial data; generating a standardized configuration file based on the data of each field in the marked at least one piece of initial data, and storing the standardized configuration file in a second-level database; integrating the data in the standardized configuration file in the second-level database into a target data table with a preset format, and storing the target data table in a third-level database. This application embodiment can uniformly and standardizedly store numerous and scattered data, reducing the complexity of subsequent data analysis.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a data storage method, apparatus, device and computer-readable storage medium. Background Technology

[0002] Currently, related technologies typically involve pre-agreing with data providers on various data interface specifications when storing data. Corresponding data tables are then created based on each interface specification for data storage. Ultimately, data analysis and visualization are implemented by writing separate programs for these dispersed data tables. Following this approach, the more diverse the data sources and types, the more data tables are required. While simple analyses might only require querying a single table, complex analyses may necessitate queries across multiple tables. This significantly depletes system performance, resulting in very slow response times and potentially causing database host crashes when handling multiple concurrent requests. Summary of the Invention

[0003] This application provides a data storage method, apparatus, device, and computer-readable storage medium that can uniformly and standardizedly store numerous and scattered data, reduce the complexity of subsequent data analysis, improve analysis and query performance, and reduce management difficulty.

[0004] In a first aspect, embodiments of this application provide a data storage method, which includes: acquiring at least one initial data to be stored; storing the at least one initial data in a first-level database, each initial data including data from multiple different fields; decomposing the at least one initial data according to the different fields, and marking each field in the decomposed at least one initial data; generating a standardized configuration file based on the data of each field in the marked at least one initial data, and storing the standardized configuration file in a second-level database; integrating the data in the standardized configuration file in the second-level database into a target data table of a preset format, and storing the target data table in a third-level database.

[0005] According to the first aspect of this application, before storing the target data table in the third-level database, the data storage method further includes: dividing the target data table into multiple partitions, each partition storing a portion of the data in the target data table; after storing the target data table in the third-level database, the data storage method further includes: receiving a query instruction; determining the target partition corresponding to the partition identifier according to the partition identifier in the query instruction; and querying data from the target partition.

[0006] According to any of the foregoing embodiments of the first aspect of this application, the target data table is divided into multiple partitions, specifically including: dividing the target data table into multiple first partitions according to time periods, with different first partitions corresponding to different time periods; and / or dividing the target data table into multiple second partitions according to target frequency types, where the target frequency type includes the region and / or the client to which the data belongs.

[0007] According to any of the foregoing embodiments of the first aspect of this application, before storing the standardized configuration file to the second-level database, the data storage method further includes: for any standardized configuration file, establishing a lineage relationship between multiple data in the standardized configuration file, the lineage relationship including a first lineage relationship between time, a second lineage relationship between space and / or a third lineage relationship between versions; storing the standardized configuration file to the second-level database specifically includes: storing the standardized configuration file after establishing the lineage relationship to the second-level database.

[0008] According to any of the foregoing embodiments of the first aspect of this application, storing at least one initial data record in a first-level database specifically includes: naming at least one initial data record according to a preset file naming rule; and storing the named at least one initial data record in the first-level database according to a preset related path rule.

[0009] According to any of the foregoing embodiments of the first aspect of this application, one initial data corresponds to one file access task; at least one initial data is decomposed according to different fields, specifically including: for any file access task, reading the configuration information of the file access task; and identifying and extracting the data of each field in the initial data corresponding to the file access task according to the configuration information.

[0010] According to any of the foregoing embodiments of the first aspect of this application, before generating a standardized configuration file based on the data of each field in at least one initial data after marking, the data storage method further includes: determining whether the data of each field in the initial data is successfully identified; if the data of each field in the initial data fails to be identified, recording the reason for the failure in the task log; generating a standardized configuration file based on the data of each field in at least one initial data after marking, specifically including: if the data of each field in the initial data is successfully identified, generating a standardized configuration file based on the data of each field in at least one initial data after marking.

[0011] According to any of the foregoing embodiments of the first aspect of this application, after identifying and extracting the data of each field in the initial data corresponding to the file access task based on the configuration information, the data storage method further includes: recording the source, version and / or entry time of the data of each field in the initial data.

[0012] According to any of the foregoing embodiments of the first aspect of this application, integrating the data in the standardized configuration file in the second-layer database into a target data table with a preset format specifically includes: when the standardized configuration file includes data of multiple versions, integrating the latest version of the data in the standardized configuration file into the target data table.

[0013] Secondly, embodiments of this application provide a data storage device, comprising: an acquisition module for acquiring at least one piece of initial data to be stored; a first storage module for storing the at least one piece of initial data in a first-level database, each piece of initial data including data from multiple different fields; a marking module for disassembling the at least one piece of initial data according to different fields and marking each field in the disassembled at least one piece of initial data; a second storage module for generating a standardized configuration file based on the data from each field in the marked at least one piece of initial data and storing the standardized configuration file in a second-level database; and a third storage module for integrating the data in the standardized configuration file in the second-level database into a target data table of a preset format and storing the target data table in a third-level database.

[0014] Thirdly, embodiments of this application provide an electronic device, which includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the data storage method provided in the first aspect.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the data storage method provided in the first aspect.

[0016] The data storage method, apparatus, device, and computer-readable storage medium of this application embodiment store data in a three-layer database. Specifically, at least one initial data record is stored in the first-layer database; the initial data record is decomposed according to different fields, and each field in the decomposed initial data record is marked; a standardized configuration file is generated based on the data of each field in the marked initial data record, and the standardized configuration file is stored in the second-layer database; the data in the standardized configuration file in the second-layer database is integrated into a target data table with a preset format, and the target data table is stored in the third-layer database. By storing the initial data, the standardized configuration file, and the target data table, it is not only beneficial to achieve data traceability, i.e., tracing back to the initial data, but also to be able to query multiple types of data with only the target data table, avoiding the program complexity and maintenance difficulty caused by cross-table queries, reducing the complexity of subsequent data analysis, improving analysis and query performance, reducing management difficulty, and effectively avoiding database host downtime. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a data analysis operation in related technologies;

[0019] Figure 2 A schematic flowchart of a data storage method provided in an embodiment of this application;

[0020] Figure 3 Another flowchart illustrating the data storage method provided in this application embodiment;

[0021] Figure 4 This is a flowchart illustrating step S301 of the data storage method provided in an embodiment of this application.

[0022] Figure 5 A flowchart illustrating step S204 of the data storage method provided in this application embodiment;

[0023] Figure 6 This illustration illustrates the lineage relationships between multiple data points;

[0024] Figure 7 The diagram illustrates the bloodline relationships in region a during a given month.

[0025] Figure 8This is a schematic flowchart of step S202 in the data storage method provided in the embodiments of this application;

[0026] Figure 9 A schematic flowchart of S203 in the data storage method provided in the embodiments of this application;

[0027] Figure 10 A schematic flowchart of a data storage device provided in an embodiment of this application;

[0028] Figure 11 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0029] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0030] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0031] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0032] Various modifications and variations can be made to this application without departing from its spirit or scope, which will be apparent to those skilled in the art. Therefore, this application is intended to cover modifications and variations falling within the scope of the corresponding claims (the claimed technical solutions) and their equivalents. It should be noted that the embodiments provided in this application can be combined with each other without contradiction.

[0033] To facilitate understanding, the technical terms involved in the embodiments of this application will be explained below.

[0034] Frequency indicates the period of data, such as day, week, or month.

[0035] Dimension refers to the way a set of data is organized, forming specific relationships between data and expressing multiple meanings. It is a very important basic concept.

[0036] Caliber, or indicator, is a statistical term referring to the logic of data retrieval. It involves combining dimensions and indicators to form standardized data output, enabling upstream and downstream products to achieve consistency, and is the basic form of data products.

[0037] A horizontal table is a common way to create tables, such as a table that includes a primary key, field 1, field 2, and field 3.

[0038] A vertical table can be understood as splitting a single row of a horizontal table into multiple rows and storing them using key-value pairs.

[0039] Partitioning means physically distributing a data table into multiple tables (i.e., multiple regions) according to certain rules, but logically there is only one table.

[0040] Before describing the technical solutions provided in the embodiments of this application, in order to facilitate understanding of the embodiments of this application, this application will specifically explain the problems existing in the related technologies:

[0041] The main function of a data analysis system is to collect relevant business data from numerous external systems and centrally store it in the system's database. Internally, the system processes and transforms all raw data before storing it in the base database. Then, based on business needs, it performs further data transformations to corresponding tables for use by other upper-level data application components for specialized analysis or visualization.

[0042] Based on the timeliness of data analysis, data analysis systems are generally divided into two types: real-time and offline. Real-time data analysis systems have a strong guarantee of timeliness, as data flows in real time, and corresponding analysis results are also real-time. Offline data analysis systems, on the other hand, analyze existing data more often, and their timeliness requirements are relatively lower.

[0043] For offline data analysis systems, the data may come from multiple sources and be of various types, requiring the system to interface with different external systems (data providers). The data provided by these external systems is also diverse, with varying frequencies, dimensions, definitions, versions, and other factors. When the system needs to quickly and efficiently analyze and present these scattered, disorganized, and massive datasets, subsequent analysis and presentation will face numerous challenges if the data is not properly processed, transformed, and stored.

[0044] Currently, when storing data, related technologies typically involve first agreeing on various data interface specifications with the data provider in advance. Based on each interface specification, corresponding data tables are then created for data storage. Ultimately, data analysis and visualization are implemented by writing separate programs for these dispersed data tables. Following this approach, the more diverse the data sources and types, the more data tables will be required. Figure 1 This is a schematic diagram illustrating a data analysis operation in related technologies. For example... Figure 1 As shown, when performing simple analysis, you may only need to query one data table (such as data table 1), but when performing complex data analysis, you may need to query across multiple data tables (such as data table 2 to data table N), where N is an integer greater than 2.

[0045] In other words, when performing a complex analysis, queries may be performed across multiple tables. Performance is acceptable when the data volume is small, but when the data volume is very large and involves joins between multiple large tables, it will cause the database host resource utilization to spike, severely consuming system performance and resulting in very slow response times. While a single request might be manageable for the host, multiple concurrent requests could very likely cause the database host to crash.

[0046] Furthermore, it can lead to problems in terms of program complexity and reusability. Specifically, this can include the following:

[0047] (1) When performing a complex analysis, in the best case, only a few tables need to be queried, but in the worst case, a dozen or even more tables may need to be queried. Whether the implementation logic is a single Structured Query Language (SQL) statement that directly joins across tables, or multiple SQL statements that are executed separately and then aggregated, the logical complexity of the program will increase.

[0048] (2) R&D personnel need to be familiar with the business. If a certain analysis program is handled by a different person during the development and maintenance process, the new person will need to spend a lot of time familiarizing themselves with the program and understanding the business, resulting in low efficiency. When the program has a bug or the customer has an urgent need, if the new person does not have sufficient familiarity with the program before starting, it may cause a series of online problems and reduce customer satisfaction.

[0049] (3) Because the data tables are scattered, each analysis business requires a different data table, which means that each analysis business needs to write a separate analysis program, and the analysis programs between different analysis businesses are difficult to reuse.

[0050] Furthermore, it can lead to difficulties in managing distributed tables. Specifically, this can include the following:

[0051] (4) When storing data using distributed tables, the system needs to manage these tables appropriately. For example, it is necessary to sort out these numerous and scattered tables, classify and label them, configure the relationships between tables, analyze each table and then create suitable indexes, and so on. As the number of data tables increases, the time and labor costs required will also increase, leading to increased management difficulty and costs.

[0052] (5) When the interface specification changes, the table structure used in production needs to be changed. If the operation is not done properly, it will affect the stability of the production system.

[0053] In view of the inventors’ above-mentioned research findings, the embodiments of this application provide a data storage method, apparatus, device and computer-readable storage medium, which can solve at least one of the above-mentioned technical problems existing in the related art.

[0054] The technical concept of this application embodiment is as follows: data is stored in a three-layer database. Specifically, at least one initial data record is stored in the first-layer database; the initial data record is decomposed according to different fields, and each field in the decomposed initial data record is marked; based on the data of each field in the marked initial data record, a standardized configuration file is generated and stored in the second-layer database; the data in the standardized configuration file in the second-layer database is integrated into a target data table with a preset format, and the target data table is stored in the third-layer database. By storing the initial data, the standardized configuration file, and the target data table, it is not only beneficial to achieve data traceability, i.e., tracing back to the initial data, but also to be able to query multiple types of data that are expected to be queried through only the target data table, avoiding the program complexity and maintenance difficulty caused by cross-table queries, reducing the complexity of subsequent data analysis, improving analysis and query performance, reducing management difficulty, and effectively avoiding database host downtime.

[0055] The data storage method provided in the embodiments of this application will be described below.

[0056] Figure 2 This is a schematic flowchart illustrating a data storage method provided in an embodiment of this application. Figure 2 As shown, the data storage method may include the following steps S201 to S205.

[0057] S201. Obtain at least one initial data entry that is expected to be stored.

[0058] The initial data can be referred to as raw data or data in the original file. The initial data can be uploaded or actively collected by at least one data provider.

[0059] S202. Store at least one initial data record in the first-level database, where each initial data record includes data from multiple different fields.

[0060] In this embodiment, the data is stored in three layers of databases: a first-layer database, a second-layer database, and a third-layer database. The first-layer database, also known as a file repository, is used to store data from the original files (i.e., initial data). The purpose of the file repository is to facilitate subsequent data tracing, problem verification, and data reprocessing by storing the data from the original files.

[0061] The first-level database can store multiple different initial data. These multiple different initial data can be provided by multiple different data providers or by a single data provider. This application embodiment does not limit this.

[0062] Each initial data entry can include data from multiple different fields, such as time, location, source, and / or statistical values.

[0063] S203. According to different fields, split at least one initial data, and mark each field in the split initial data.

[0064] In S203, at least one initial data entry can be decomposed into data with multiple fields. For example, if one initial data entry includes M fields, then that initial data entry can be decomposed into data with M fields, where M is a positive integer. After decomposition, each field can be labeled, i.e., tagged.

[0065] S204. Based on the data of each field in at least one initial data after marking, generate a standardized configuration file and store the standardized configuration file in the second-level database.

[0066] The second-layer database, also known as the process database, executes background tasks to deduplicate, validate, and transform the original files before inserting them into the database. It can also tag each data entry, recording information such as data source, version, and insertion time, facilitating subsequent data traceability and problem investigation. When the same data changes, the second-layer database can insert data from new versions uploaded by data providers or actively collected into the database, marking the version information for easy file tracing later.

[0067] S205. Integrate the data in the standardized configuration file in the second-level database into the target data table with a preset format, and store the target data table in the third-level database.

[0068] A standardized target data table can be established, and the fields included in the target data table can be pre-defined, such as topic type, time, space, source, statistical value ID, and / or statistical value. The fields in the target data table can be flexibly adjusted according to actual conditions, and this application embodiment does not limit this.

[0069] The third-layer database, also known as the asset repository, primarily stores target data tables for front-end queries. The front-end can connect to the asset repository through back-end services to obtain the various data it needs. The asset repository's data originates from the process repository, and it integrates and aggregates data from different themes, frequencies, and sources within the process repository into a unified, standardized target data table.

[0070] The data storage method provided in this application stores data in a three-tiered database. Specifically, at least one initial data record is stored in the first-tier database; the initial data record is then broken down according to different fields, and each field in the broken-down initial data record is marked; based on the data of each field in the marked initial data record, a standardized configuration file is generated and stored in the second-tier database; the data in the standardized configuration file in the second-tier database is integrated into a target data table with a preset format, and the target data table is stored in the third-tier database. By storing the initial data, the standardized configuration file, and the target data table, not only is data traceability (i.e., tracing back to the initial data) facilitated, but multiple types of data can also be queried using only the target data table. This avoids the complexity and maintenance difficulties caused by cross-table queries, reduces the complexity of subsequent data analysis, improves analysis and query performance, reduces management difficulty, and effectively avoids database host downtime.

[0071] Figure 3 This is another schematic flowchart illustrating the data storage method provided in an embodiment of this application. For example... Figure 3As shown, according to some embodiments of this application, optionally, before storing the target data table to the third-level database in S205, the data storage method may further include the following steps:

[0072] S301. Divide the target data table into multiple partitions, with each partition storing a portion of the data in the target data table.

[0073] To improve analytical query efficiency, in some implementations, appropriate partitions can be created for the target data table. Creating partitions physically distributes the data, allowing queries to precisely locate the corresponding partition by specifying the partition key. The more partitions there are, the more dispersed the data becomes, and the smaller the data volume in each partition, thus eliminating the need for full table scans and significantly improving data retrieval efficiency.

[0074] Accordingly, after storing the target data table in the third-level database, the data storage method may further include the following steps S302 to S304.

[0075] S302, Receive query command.

[0076] Query commands can be issued by the user, for example.

[0077] S303. Determine the target partition corresponding to the partition identifier based on the partition identifier in the query instruction.

[0078] Each partition can be assigned a corresponding partition identifier, such as a partition key. By querying the partition identifier in the query command, the target partition corresponding to the partition identifier can be determined.

[0079] S304. Query data from the target partition.

[0080] Once the target partition is determined, data can be queried only from the target partition without a full table scan, greatly improving data retrieval efficiency.

[0081] Figure 4 This is a schematic flowchart of step S301 in the data storage method provided in an embodiment of this application. Figure 4 As shown, according to some embodiments of this application, optionally, S301, dividing the target data table into multiple partitions, may specifically include the following steps S401 and / or S402.

[0082] S401. Divide the target data table into multiple first partitions according to the time period. Different first partitions correspond to different time periods.

[0083] When creating partitions, you can set the partition field and the partition strategy. For the target data table, the partition strategy can use either continuous partitioning (i.e., list partitioning) or discrete partitioning (i.e., range partitioning). Continuous partitioning means that the data in the partition can be continuous, while discrete partitioning means that the data in the partition can be discrete.

[0084] In some embodiments, time can be partitioned using a range, meaning partitions can be created by time period, by year, or by month, depending on the size of the data in a single partition. For example, the daily data volume might be quite large; with 1 million data entries per day over 365 days, that's 365 million data entries per year. Putting all 365 million data entries into one partition would be too much of a burden. In this case, creating partitions by month, assuming a 30-day month, would result in one partition per month, with each partition containing 30 million data entries, significantly reducing the burden.

[0085] The first partition may include the range partition.

[0086] S402. Divide the target data table into multiple second partitions according to the target frequency type. The target frequency type includes the region and / or the client to which the data belongs.

[0087] In some embodiments, the target frequency type can use list partitioning, that is, creating partitions by specifying the enumerated values ​​of the partition field. The second partition can include list partitioning. The target frequency type can include a region and / or the client to which the data belongs. That is, data from a region can reside in the same second partition. And / or, data from a client (APP) can reside in the same second partition.

[0088] In this way, by creating appropriate partitions according to different dimensions, the target data table can be physically divided, greatly improving query performance.

[0089] In some embodiments, the database may include an object-relational database (i.e., a PostgreSQL database), which may support nested partitions, i.e., a first partition and a second partition exist simultaneously.

[0090] In some embodiments, the basic principles for creating partitions may include: (1) minimizing the amount of data in each partition to improve query speed; (2) distributing the amount of data in each partition as evenly as possible; and (3) avoiding querying data across multiple partitions as much as possible.

[0091] Figure 5 This is a schematic flowchart of step S204 in the data storage method provided in an embodiment of this application. Figure 5As shown, according to some embodiments of this application, optionally, in step S204, a standardized configuration file is generated based on the data of each field in at least one initial data after being marked, which may specifically include the following steps S501 and S502.

[0092] S501. Set multiple different target fields, which may include time, space, statistics, source and / or version.

[0093] As mentioned earlier, the initial data may include multiple fields. In S501, at least some fields can be selected from the multiple fields of the initial data as multiple target fields. For example, the multiple target fields may include time, location, statistics, source, and / or version.

[0094] S502. For any target field in the initial data, generate a standardized configuration file corresponding to the target field based on the data of the target field in at least one piece of the initial data after marking.

[0095] Different standardized configuration files can be generated for different target fields. For example, for the target field of time, a standardized configuration file corresponding to the time field can be generated based on the data of the time field in at least one initial data record after marking. For example, for the target field of space, a standardized configuration file corresponding to the space field can be generated based on the data of the space field in at least one initial data record after marking. ... For example, for the target field of version, a standardized configuration file corresponding to the version field can be generated based on the data of the version field in at least one initial data record after marking.

[0096] Accordingly, in S205, data from standardized configuration files corresponding to multiple target fields can be integrated into a target data table with a preset format, and the target data table can be stored in a third-level database.

[0097] According to some embodiments of this application, optionally, before S204, storing the standardized configuration file to the second-level database, the data storage method provided in the embodiments of this application may further include the following steps:

[0098] For any standardized configuration file, establish the lineage relationships between multiple data points within the standardized configuration file.

[0099] Figure 6 This schematically illustrates the lineage relationships between multiple data points. For example... Figure 6 As shown, in some embodiments, blood relations may include a first blood relation between time periods, a second blood relation between spaces, and / or a third blood relation between versions. In other embodiments, blood relations may also include a fourth blood relation, etc., which are not limited in this application.

[0100] Accordingly, S204, storing the standardized configuration file in the second-level database may specifically include the following steps:

[0101] The standardized configuration files established after establishing blood relations are stored in the second-level database.

[0102] In some embodiments, task logs are recorded during the insertion of data into the target data table, forming task ledger information. Based on the lineage relationships in the standardized configuration file and the task ledger information, the data lineage relationships can be generated. Lineage relationships are crucial for analyzing data relationships, tracing data formation, and reflecting data issues in data asset management.

[0103] Figure 7 The diagram illustrates the kinship distribution in region a in March 2021. (As shown) Figure 7 As shown, residents in region a can display their blood relations by region, caliber, origin, age, gender, place of origin, and place of origin.

[0104] Figure 8 This is a schematic flowchart of step S202 in the data storage method provided in an embodiment of this application. Figure 8 As shown, according to some embodiments of this application, optionally, in step S202, at least one initial data is stored in the first-level database, which may specifically include the following steps S801 and S802.

[0105] S801. Name at least one initial data entry according to the preset file naming rules.

[0106] S802. According to the preset relevant path rules, store at least one named initial data into the first-level database.

[0107] Specifically, each initial data entry can be named according to preset file naming rules and categorized and stored according to preset relevant path rules to facilitate data retrieval.

[0108] According to some embodiments of this application, optionally, one initial data entry can correspond to one file access task.

[0109] Figure 9 This is a schematic flowchart of step S203 in the data storage method provided in an embodiment of this application. Figure 9 As shown, S203, according to different fields, at least one initial data is split, which may specifically include the following steps S901 and S902.

[0110] S901. For any file access task, read the configuration information of the file access task.

[0111] The process of converting data from the original file into data stored in the database requires prior configuration and loading, which may include basic task information configuration and detailed task attribute information configuration.

[0112] Accordingly, the configuration information for a file access task can include basic configuration information and detailed attribute configuration information. In S901, the basic configuration information of the file access task can be read. Table 1 schematically illustrates the basic configuration information of a file access task.

[0113] Table 1

[0114]

[0115] As shown in Table 1, in some examples, the basic configuration information of the file access task includes, but is not limited to, task ID, source information, file path wildcard, file name wildcard, file content encoding format, data column separator, data row separator, and / or whether there is a verification file.

[0116] In S901, detailed attribute configuration information for the file access task can be read. Table 2 schematically illustrates the detailed attribute configuration information for the file access task.

[0117] Table 2

[0118]

[0119] As shown in Table 2, in some examples, the detailed attribute configuration information for the file access task includes, but is not limited to, access table name, attribute configuration, and / or mapping configuration in the statistics table. Attribute configuration includes, but is not limited to, the configuration of time attributes, spatial attributes, and statistical value attributes.

[0120] S902. Based on the configuration information, identify and extract the data of each field in the initial data corresponding to the file access task.

[0121] The process of reading the original file content and picking up relevant data can identify and extract the content of the initial data corresponding to the file access task based on the configuration information read in the previous step. This includes data such as fields such as time, space, source, and statistical value attributes, in preparation for the next step of data reorganization, i.e., generating a standardized configuration file.

[0122] According to some embodiments of this application, optionally, before generating a standardized configuration file based on the data of each field in at least one initial data after marking in S204, the data storage method may further include the following steps:

[0123] Determine whether the data in each field of the initial data has been successfully recognized;

[0124] If data recognition fails in any field of the initial data, the reason for the failure is recorded in the task log.

[0125] Accordingly, S204, based on the data of each field in at least one initial data entry after marking, generates a standardized configuration file, which may specifically include the following steps:

[0126] If the data of each field in the initial data is successfully identified, a standardized configuration file is generated based on the data of each field in at least one piece of the initial data after it has been marked.

[0127] After storing the standardized configuration file to the second-level database, a successful storage can be recorded in the task log.

[0128] According to some embodiments of this application, optionally, after identifying and extracting data from each field in the initial data corresponding to the file access task based on configuration information, the data storage method may further include the following steps:

[0129] The first layer of database records the source, version, and / or entry time of each field in the initial data. The second layer can retain all historical versions of the file, recording information such as data source, version, and entry time, facilitating subsequent data traceability, problem verification, and other operations.

[0130] According to some embodiments of this application, optionally, S205, integrating the data in the standardized configuration file in the second-level database into a target data table with a preset format, specifically includes:

[0131] When the standardized configuration file includes data from multiple versions, the latest version of the data in the standardized configuration file is integrated into the target data table.

[0132] When there are different versions of the same data, the third-level database can store the latest version of the data.

[0133] According to some embodiments of this application, optionally, in S205, the standardized configuration file can be content identified, for example, the time, space, source, and statistical value attribute information in the content of the standardized configuration file can be picked up and inserted into the corresponding fields of the unified target data table.

[0134] Table 3 schematically illustrates the target data table.

[0135] Table 3

[0136]

[0137] As shown in Table 3, in some examples, the target data table may include fields such as subject type, time, space (i.e., region), period type, data type, indicator code, indicator value, task ID, data version, and / or insertion time.

[0138] According to some embodiments of this application, optionally, each piece of data in the third-layer database can record a task execution ID, which can be traced back to the first-layer database and the second-layer database.

[0139] According to some embodiments of this application, optionally, the target data table may include a vertical table, i.e., a single data table.

[0140] With data-using departments having multiple data sources, complex data, and increasing data volume, the traditional model of personalized data asset management and separation of asset management systems from data can no longer meet market demands. Problems such as chaotic data access, inability to manage, and unclear data are becoming increasingly prominent.

[0141] The data sources are numerous and complex, making it difficult to directly demonstrate the value of data resources. The data sources are diverse, including structured and semi-structured data. Raw data needs to be processed before it can directly provide data services to the platform. Data preprocessing is necessary before data processing to provide relatively accurate and on-demand data services for business applications.

[0142] The data is complex and varied, and the results generated by the application of data from different periods and versions lack traceability. When the data provided by the original data producer enters the platform, it is verified by the business application. Data with problems needs to be resubmitted by the data producer. During the data processing process of the business application, a large amount of process data with different versions is generated. Different business processes choose different versions of process data, resulting in some differences in the results displayed by the business application. It is necessary to monitor the process of data entry, processing, and application, and to track and trace the application process of different versions of data.

[0143] There is a lack of intuitive display of the flow and changes of valuable data resources. When designing data processing logic, business applications need to access the platform multiple times to query data and make judgments such as whether the data exists and whether it can be supported. Therefore, it is necessary to display data assets so that data assets can be "visible".

[0144] The data resource management system constructs a comprehensive, effective, and practical data asset management system through data resourceization, improves data quality, enriches data asset application scenarios through data assetization, establishes a data asset ecosystem, summarizes data management processes, and builds a systematic data management system. It designs a unified architecture in terms of data storage structure, processing flow, and management methods, supports continuous access, standardized management, and accurate use of multi-source data, and effectively solves the above-mentioned practical problems faced by government and enterprise data management.

[0145] The data storage method provided in this application avoids the complexity and maintenance difficulties caused by cross-table queries by establishing a unified target data table. Analysis programs developed using this unified target data table have high versatility, thereby improving R&D efficiency. Furthermore, it reduces the management difficulty of multi-source and multi-type data, lowers manual management costs, and improves production efficiency.

[0146] Based on the data storage method provided in the above embodiments, this application also provides specific implementations of a data storage device. Please refer to the following embodiments.

[0147] Figure 10 This is a schematic flowchart of a data storage device provided in an embodiment of this application. Figure 10 As shown, the data storage device 100 provided in this application embodiment may include the following modules:

[0148] The acquisition module 1001 is used to acquire at least one piece of initial data that is expected to be stored;

[0149] The first storage module 1002 is used to store at least one initial data record into the first-level database, and each initial data record includes data from multiple different fields.

[0150] The tagging module 1003 is used to decompose at least one initial data according to different fields, and to tag each field in the decomposed at least one initial data.

[0151] The second storage module 1004 is used to generate a standardized configuration file based on the data of each field in at least one initial data after marking, and to store the standardized configuration file in the second-level database;

[0152] The third storage module 1005 is used to integrate the data in the standardized configuration file in the second-level database into the target data table with a preset format, and store the target data table in the third-level database.

[0153] The data storage device provided in this application stores data in a three-tiered database. Specifically, at least one initial data record is stored in the first-tier database; the initial data record is then broken down according to different fields, and each field in the broken-down initial data record is marked; based on the data of each field in the marked initial data record, a standardized configuration file is generated and stored in the second-tier database; the data in the standardized configuration file in the second-tier database is integrated into a target data table with a preset format, and the target data table is stored in the third-tier database. By storing the initial data, the standardized configuration file, and the target data table, not only is data traceability (i.e., tracing back to the initial data) facilitated, but multiple types of data can also be queried using only the target data table. This avoids the complexity and maintenance difficulties caused by cross-table queries, reduces the complexity of subsequent data analysis, improves analysis and query performance, reduces management difficulty, and effectively avoids database host downtime.

[0154] Optionally, according to some embodiments of this application, the data storage device 100 provided in the embodiments of this application may further include a partitioning module for dividing a target data table into multiple partitions, each partition storing a portion of the data in the target data table. The data storage device 100 provided in the embodiments of this application may further include a query module for receiving a query instruction; determining the target partition corresponding to the partition identifier according to the partition identifier in the query instruction; and querying data from the target partition.

[0155] According to some embodiments of this application, optionally, the partitioning module is specifically used to divide the target data table into multiple first partitions according to time periods, with different time periods corresponding to different first partitions; and / or, to divide the target data table into multiple second partitions according to target frequency types, where the target frequency type includes the region and / or the client to which the data belongs.

[0156] According to some embodiments of this application, optionally, the second storage module 1004 is specifically used to set multiple different target fields, including time, space, statistical value, source and / or version; for any target field in the initial data, a standardized configuration file corresponding to the target field is generated based on the data of the target field in at least one piece of initial data after being marked.

[0157] Optionally, according to some embodiments of this application, the data storage device 100 provided in the embodiments of this application may further include a establishment module, used to establish lineage relationships between multiple data in any standardized configuration file, the lineage relationships including a first lineage relationship between time, a second lineage relationship between space, and / or a third lineage relationship between versions. The second storage module 1004 is specifically used to store the standardized configuration file after the lineage relationship has been established to a second-level database.

[0158] According to some embodiments of this application, optionally, the first storage module 1002 is specifically used to name at least one initial data according to a preset file naming rule; and to store the named at least one initial data into the first-level database according to a preset related path rule.

[0159] According to some embodiments of this application, optionally, one initial data entry corresponds to one file access task. The tagging module 1003 is specifically used to, for any file access task, read the configuration information of the file access task; and, based on the configuration information, identify and extract the data of each field in the initial data corresponding to the file access task.

[0160] Optionally, according to some embodiments of this application, the data storage device 100 provided in the embodiments of this application may further include a judgment module, used to judge whether the data of each field in the initial data has been successfully recognized; if the data of each field in the initial data fails to be recognized, the reason for the failure is recorded in the task log. The second storage module 1004 is specifically used to generate a standardized configuration file based on the data of each field in at least one piece of the marked initial data when the data of each field in the initial data has been successfully recognized.

[0161] Optionally, according to some embodiments of this application, the data storage device 100 provided in the embodiments of this application may further include a recording module for recording the source, version and / or entry time of the data in each field of the initial data.

[0162] According to some embodiments of this application, optionally, the third storage module 1005 is specifically used to integrate the latest version of the data in the standardized configuration file into the target data table when the standardized configuration file includes multiple versions of data.

[0163] Figure 10 Each module / unit in the device shown has the function of implementing each step in the data storage method provided in the above method embodiments, and can achieve its corresponding technical effect. For the sake of brevity, it will not be described in detail here.

[0164] Based on the data storage method provided in the above embodiments, this application also provides specific implementation methods for electronic devices. Please refer to the following embodiments.

[0165] Figure 11 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0166] The electronic device may include a processor 1101 and a memory 1102 storing computer program instructions.

[0167] Specifically, the processor 1101 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0168] Memory 1102 may include mass storage for data or instructions. For example, and not limitingly, memory 1102 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one example, memory 1102 may include removable or non-removable (or fixed) media, or memory 1102 may be non-volatile solid-state memory. Memory 1102 may be internal or external to an electronic device.

[0169] In one example, memory 1102 may be read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0170] Memory 1102 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.

[0171] The processor 1101 reads and executes the computer program instructions stored in the memory 1102 to implement the methods / steps in the above method embodiments and achieve the corresponding technical effects achieved by the method embodiments in executing their methods / steps. For the sake of brevity, these will not be elaborated here.

[0172] In one example, the electronic device may also include a communication interface 1103 and a bus 1110. For example, Figure 11 As shown, the processor 1101, memory 1102, and communication interface 1103 are connected through bus 1110 and complete communication with each other.

[0173] The communication interface 1103 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0174] Bus 1110 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 1110 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0175] Furthermore, in conjunction with the data storage methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data storage methods in the above embodiments. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as electronic circuits, semiconductor memory devices, ROM, random access memory, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, and hard disks.

[0176] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0177] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0178] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0179] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0180] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data storage method, characterized by, include: Retrieve at least one initial piece of data that is expected to be stored; The at least one initial data item is stored in the first-level database, and each initial data item includes data from multiple different fields; According to different fields, the at least one initial data is decomposed and each field in the decomposed at least one initial data is marked; Based on the data of each field in the marked initial data, a standardized configuration file is generated and stored in the second-level database; The data in the standardized configuration file in the second-level database is integrated into a target data table with a preset format, and the target data table is stored in the third-level database; The process of generating a standardized configuration file based on the data of each field in the at least one initial data set after labeling specifically includes: Set multiple different target fields, including time, space, statistical value, source and / or version; For any target field in the initial data, based on the data of the target field in the at least one initial data after marking, a standardized configuration file corresponding to the target field is generated; Before storing the standardized configuration file to the second-level database, the method further includes: For any one of the standardized configuration files, establish lineage relationships between multiple data in the standardized configuration file, including a first lineage relationship between time, a second lineage relationship between space, and / or a third lineage relationship between versions; The step of storing the standardized configuration file in the second-level database specifically includes: The standardized configuration file, after establishing blood relations, is stored in the second-level database.

2. The method according to claim 1, characterized in that, Before storing the target data table in the third-level database, the method further includes: The target data table is divided into multiple partitions, and each partition stores a portion of the data in the target data table. After storing the target data table in the third-level database, the method further includes: Receive query command; Based on the partition identifier in the query instruction, determine the target partition corresponding to the partition identifier; Query data from the target partition.

3. The method according to claim 2, characterized in that, The step of dividing the target data table into multiple partitions specifically includes: The target data table is divided into multiple first partitions according to time periods, and different first partitions correspond to different time periods; And / or, according to the target frequency type, the target data table is divided into multiple second partitions, the target frequency type including the region and / or the client to which the data belongs.

4. The method according to claim 1, characterized in that, The step of storing the at least one initial data record in the first-level database specifically includes: Name the at least one initial data entry according to the preset file naming rules; According to the preset relevant path rules, the named initial data is stored in the first-level database.

5. The method according to claim 1, characterized in that, One piece of initial data corresponds to one file access task; The step of splitting the at least one initial data entry according to different fields specifically includes: For any of the file access tasks, read the configuration information of the file access task; Based on the configuration information, identify and extract data from each field in the initial data corresponding to the file access task.

6. The method according to claim 5, characterized in that, Before generating a standardized configuration file based on the data of each field in the at least one initial data set after tagging, the method further includes: Determine whether the data in each field of the initial data has been successfully recognized; If data recognition fails in any field of the initial data, the reason for the failure is recorded in the task log. The process of generating a standardized configuration file based on the data of each field in the at least one initial data set after labeling specifically includes: If the data of each field in the initial data is successfully identified, the standardized configuration file is generated based on the data of each field in the at least one initial data after being marked.

7. The method according to claim 5, characterized in that, After identifying and extracting data from each field of the initial data corresponding to the file access task based on the configuration information, the method further includes: Record the source, version, and / or entry time of each field in the initial data.

8. The method according to claim 1, characterized in that, The step of integrating the data from the standardized configuration file in the second-layer database into a target data table with a preset format specifically includes: When the standardized configuration file includes data from multiple versions, the latest version of the data in the standardized configuration file is integrated into the target data table.

9. A data storage device, characterized in that, include: The acquisition module is used to acquire at least one piece of initial data that is expected to be stored. The first storage module is used to store the at least one initial data in a first-level database, wherein each initial data includes data with multiple different fields. The tagging module is used to decompose the at least one initial data according to different fields, and to tag each field in the decomposed at least one initial data. The second storage module is used to generate a standardized configuration file based on the data of each field in the marked initial data, and store the standardized configuration file in the second-level database; The second storage module is also used to establish a lineage relationship between multiple data in any one of the standardized configuration files, wherein the lineage relationship includes a first lineage relationship between time, a second lineage relationship between space and / or a third lineage relationship between versions; The second storage module is used to generate a standardized configuration file based on the data of each field in the at least one initial data entry after it has been tagged, specifically including: Set multiple different target fields, including time, space, statistical value, source and / or version; For any target field in the initial data, based on the data of the target field in the at least one initial data after marking, a standardized configuration file corresponding to the target field is generated; The second storage module is used to store the standardized configuration file to the second-level database, specifically including: storing the standardized configuration file after establishing blood relations to the second-level database; The third storage module is used to integrate the data in the standardized configuration file in the second-level database into a target data table with a preset format, and store the target data table in the third-level database.

10. An electronic device, characterized in that, The electronic device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data storage method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the data storage method as described in any one of claims 1 to 8.

Images