Data storage method, device, system, electronic device and storage medium

Abstract

The present application relates to a data storage method, device, system, electronic device and storage medium, wherein the data storage method comprises: obtaining the original data to be stored and a preset dictionary table, and arranging the original data to be stored in sequence according to the preset dictionary table to obtain storage data; generating a storage template according to the preset dictionary table, and encapsulating the storage data according to the storage template to obtain encapsulated data; and inserting the encapsulated data into a database in batches to obtain a database storage result. Through the present application, the problem of low accuracy and efficiency of data storage is solved, and the purpose of inserting disordered data into a database in batches is achieved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data storage method, device, system, electronic device and storage medium. Background Art

[0002] In a big data platform, it is necessary to batch store data after consumption processing in the big data platform (such as Kafka) into a database (such as ClickHouse, a column-based database management system for online analysis). However, the message data after consumption processing on the big data platform often has missing field values ​​and disordered field order.

[0003] In the current related technologies, in order to solve the data warehousing problems such as missing field values ​​and disordered field order after big data technology processing, the main method used is to define the data item name and data item content to match and then re-enter the data. This method has low warehousing accuracy and low efficiency.

[0004] There is currently no effective solution to the problems of low accuracy and efficiency of data entry in related technologies. Summary of the invention

[0005] In this embodiment, a data warehousing method, device, system, electronic device and storage medium are provided to solve the problems of low accuracy and efficiency of data warehousing in related technologies.

[0006] In a first aspect, a data storage method is provided in this embodiment, and the method includes:

[0007] Acquire the original data to be stored and a preset dictionary table, and arrange the original data to be stored in sequence according to the preset dictionary table to obtain storage data;

[0008] Generate a storage template according to the preset dictionary table, and encapsulate the storage data according to the storage template to obtain encapsulated data;

[0009] The encapsulated data is inserted into the database in batches to obtain the database entry result.

[0010] In some embodiments, the step of sequentially arranging the raw data to be stored to obtain the stored data includes:

[0011] The data to be stored is preliminarily packaged to obtain preliminarily packaged original data; and the preset dictionary table is preliminarily packaged to obtain a preliminarily packaged preset dictionary table; wherein the preliminarily packaged preset dictionary table includes a preset dictionary table field name, a preset dictionary table field initial value, and a preset dictionary table field value initial value;

[0012] Compare the fields in the preliminary encapsulated original data with the preliminary encapsulated preset dictionary table to obtain a comparison result, and if the comparison result indicates that the preliminary encapsulated original data field is missing, assign an initial value to the missing preliminary encapsulated original data field according to the initial value of the field in the preset dictionary table to obtain the original data to be circulated;

[0013] Obtain the current data strip to be circulated of the original data to be circulated;

[0014] Read the dictionary table field names of the preset dictionary table in sequence, obtain the current field value corresponding to the dictionary table field name in the current data strip to be circulated, and obtain the current storage data according to the current field value;

[0015] Obtain the next data strip to be circulated of the original data to be circulated, read the field names of the dictionary table in sequence, obtain the next field value corresponding to the field name of the dictionary table in the next data strip to be circulated, and obtain the next storage data according to the next field value, until the reading of the data strip to be circulated in the data to be circulated is completed;

[0016] The storage data is obtained according to the current storage data and the next storage data.

[0017] In some embodiments, the step of sequentially arranging the raw data to be stored to obtain the stored data further includes:

[0018] Obtaining data entries to be stored in the original data to be stored;

[0019] According to the dictionary table field of the preset dictionary table, the to-be-stored field corresponding to the dictionary table field in the to-be-stored data strip is obtained; wherein the dictionary table field includes the dictionary table field name, the dictionary table field initial value and the dictionary table field value initial value; the to-be-stored field includes the to-be-stored field name and the to-be-stored field value;

[0020] According to the dictionary table fields, the fields to be stored are extracted to obtain the stored data.

[0021] In some embodiments, extracting the to-be-stored fields according to the dictionary table fields to obtain the first stored data includes:

[0022] Performing a field detection on the field to be stored to obtain a field detection result, if the field detection result indicates that the field to be stored fails to be obtained, then obtaining the corresponding initial value of the dictionary table field according to the name of the field to be stored, and obtaining the non-empty field storage data according to the initial value of the dictionary table field; wherein the non-empty field storage data includes the non-empty field storage data name and the non-empty field storage data value;

[0023] The non-empty field storage data is extracted according to the dictionary table fields to obtain the storage data.

[0024] In some embodiments, extracting the to-be-stored fields according to the dictionary table fields to obtain the first stored data further includes:

[0025] Performing a value detection on the field value to be stored in the field to be stored in order to obtain a value detection result; if the value detection result indicates that the field value to be stored in is a null value, obtaining the corresponding initial value of the dictionary table field value according to the name of the field to be stored in order to obtain the non-null value storage data according to the initial value of the dictionary table field value; wherein the non-null value storage data includes the name of the non-null value storage data and the value of the non-null value storage data;

[0026] The non-null value storage data is extracted according to the dictionary table field to obtain the first storage data.

[0027] In some embodiments, generating a storage template according to the preset dictionary table includes:

[0028] An initial storage template is obtained, and the storage template is generated according to the preset dictionary table.

[0029] In a second aspect, a data storage device is provided in this embodiment, characterized in that it includes: a sorting module, a packaging module and a storage module;

[0030] The sorting module is used to obtain the original data to be stored and a preset dictionary table, and to arrange the original data to be stored in sequence according to the preset dictionary table to obtain the first storage data;

[0031] The encapsulation module is used to generate a storage template according to the preset dictionary table, and encapsulate the first storage data according to the storage template to obtain encapsulated data;

[0032] The storage module is used to insert the encapsulated data into the database in batches to obtain the database storage result.

[0033] In a third aspect, a data storage system is provided in this embodiment, characterized in that it includes: a terminal device, a transmission device and a server device; wherein the terminal device is connected to the server device via the transmission device;

[0034] The server device is used to execute the data storage method described in any one of the first aspects above;

[0035] The transmission device is used to transmit the database storage result from the server device to the terminal device;

[0036] The terminal device is used to display the database entry result.

[0037] In a third aspect, an electronic device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data warehousing method described in the first aspect is implemented.

[0038] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the data warehousing method described in the first aspect is implemented.

[0039] Compared with the related art, the data warehousing method, device, system, electronic device and storage medium provided in this embodiment obtain the original data to be stored and the preset dictionary table, and arrange the original data to be stored in sequence according to the preset dictionary table to obtain the storage data; generate a storage template according to the preset dictionary table, and encapsulate the storage data according to the storage template to obtain encapsulated data; insert the encapsulated data into the database in batches to obtain the database storage results, which solves the problems of low data storage accuracy and efficiency in the related art and achieves the purpose of inserting disordered data into the database in batches.

[0040] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 This is an application scenario diagram of a data storage method in an embodiment;

[0043] Figure 2 A schematic diagram of a process of a data storage method in one embodiment;

[0044] Figure 3 A schematic diagram of a flow chart of another data storage method in an embodiment;

[0045] Figure 4 It is a structural block diagram of a data storage device in one embodiment;

[0046] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0048] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0049] The data storage method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The server 104 obtains the original data to be stored and the preset dictionary table, and arranges the original data to be stored in sequence according to the preset dictionary table to obtain the storage data; the server 104 generates a storage template according to the preset dictionary table, and encapsulates the storage data according to the storage template to obtain the encapsulated data; the server 104 inserts the encapsulated data into the database in batches to obtain the database storage result; the terminal 102 displays the database storage result. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0050] In this embodiment, a data storage method is provided. Figure 2 is a flow chart of the data storage method of this embodiment, such as Figure 2 As shown, the process includes the following steps:

[0051] Step S202, obtaining the original data to be stored and a preset dictionary table, and arranging the original data to be stored in sequence according to the preset dictionary table to obtain storage data.

[0052] Among them, the original data to be stored refers to the disordered data after consumption from various platforms and systems, for example, it can be data consumed from the kafka architecture of the big data platform, or it can be data consumed by Spark and storm, which will not be repeated here; the preset dictionary table refers to the dictionary table data preset in the server 104, which is stored in the database of the server 104; the dictionary table specifies the field name, field type, field default value and field value default value of the data field to be stored, as well as a fixed field storage order.

[0053] Specifically, after obtaining the original data to be stored and the preset dictionary table, each field in the original data to be stored is obtained according to the field storage order specified in the preset dictionary table, so that the original data to be stored is arranged in sequence to obtain the storage data.

[0054] Step S204, generating a storage template according to the preset dictionary table, and encapsulating the storage data according to the storage template to obtain encapsulated data.

[0055] The storage template refers to a unified format for batch data storage; the packaged data refers to the data obtained by batch packaging of multi-table data for a large amount of multi-table storage data according to the storage template.

[0056] Step S206, inserting the packaged data into the database in batches to obtain a database entry result.

[0057] Among them, the batch insertion into the database refers to inserting the above-mentioned large-scale, multi-table encapsulated data into the final result data table in the server 104 for storage according to the batch insertion program statement. For example, PreparedStatement can be used for batch warehousing operations; when the PreparedStatement performs batch data warehousing operations, it must ensure that all field names of the batch inserted data are the same before batch insertion can be performed.

[0058] Through the above steps, after obtaining the original data to be stored and the preset dictionary table, each field in the original data to be stored is obtained according to the field storage order specified in the preset dictionary table, so that the original data to be stored is arranged in sequence, and after obtaining the storage data, it is packaged according to the storage template to obtain the packaged data, and the packaged data is inserted into the database in batches to obtain the database storage result. Compared with the related art, the data storage method provided in this embodiment rearranges the disordered data after consumption, ensures that the data value corresponds to the inserted field when the data is stored, and no subsequent manual correction is required. At the same time, large quantities of multi-table data can be packaged and stored uniformly, without having to insert a single table into the database multiple times, which solves the problems of low data storage accuracy and efficiency in the related art and achieves the purpose of inserting disordered data into the database in batches.

[0059] In some embodiments, the raw data to be stored are sequentially arranged to obtain the stored data, including:

[0060] The data to be stored is preliminarily packaged to obtain preliminarily packaged original data; and the preset dictionary table is preliminarily packaged to obtain a preliminarily packaged preset dictionary table; wherein the preliminarily packaged preset dictionary table includes a preset dictionary table field name, a preset dictionary table field initial value, and a preset dictionary table field value initial value;

[0061] Compare the fields in the preliminary encapsulated original data with the preliminary encapsulated preset dictionary table to obtain a comparison result, and if the comparison result indicates that the preliminary encapsulated original data field is missing, assign an initial value to the missing preliminary encapsulated original data field according to the initial value of the field in the preset dictionary table to obtain the original data to be circulated;

[0062] Obtain the current data strip to be circulated of the original data to be circulated;

[0063] Read the dictionary table field names of the preset dictionary table in sequence, obtain the current field value corresponding to the dictionary table field name in the current data strip to be circulated, and obtain the current storage data according to the current field value;

[0064] Obtain the next data strip to be circulated of the original data to be circulated, read the field names of the dictionary table in sequence, obtain the next field value corresponding to the field name of the dictionary table in the next data strip to be circulated, and obtain the next storage data according to the next field value, until the reading of the data strip to be circulated in the data to be circulated is completed;

[0065] The storage data is obtained according to the current storage data and the next storage data.

[0066] Among them, after obtaining the data to be stored, the original data to be stored becomes data that has been processed in a loop. The data that has been processed in a loop may contain fields or values ​​that exceed the range specified by the dictionary table. The fields or values ​​that exceed the range specified by the dictionary table are automatically cleared by the server 104 after the above-mentioned loop processing.

[0067] Through the above embodiment, the original data to be circulated is obtained after the initial value of the field is assigned to the preliminary packaged original data, and the current loop and the next loop are performed on the original data to be circulated to obtain the current storage data and the next storage data, and the preset dictionary table is read in turn in the current loop and the next loop to form a double loop to obtain the storage data, thereby realizing the sequential arrangement of the data to be stored through the double loop, solving the problem of low accuracy and efficiency of data storage.

[0068] In some embodiments, the raw data to be stored are sequentially arranged to obtain the stored data, and further includes:

[0069] Obtain the data item to be stored in the original data to be stored;

[0070] According to the dictionary table field of the preset dictionary table, the to-be-stored field corresponding to the dictionary table field in the to-be-stored data strip is obtained; wherein the dictionary table field includes the dictionary table field name, the dictionary table field initial value and the dictionary table field value initial value; the to-be-stored field includes the to-be-stored field name and the to-be-stored field value;

[0071] According to the dictionary table fields, the fields to be stored are extracted to obtain the stored data.

[0072] The dictionary table field also includes a field type, and the field type determines the initial value of the dictionary table field corresponding to the dictionary table field; extracting the field to be stored means extracting the field to be stored according to the order of the dictionary table fields.

[0073] It can be understood that the storage data can be the current storage data obtained according to the current data strip to be circulated in the above embodiment, or the next storage data obtained according to the next data strip to be circulated in the above embodiment.

[0074] Through the above embodiment, by obtaining the data strips to be stored and obtaining the fields to be stored according to the fields in the dictionary table, the data strips to be stored are arranged in sequence, thus solving the problem of low accuracy and efficiency of data storage.

[0075] In some embodiments, the to-be-stored field is extracted according to the dictionary table field to obtain the first stored data, including:

[0076] Perform a field detection on the field to be stored to obtain a field detection result. If the field detection result indicates that the field to be stored fails to be obtained, then obtain the corresponding initial value of the dictionary table field according to the name of the field to be stored, and obtain the non-empty field storage data according to the initial value of the dictionary table field; wherein the non-empty field storage data includes the non-empty field storage data name and the non-empty field storage data value;

[0077] The stored data with no empty fields is extracted according to the fields in the dictionary table to obtain the stored data.

[0078] The extracting of the non-empty field stored data refers to extracting the non-empty field stored data according to the order of the fields in the dictionary table.

[0079] Through the above embodiment, when the field detection result indicates that the acquisition of the field to be stored in the warehouse fails, the field to be stored in the warehouse is missing, and the corresponding initial value of the dictionary table field is obtained through dictionary table query to assign the field to be stored in the warehouse; at the same time, the dictionary table field includes the field type, and the field value of the field to be stored in the warehouse can be obtained according to the field type specified in the dictionary table field, thereby solving the problem of missing fields of data to be stored and achieving the purpose of inserting disordered data into the database in batches.

[0080] In some embodiments, the method further comprises: extracting the to-be-stored field according to the dictionary table field to obtain the first stored data;

[0081] Performing a value detection on the field to be stored in the field to be stored in the warehouse to obtain a value detection result. If the value detection result indicates that the value of the field to be stored in the warehouse is a null value, obtaining the corresponding initial value of the field in the dictionary table according to the name of the field to be stored in the warehouse, and obtaining the non-null value storage data according to the initial value of the field in the dictionary table; wherein the non-null value storage data includes the name of the non-null value storage data and the value of the non-null value storage data;

[0082] The non-null value storage data is extracted according to the dictionary table field to obtain the first storage data.

[0083] The extracting of the non-null value stored data refers to extracting the non-null value stored data according to the order of the fields in the dictionary table.

[0084] Through the above embodiment, when the field value detection result indicates that the acquisition of the field value to be stored in the warehouse fails, the field value to be stored in the warehouse is missing, and the corresponding initial value of the field value of the dictionary table is obtained through the dictionary table query to assign the field value to be stored in the warehouse; at the same time, the dictionary table field includes the field type, and the field value of the field to be stored in the warehouse can be obtained according to the field type specified in the dictionary table field, thereby solving the problem of missing field values ​​of the data to be stored, and achieving the purpose of inserting disordered data into the database in batches.

[0085] This embodiment also provides another method for extracting fields to be stored, which uses a double loop to assign values ​​to empty value data in the data to be stored; the double loop is divided into two layers, the outer layer is the data layer, and the inner layer is the dictionary layer.

[0086] The data layer encapsulates the data to be stored into a List <Map<String,Object> The data to be stored is disordered data, which is a Map set of KV structure containing the field names to be stored and the field values ​​to be stored, and the field names to be stored and the field values ​​to be stored have redundant fields and vacancies; wherein, the KV structure refers to a KV storage mechanism (key-value-store) that encapsulates and stores infrequently changing data in a KV structure.

[0087] The dictionary layer is used to ensure that the corresponding field name and value to be stored are pulled from a piece of data to be stored in the outer data layer; the structure of the dictionary layer is List <sysdictionary>; Among them, the SysDictionary is the entity class of the dictionary table, which represents all the properties of a field, including the field name, field type, whether it is the primary key, field description and other detailed information of the field.

[0088] Specifically, traverse the List of the inner dictionary layer <sysdictionary>In a set, from the Map set of a piece of data obtained from the outer layer according to the dictionary table field name, obtain the corresponding value of the field to be stored in the database according to the dictionary table field name (the key in the KV); if the value of the field to be stored in the database is a null value, then assign the default value of the corresponding type according to the field type of this dictionary table field in the dictionary table, and finally initialize the values part of the PreparedStatement to obtain the data to be stored in the database.

[0089] Specifically, for example, a piece of data in the outer data layer is {"name":"Zhang San","age":"","address":"Zhejiang Province","sex":"male"}; the inner dictionary table order is name, address, age; the corresponding types are String, String, int respectively; the inner loop starts from name and obtains the value according to the key name from the corresponding map set in the outer layer. After the inner loop is completed, the final order is name, address, age, and the corresponding initialized values are Zhang San, Zhejiang Province, 0 respectively. Since the age is a null value, the default value 0 is initialized according to the field type int of age; and since the sex field is not in the dictionary table, it is not used as the value to be initialized and stored in the database.

[0090] Through the above steps, the data layer stores the data to be stored in the database, the dictionary layer stores the preset dictionary table, and arranges the order of the data to be stored in the database and assigns null values according to the regulations of the dictionary table, so that no matter whether the order of the data in the data layer is disordered or in positive order, and whether the field value is a null value, the data to be stored in the database has complete data fields, and the order conforms to the order of name, address, age in the dictionary table; the above steps ensure the consistency of the field order for batch storage in the database and solve the problems of null data values and default value assignment according to the type.

[0091] In some of the embodiments, generating the storage template according to the preset dictionary table includes:

[0092] Obtain an initial storage template and generate the storage template according to the preset dictionary table.

[0093] Among them, the initial storage template can be a database insertion template specified by the system or the program itself. For example, the template can be "insert into table Name(a,b,c,d)values(,,,)”.

[0094] Specifically, after obtaining the initial warehousing template, according to the dictionary table field order and name specified in the preset dictionary table, obtain the warehousing field values in the warehousing data and add them to the initial warehousing template to generate a warehousing template. For example, the warehousing template can be insert into table Name(name,address,age)values(Zhang San,Zhejiang Province,0).

[0095] Through the above embodiments, the sorted warehousing data is used to generate a warehousing template according to the initial warehousing template, making it more convenient and efficient to insert a large amount of data, solving the problem of low accuracy and efficiency in data warehousing in the related art, and achieving the purpose of batch inserting disordered data into the database.

[0096] It should be understood that although Figure 2 the steps in the flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, there is no strict order limit for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 at least a part of the steps in

[0097] include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0098] In this embodiment, another data warehousing method is also provided. Figure 3 is a schematic flowchart of another data warehousing method of this embodiment. As Figure 3 shown, the data warehousing method includes the following steps:

[0099] Step S302, obtain a set of disordered data. Three pieces of disordered data are obtained as follows:

[0101] {"name":"Zhang San","age":"","address":"Zhejiang Province"},

[0102] {"age":"12","name":"Li Si"},

[0103] {"address":"Zhejiang Province","name":"Wang Wu","age":"13","sex":"male"}, ​​

[0105] Step S304, obtain a dictionary table that guarantees the order. Not every field of the message data consumed by Kafka has a value, and the order of the fields is not necessarily sorted in a certain order. When the data consumed by Kafka is stored in batches into CK, it is necessary to ensure the insertion order of the fields and the absence of null values ​​in the fields, and to ensure that all fields are configured in batches according to the order of the storage SQL module specified by PreparedStatement. Therefore, a dictionary table is used here to cache all the field names and field types to be stored in advance and encapsulate them into a list<entity class>. The order of the cached dictionary table fields obtained in this embodiment is name, age, address; the corresponding data types are String, int, String respectively. At the same time, the storage SQL template is initialized according to the dictionary table. The order of the dictionary table is name, age, address, and the template is insert into tableName(name, age, address)values(?,?,?).

[0106] Step S306, obtain a data set with consistent order. The process of obtaining the data set with consistent order is step S305. Step S305 adopts a double loop, the outer layer is the data layer, which is encapsulated into a List <Map<String,Object> ; The inner layer is the dictionary layer, which is used to ensure that the corresponding fields are pulled from a piece of data in the outer layer, and to ensure that data assignment and type judgment are performed according to the field order of the inner entity class, thereby ensuring that the order of fields in batch storage is all consistent, and solving the problem of empty data and default value assignment according to type. In this embodiment, the inner dictionary table loops to retrieve and rearrange the order of each piece of data according to the order of name, age, and address in the dictionary table. First, the value of the key name, age, and address is obtained from the first piece of data and the data is initialized and assigned. The field with an empty value is assigned a default value according to the corresponding field type, and the missing field is directly assigned a default value according to the field type. Specifically, the processing results in this embodiment are as follows:

[0107] insert into tableName(name,age,address)values

[0108] ("Zhang San",0,"Zhejiang Province"),

[0109] ("Li Si",12,""),

[0110] ("Wang Wu", 13,"Zhejiang Province");

[0111] In the above embodiment, the first data age in the processed result is empty, and is assigned a value of 0 according to the type judgment; the second data does not have an address field, and is assigned a default value of an empty string according to the type judgment; the third data has all fields, but has an extra useless field sex, which is not in the fields specified in the dictionary table, so it is discarded. The above process implements the assignment of data and the assignment of the default value of the empty value default value field. At the same time, the field order is always taken according to name, age, address, ensuring the consistency of the order.

[0112] Step S308, using native jdbc precompilation to perform batch storage operation of fixed field sequence templates. In this embodiment, step S307 is used to batch store the processed PreparedStatement using the batch insert method.

[0113] It should be noted that the steps shown in the above process or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here. For example, step S302 and step S304 can be performed in the following order: Figure 3 The steps may be performed in the order of step S304 or step S302.

[0114] In this embodiment, a data storage device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. The terms "module", "unit", "subunit", etc. used below can be a combination of software and / or hardware that implements the predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0115] Figure 4 is a structural block diagram of the data storage device of this embodiment, such as Figure 4 As shown, the device includes: a sorting module 20, a packaging module 30 and a warehousing module 40;

[0116] The sorting module 20 is used to obtain the original data to be stored and a preset dictionary table, and to arrange the original data to be stored in sequence according to the preset dictionary table to obtain the first storage data;

[0117] The encapsulation module 30 is used to generate a storage template according to the preset dictionary table, and encapsulate the first storage data according to the storage template to obtain encapsulated data;

[0118] The storage module 40 is used to insert the packaged data into the database in batches to obtain the database storage result.

[0119] For the specific definition of the data storage device, please refer to the definition of the data storage method above, which will not be repeated here. Each module in the above-mentioned data storage device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0120] In this embodiment, a data storage system is also provided, which includes: a terminal device, a transmission device and a server device; wherein the terminal device is connected to the server device via the transmission device;

[0121] The server device is used to execute the data storage method in any one of the above embodiments;

[0122] The transmission device is used to transmit the database storage result from the server device to the terminal device;

[0123] The terminal device is used to display the database entry results.

[0124] Through the above embodiment, after obtaining the original data to be stored and the preset dictionary table, the server device obtains each field in the original data to be stored according to the field storage order specified in the preset dictionary table, thereby arranging the original data to be stored in order, and after obtaining the storage data, encapsulates it according to the storage template to obtain the encapsulated data, and inserts the encapsulated data into the database in batches to obtain the database storage result, thereby displaying the storage result on the terminal device. Compared with the related art, the data storage method provided in this embodiment rearranges the disordered data after consumption, ensures that the data value corresponds to the inserted field when the data is stored, and does not need to be manually corrected later. At the same time, a large number of multi-table data can be packaged and stored uniformly, without having to insert a single table into the database multiple times, solving the problem of low data storage accuracy and efficiency in the related art, and realizing batch insertion of disordered data into the database.

[0125] In this embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0126] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0127] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0128] S1, obtaining the original data to be stored and a preset dictionary table, and arranging the original data to be stored in sequence according to the preset dictionary table to obtain storage data.

[0129] S2, generating a storage template according to the preset dictionary table, and encapsulating the storage data according to the storage template to obtain encapsulated data.

[0130] S3, insert the encapsulated data into the database in batches to obtain the database entry result.

[0131] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0132] In addition, in combination with the data storage method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the data storage method. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the data storage methods in the above embodiments is implemented.

[0133] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data for warehousing. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data warehousing method is implemented.

[0134] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0136] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0137] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0138] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0139] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.< / sysdictionary> < / sysdictionary>

Claims

1. A data storage method, characterized in that: include: Obtain the original data to be stored and the preset dictionary table; According to the preset dictionary table, the original data to be stored are arranged in order to obtain the stored data, including: Preliminarily encapsulating the raw data to be stored to obtain preliminarily encapsulated raw data; and preliminarily encapsulating the preset dictionary table to obtain a preliminarily encapsulated preset dictionary table; Compare the fields in the preliminary packaged original data with the preliminary packaged preset dictionary table to obtain a comparison result; if the comparison result indicates that the preliminary packaged original data field is missing, assign an initial value to the missing preliminary packaged original data field according to the initial value of the field in the preset dictionary table to obtain the original data to be circulated and the current data strip to be circulated; Read the dictionary table field names of the preset dictionary table in sequence, obtain the current field value corresponding to the dictionary table field name in the current data strip to be circulated, and obtain the current storage data according to the current field value; Obtain the next data strip to be circulated of the original data to be circulated, read the field names of the dictionary table in sequence, obtain the next field value corresponding to the field name of the dictionary table in the next data strip to be circulated, and obtain the next storage data according to the next field value, until the reading of the data strip to be circulated in the data to be circulated is completed; Obtaining the storage data according to the current storage data and the next storage data; Generate a storage template according to the preset dictionary table, and encapsulate the storage data according to the storage template to obtain encapsulated data; the storage template refers to a unified format for batch data storage; The encapsulated data is inserted into the database in batches to obtain the database entry result.

2. The data storage method according to claim 1, characterized in that: The step of sequentially arranging the original data to be stored in the warehouse to obtain the stored data further includes: Obtaining data entries to be stored in the original data to be stored; According to the dictionary table field of the preset dictionary table, the to-be-stored field corresponding to the dictionary table field in the to-be-stored data strip is obtained; wherein the dictionary table field includes the dictionary table field name, the dictionary table field initial value and the dictionary table field value initial value; the to-be-stored field includes the to-be-stored field name and the to-be-stored field value; According to the dictionary table fields, the fields to be stored are extracted to obtain the stored data.

3. The data storage method according to claim 2, characterized in that: The step of extracting the to-be-stored fields according to the dictionary table fields to obtain the first stored data includes: Performing a field detection on the field to be stored to obtain a field detection result, if the field detection result indicates that the field to be stored fails to be obtained, then obtaining the corresponding initial value of the dictionary table field according to the name of the field to be stored, and obtaining the non-empty field storage data according to the initial value of the dictionary table field; wherein the non-empty field storage data includes the non-empty field storage data name and the non-empty field storage data value; The non-empty field storage data is extracted according to the dictionary table fields to obtain the storage data.

4. The data storage method according to claim 2, characterized in that: The step of extracting the to-be-stored fields according to the dictionary table fields to obtain the first stored data further includes: Performing a value detection on the field value to be stored in the field to be stored in order to obtain a value detection result; if the value detection result indicates that the field value to be stored in is a null value, obtaining the corresponding initial value of the dictionary table field value according to the name of the field to be stored in order to obtain the non-null value storage data according to the initial value of the dictionary table field value; wherein the non-null value storage data includes the name of the non-null value storage data and the value of the non-null value storage data; The non-null value storage data is extracted according to the dictionary table field to obtain the first storage data.

5. The data storage method according to any one of claims 1 to 4, characterized in that: The step of generating a storage template according to the preset dictionary table includes: An initial storage template is obtained, and the storage template is generated according to the preset dictionary table.

6. A data storage device, characterized in that: include: Sorting module, packaging module and warehousing module; The sorting module is used to obtain the original data to be stored and a preset dictionary table, and to arrange the original data to be stored in sequence according to the preset dictionary table to obtain the storage data, including: Preliminarily encapsulating the raw data to be stored to obtain preliminarily encapsulated raw data; and preliminarily encapsulating the preset dictionary table to obtain a preliminarily encapsulated preset dictionary table; Compare the fields in the preliminary packaged original data with the preliminary packaged preset dictionary table to obtain a comparison result; if the comparison result indicates that the preliminary packaged original data field is missing, assign an initial value to the missing preliminary packaged original data field according to the initial value of the field in the preset dictionary table to obtain the original data to be circulated and the current data strip to be circulated; Read the dictionary table field names of the preset dictionary table in sequence, obtain the current field value corresponding to the dictionary table field name in the current data strip to be circulated, and obtain the current storage data according to the current field value; Obtain the next data strip to be circulated of the original data to be circulated, read the field names of the dictionary table in sequence, obtain the next field value corresponding to the field name of the dictionary table in the next data strip to be circulated, and obtain the next storage data according to the next field value, until the reading of the data strip to be circulated in the data to be circulated is completed; Obtaining the storage data according to the current storage data and the next storage data; The encapsulation module is used to generate a storage template according to the preset dictionary table, and encapsulate the storage data according to the storage template to obtain encapsulated data; the storage template refers to a unified format for batch data storage; The storage module is used to insert the encapsulated data into the database in batches to obtain the database storage result.

7. A data warehousing system, characterized in that: include: Terminal equipment, transmission equipment and server equipment; wherein the terminal equipment is connected to the server equipment via the transmission equipment; The server device is used to execute the data storage method described in any one of claims 1 to 5; The transmission device is used to transmit the database storage result from the server device to the terminal device; The terminal device is used to display the database entry result.

8. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the data warehousing method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data storage method described in any one of claims 1 to 5 are implemented.

Images