Data storage method, system, device and readable storage medium

Abstract

The present invention relates to a data storage method, system, device and readable storage medium, wherein the method comprises acquiring storage data, the storage data comprising data format and data content; unifying the format of the data content according to the data format and a preset format unification method to form target format data; storing the target format data according to the data format and a preset storage rule. The present invention has the effect of unifying the format of the data content according to the data format and the format unification method and forming target format data, and storing the target format data according to the storage rule, and has the effect of unifying the data format and storing it, thereby improving the data retrieval efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data storage method, system, device and readable storage medium. Background Art

[0002] Currently, no matter software or hardware, different manufacturers and different models of equipment have separate data transmission formats. At present, data of various formats from different sources are stored in corresponding formats. However, different data require different search methods. For databases that need to store data of various formats, storing data in different formats will make it difficult to find the corresponding data when retrieving data. Summary of the invention

[0003] In order to improve the problem that data is difficult to find, the present application provides a data storage method, system, device and readable storage medium.

[0004] In a first aspect of the present application, a data storage method is provided. The method comprises:

[0005] Acquire stored data, wherein the stored data includes data format and data content;

[0006] According to the data format and a preset format unification method, the format of the data content is unified to form target format data;

[0007] According to the data format and preset storage rules, the target format data is stored.

[0008] It can be seen from the above technical solution that by acquiring the stored data, reading the data format in the stored data, using the corresponding format unification method according to different data formats, storing the target format data after format unification according to preset storage rules, and converting data in different formats into a unified format, it is convenient to retrieve data content in different formats, thereby improving the problem of data being difficult to find.

[0009] In a possible implementation, before the method of acquiring the stored data, the method includes:

[0010] Get model building instructions;

[0011] According to the model establishment instruction, a correspondence between batch processing fields, data content and data format is established to determine a data type model;

[0012] The data format includes a first format, a second format and a third format;

[0013] The data content is obtained, and according to the data content and the data type model, the data format and batch processing fields are determined to form storage data.

[0014] In a possible implementation, unifying the format of the data content according to the data format and a preset format unification method to form target format data includes:

[0015] When the data content includes a preset terminator, judging whether the data content needs batch processing according to the batch processing field;

[0016] If batch processing is required, the data content is split according to the preset separator and the data content to form multiple data to be stored;

[0017] Each piece of data to be stored is split into field names and field values ​​according to preset keywords to form target format data.

[0018] In a possible implementation, the method further includes:

[0019] When the data content does not include a preset end character, the data content is stored in an abnormal database.

[0020] In a possible implementation, the step of unifying the format of the data content according to the data format and a preset format unification method to form target format data further includes:

[0021] When the data format is a second format;

[0022] According to the batch processing field, determining whether the data content requires batch processing;

[0023] If so, the data content is divided into batch data content and non-batch data content according to the batch judgment rule and the data content, and the batch data content and the non-batch data content constitute the target format data.

[0024] In a possible implementation, the step of unifying the format of the data content according to the data format and a preset format unification method to form target format data further includes:

[0025] When the data format is a third format;

[0026] According to the regular splitting rule and the data content, the data content is split to form target format data.

[0027] In a possible implementation manner, before storing the target format data, the method further includes:

[0028] Verifying the target format data according to a preset verification rule;

[0029] If the verification fails, the target format data is stored in the exception database.

[0030] In a second aspect of the present application, a data storage system is provided. The system comprises:

[0031] A data acquisition module, used to acquire stored data, wherein the stored data includes data format and data content;

[0032] A data processing module, used to unify the format of the data content according to the data format and a preset format unification method to form target format data;

[0033] The data storage module is used to store the target format data according to the data format and preset storage rules.

[0034] In a third aspect of the present application, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the data storage method described above is implemented.

[0035] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method according to the first aspect of the present application is implemented.

[0036] In summary, the present application includes at least one of the following beneficial technical effects:

[0037] 1. By converting data in different formats into a unified format, it is convenient to retrieve data content in different formats, improving the problem of data being difficult to find.

[0038] 2. By setting verification rules, further verification is performed before the data is stored in the target database to improve the accuracy of data storage. Abnormal data is not stored in the target database, which can reduce the number of abnormal data in the target database and improve the query efficiency of users' query data. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of the data storage method provided by this application.

[0040] Figure 2 It is a structural diagram of the data storage system provided by this application.

[0041] Figure 3 It is a schematic diagram of the structure of the electronic device provided by this application.

[0042] In the figure, 200, data storage system; 201, model acquisition module; 202, data processing module; 203, data storage module; 301, CPU; 302, ROM; 303, RAM; 304, I / O interface; 305, input part; 306, output part; 307, storage part; 308, communication part; 309, drive; 310, removable medium. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0045] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.

[0046] An embodiment of the present application provides a data storage method, and the main process of the method is described as follows.

[0047] like Figure 1 As shown:

[0048] Step S101: Acquire storage data, where the storage data includes data format and data content.

[0049] Specifically, before storing data, it is necessary to first establish a data type model, which includes data content, a data format corresponding to the data content, and a batch processing field corresponding to the data content. In this embodiment, the field name corresponding to the data format is topic, and the batch processing field includes two attribute values ​​0 and 1. When the value of the batch processing field is 0, it indicates that the data content does not need to be batch processed. When the value of the batch processing field is 1, it indicates that the data content needs to be batch processed. The attribute values ​​corresponding to topic include a first format, a second format, and a third format, that is, the data format includes a first format, a second format, and a third format. In this embodiment, the first format is a standard format, the second format is a JSON format, and the third format is a regular format.

[0050] Obtain data content, which can be manually input or data monitored by a monitoring device, input the data content into a data type model, determine the data format and batch processing field of the data content, and mark the data content according to the corresponding data format and the corresponding batch processing field to form storage data. The storage data includes data content, data format and batch processing field.

[0051] Step S102: Determine and store target format data according to the data format, a preset format unification method and a preset storage rule.

[0052] Specifically, the data format of the stored data is obtained. When the data format is a standard format, the corresponding preset keyword, the corresponding preset keyword, the corresponding preset terminator, and the corresponding data parsing field need to be retrieved. The data parsing fields include none, first row header information, and first row public information. When the data parsing field is none, it means that the data content is stored normally; when the data parsing field is the first row header information, it means that the first piece of data is the header information and does not need to be stored in the target database, and the remaining data content is stored normally; when the data parsing field is the first row public information, it means that the first piece of data is public data, and the first piece of data needs to be added to all data except the first piece of data.

[0053] For data content in standard format, first determine whether the last character of the data content is the same as the preset terminator. If they are the same, it means that the data content is complete and can be stored in the target database and the last character in the data content is deleted; if they are not the same, it means that the data content is incomplete, and the stored data is entered and exited in the abnormal database, and the abnormal information is marked as incomplete data. For example, if the preset terminator is ".", determine whether the last character of the data content is ".". If so, proceed to the next step of storage judgment. If not, store the stored data in the abnormal database and mark it as abnormal.

[0054] When the data content is complete, the batch processing field corresponding to the data content is obtained to determine whether the data content needs to be batch processed. If necessary, the corresponding preset delimiter is obtained, and the above data content is split according to the preset delimiter. According to the preset keyword, each piece of data after splitting is further split into field names and field values. If not necessary, the data content is directly split according to the preset keyword into field names and field values. The above multiple field names and field values ​​constitute the target format data. It is determined whether the data type and the number of field names and field values ​​of the above field names and field values ​​are the same as the preset values. If they are not the same, it means that the data content is abnormal, and the data content is stored in the abnormal database and the abnormal information is marked as data verification abnormality. If they are the same, it means that the data content is normal and is stored in the target database. For example, the stored data is "'Data format': 'Standard format', 'Batch processing field': '1', 'Data content': 'Name 1, gender 1, age 1; Name 2, gender 2, age 2.'". First, determine whether the last character of the data content is the preset terminator. If the preset terminator is ".", the last character of the data content is the same as the preset terminator. Then determine whether batch processing is required. When the value of the batch processing field is 1, it means batch processing is required. The preset delimiter is ";". According to the preset delimiter, the data content is split into "Name 1, Gender 1, Age 1", "Name 2, Gender 2, Age 2", and the preset keywords are "Name", "Gender" and "Age". Then, "'Name':'Name 1'", "'Gender':'Gender 1'", "'Age':'Age 1'", "'Name':'Name 2'", "' Gender':'Gender2'",'Age':'Age2'", and then determine the data type of each data and the number of data in each data. For example, "'Name':'Name1'", "'Gender':'Gender1'", "'Age':'Age1'" is a data, the preset value is 3, and the number of data is also 3. At the same time, determine the data type of each field segment. For example, the data type of age should be numeric. In this data, it is "Age1" and the data type is character. Therefore, there is an exception in this data. It is stored in the exception database and the exception information is marked as data verification exception. If there is no exception in the data type, the data content is stored in the target database.

[0055] For data content in JSON format, obtain the batch processing field corresponding to the data content, and determine whether the data content needs to be batch processed according to the batch processing field. If necessary, determine whether there is a batch field in the above data content. If there is a batch field, parse the batch field into a JSON array, and then treat other data that are not batch fields as public data and add the public data to each array. If there is no batch field or batch processing is not required, directly convert the data content into a JSON array. The above JSON data is recorded as the target format data. The data content that has been processed above, that is, the JSON data, is judged for the data type and the number of data. If the data type and the number of data are inconsistent with the preset value, the above data content is stored in the exception database and the exception information is marked as a data verification exception. If they are the same, it means that the data content is normal and is stored in the target database.

[0056] For the data content in regular format, the default value of the batch processing field is 0, which means that batch processing is not required. According to the regular splitting rules and the above data content, the above data content is split, and the split data content constitutes the target format data. Determine whether the data type and number of data in the target format data are the same as the preset values. If they are not the same, it means that there is an abnormality in the data content. The data content is stored in the abnormal database and the abnormal information is marked as data verification abnormality. If they are the same, the split data content is stored in the target database. For example, the regular expression configuration is: (?<@timestamp>\S+)\s+(? <hostname>\S+)\s+(? <message>.*), first parse the regular expression, a single field is defined by "()" to define the single field parsing method, starting with "?", "<>" sets the field name, ">" defines the matching rules, and the parentheses define the parsing method between fields. It can be determined through the above regular expression that "\s" represents a blank character in the above regular expression. The above blank character may be a space, a tab or other blank. It can be known that the fields are separated by blank characters. A single data can be parsed into three fields: @timestamp, hostname, and message. The data is parsed into the above three fields, and then the data type and field number of the above three fields are determined to be consistent with the preset value. If the data type and data number are inconsistent with the preset value, the above data content is stored in the abnormal database and the abnormal information is marked as data verification abnormality. If they are the same, it means that the data content is normal and is stored in the target database.

[0057] The present application embodiment provides a data storage system 200, referring to Figure 2 , the data storage system 200 comprises:

[0058] The data acquisition module 201 is used to acquire the stored data, wherein the stored data includes the data format and the data content;

[0059] The data processing module 202 is used to unify the format of the data content according to the data format and a preset format unification method to form target format data;

[0060] The data storage module 203 is used to store the target format data according to the data format and preset storage rules.

[0061] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0062] The present application embodiment discloses an electronic device. Figure 3 The electronic device includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 307 to the random access memory (RAM) 303. In the RAM 303, various programs and data required for system operation are also stored. The CPU 301, the ROM 302 and the RAM 303 are connected to each other through a bus. The input / output (I / O) interface 304 is also connected to the bus.

[0063] The following components are connected to the I / O interface 304: an input section 305 including a keyboard, a mouse, etc.; an output section 306 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 307 including a hard disk, etc.; and a communication section 308 including a network interface card such as a LAN card, a modem, etc. The communication section 308 performs communication processing via a network such as the Internet. A drive 309 is also connected to the I / O interface 304 as needed. A removable medium 310, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 309 as needed, so that a computer program read therefrom is installed into the storage section 307 as needed.

[0064] In particular, according to an embodiment of the present application, the above reference flow chart Figure 1 The described process can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 308, and / or installed from a removable medium 310. When the computer program is executed by a central processing unit (CPU) 301, the above-mentioned functions defined in the device of the present application are executed.

[0065] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0066] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.< / message> < / hostname>

Claims

1. A data storage method, characterized in that: include: Acquire stored data, the stored data including data format and data content, and batch processing fields corresponding to the data content, the data format including a first format, a second format and a third format, the first format being a standard format; According to the data format and a preset format unification method, the format of the data content is unified to form target format data; According to the data format and the preset storage rules, the target format data is stored, and the data format in the stored data is obtained. When the data format is a standard format, it is necessary to retrieve the corresponding preset keyword, the corresponding preset keyword, the corresponding preset terminator, and the corresponding data parsing field; For the data content in the standard format, firstly, it is necessary to determine whether the last character of the data content is the same as the preset end character. If they are the same, it means that the data content is complete and can be stored in the target database and the last character in the data content is deleted; if they are not the same, it means that the data content is incomplete and the stored data is entered and exited from the abnormal database and the abnormal information is marked as incomplete data. When the data content is complete, a batch processing field corresponding to the data content is obtained to determine whether the data content needs to be batch processed. If necessary, a corresponding preset delimiter is obtained, and the data content is split according to the preset delimiter to form multiple data to be stored; Each piece of data to be stored is split into field names and field values ​​according to preset keywords to form target format data.

2. The data storage method according to claim 1, characterized in that: Prior to the method for obtaining stored data, the method includes: Get model building instructions; According to the model establishment instruction, a correspondence between batch processing fields, data content and data format is established to determine a data type model; The data content is obtained, and according to the data content and the data type model, the data format and batch processing fields are determined to form storage data.

3. The data storage method according to claim 2, characterized in that: The step of unifying the format of the data content according to the data format and the preset format unification method to form target format data further includes: When the data format is a second format; According to the batch processing field, determining whether the data content requires batch processing; If so, the data content is divided into batch data content and non-batch data content according to the batch judgment rule and the data content, and the batch data content and the non-batch data content constitute the target format data.

4. The data storage method according to claim 2, characterized in that: The step of unifying the format of the data content according to the data format and the preset format unification method to form target format data further includes: When the data format is a third format; According to the regular splitting rule and the data content, the data content is split to form target format data.

5. The data storage method according to claim 1, characterized in that: Before storing the target format data, the method further comprises: Verifying the target format data according to a preset verification rule; If the verification fails, the target format data is stored in the exception database.

6. A data storage system, characterized in that: include: A data acquisition module (201) is used to acquire stored data, wherein the stored data includes a data format and data content, wherein the data format includes a first format, a second format and a third format, wherein the first format is a standard format; A data processing module (202) is used to unify the format of the data content according to the data format and a preset format unification method to form target format data; A data storage module (203) is used to store the target format data according to the data format and preset storage rules, obtain the data format in the stored data, and when the data format is a standard format, it is necessary to retrieve the corresponding preset keyword, the corresponding preset keyword, the corresponding preset terminator and the corresponding data parsing field; For the data content in the standard format, firstly, it is necessary to determine whether the last character of the data content is the same as the preset end character. If they are the same, it means that the data content is complete and can be stored in the target database and the last character in the data content is deleted; if they are not the same, it means that the data content is incomplete and the stored data is entered and exited from the abnormal database and the abnormal information is marked as incomplete data. When the data content is complete, a batch processing field corresponding to the data content is obtained to determine whether the data content needs to be batch processed. If necessary, a corresponding preset delimiter is obtained, and the data content is split according to the preset delimiter to form multiple data to be stored; Each piece of data to be stored is split into field names and field values ​​according to preset keywords to form target format data.

7. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes any one of the methods of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 5.

Images