Data analysis method, device, equipment and storage medium

Abstract

The present invention discloses a data parsing method, device, equipment and storage medium. The configuration information of a data file is obtained; a parsing script template is determined according to the configuration information; a target table definition script is obtained according to the parsing script template and the configuration information; a loading script is determined according to the file type of the data file; and the data file is parsed according to the target table definition script and the loading script. The data parsing method provided by the embodiment of the present invention can improve the efficiency of data parsing and reduce the cost of data parsing.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a data analysis method, device, equipment and storage medium. Background Art

[0002] Nowadays, with the development of information society, various industries have higher and higher requirements for data file transmission. While meeting the requirements of efficient transmission, data accuracy is also particularly important. For example, bank reconciliation files, teller operation information, user bills, etc. all require efficient and accurate data transmission. Although such files can automatically interact with relevant application systems within a precise time window, for the relevant data receivers, in order to realize the parsing of received data, developers need to go through data field type conversion, data quality logic verification, data format conversion, Structured Query Language (SQL) script development and other work. The commonly used technical solutions are as follows: the data provider provides data processing logic, and the developer needs to parse the file according to the file processing logic, develop the parsing script in combination with the current database version, and obtain the corresponding field information. At present, the parsing script is mainly written manually, with high development cost and prone to parsing errors, making the data parsing process inefficient and costly. Summary of the invention

[0003] The embodiments of the present invention provide a data analysis method, device, equipment and storage medium, which can improve the efficiency of data analysis and reduce the cost of data analysis.

[0004] In a first aspect, an embodiment of the present invention provides a data parsing method, comprising:

[0005] Get the configuration information of the data file;

[0006] Determine a parsing script template according to the configuration information;

[0007] Acquire a target table definition script according to the parsing script template and the configuration information;

[0008] Determine a loading script according to the file type of the data file;

[0009] The data file is parsed according to the target table definition script and the loading script.

[0010] In a second aspect, an embodiment of the present invention further provides a data parsing device, comprising:

[0011] A configuration information acquisition module is used to obtain configuration information of a data file;

[0012] A parsing script template acquisition module, used to determine a parsing script template according to the configuration information;

[0013] A root target table definition script acquisition module, used to acquire a target table definition script according to the parsed script template and the configuration information;

[0014] A loading script determination module, used to determine a loading script according to the file type of the data file;

[0015] The data file parsing module is used to parse the data file according to the target table definition script and the loading script.

[0016] In a third aspect, an embodiment of the present invention further provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data parsing method as described in the embodiment of the present invention is implemented.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processing device, implements the data parsing method as described in the embodiment of the present invention.

[0018] The embodiment of the present invention discloses a data parsing method, device, equipment and storage medium. The configuration information of the data file is obtained; the parsing script template is determined according to the configuration information; the target table definition script is obtained according to the parsing script template and the configuration information; the loading script is determined according to the file type of the data file; the data file is parsed according to the target table definition script and the loading script. The data parsing method provided by the embodiment of the present invention can improve the efficiency of data parsing and reduce the cost of data parsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of a data parsing method in Embodiment 1 of the present invention;

[0020] Figure 2 is a structural schematic diagram of a data analysis device in Embodiment 2 of the present invention;

[0021] Figure 3 It is a structural diagram of a computer device in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0023] The prior art has the following disadvantages:

[0024] 1. Since the data provided by the data provider needs to be used by multiple related systems, and the database versions used by each system vary (MYSQL, SYBASE, SQL SERVER, etc.), the data receiver needs to develop corresponding scripts according to its own database version when parsing the data. The development of such scripts is highly repetitive, and manual writing is bound to have errors, which greatly increases the cost of subsequent testing.

[0025] 2. The data provider usually provides a data file, such as a text file or an EXCEL file. Due to differences in database versions, data quality and other factors, such problems are usually difficult to find during testing. In order to avoid such problems, the data recipient will process specific data types, which increases development costs.

[0026] In view of this, the present invention proposes an automated script generation tool suitable for multiple scenarios. By filling in specified configuration information, the tool can automatically generate parsing scripts for multiple databases, greatly reducing development costs and improving the accuracy of file parsing.

[0027] Embodiment 1

[0028] Figure 1 This is a flow chart of a data parsing method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case where received data is parsed. The method can be executed by a data parsing device, such as Figure 1 As shown, the method specifically comprises the following steps:

[0029] Step 110, obtaining configuration information of the data file.

[0030] The configuration information may include data format, database type, target table name, field information, data type and validation rule. The validation rule may be a regular expression. The configuration information may be recorded in the form of a web form or a table (such as an EXCEL table). The configuration information may be sent by the data sender.

[0031] Step 120: Determine a parsing script template according to the configuration information.

[0032] The parsing script template may be pre-edited and generated by a developer. In this embodiment, a correspondence between a database type and a parsing script template may be pre-established, and then the parsing script template may be determined from the correspondence according to the database type in the configuration information.

[0033] Step 130: Obtain a target table definition script according to the parsed script template and configuration information.

[0034] The target table definition script may be a script composed of table creation statements, which is used to create a data table in a database. Specifically, the target table definition script is generated according to the target table naming, field definition, and data type in the parsing script module and configuration information.

[0035] Step 140: Determine the loading script according to the file type of the data file.

[0036] The file type may include a text file type and a table type (such as EXCEL).

[0037] Optionally, the method of determining the loading script according to the file type of the data file may be: if the file type of the data file is a text file type, obtaining symbol information in the data file; and generating the loading script according to the symbol information and configuration information.

[0038] The symbol information includes separator information and line break information. Specifically, if the data file type is a text file type, the separator and line break information in the data file is obtained, and then the relevant loading script is automatically generated in combination with the field information of the configuration file.

[0039] Optionally, the method of determining the loading script according to the file type of the data file may be: if the file type of the data file is a table type, the data file is read line by line to generate the loading script.

[0040] Specifically, if the data file is an EXCEL type file, the data file is read line by line in combination with the field information of the configuration information, and a corresponding loading script is automatically generated.

[0041] Step 150: Parse the data file according to the target table definition script and the loading script.

[0042] In this embodiment, the process of parsing the data file according to the target table definition script and the loading script may be: establishing a target table in the database according to the target table definition script; and loading the data file into the target table according to the loading script.

[0043] Optionally, after the data file is loaded into the target table according to the loading script, the following step is also included: verifying the data loaded into the target table according to the verification rules.

[0044] If the data file type is a text file type, after the data file is loaded, a regular check is performed on the data file according to the field sequence to verify the integrity of the data.

[0045] If the file type of the data file is a table type, after the data file is loaded, when reading the data line by line, it is necessary to perform special processing on the fields in text format, match each column of data according to the regular expression, and verify the data at the same time. If the data is empty or there are other abnormal situations, the abnormal information will be displayed and rollback and restoration will be performed.

[0046] Optionally, after parsing the data file according to the target table definition script and the loading script, the method further includes: storing the target table definition script and the loading script.

[0047] The technical solution of this embodiment is to obtain the configuration information of the data file; determine the parsing script template according to the configuration information; obtain the target table definition script according to the parsing script template and the configuration information; determine the loading script according to the file type of the data file; and parse the data file according to the target table definition script and the loading script. The data parsing method provided by the embodiment of the present invention can improve the efficiency of data parsing and reduce the cost of data parsing.

[0048] Embodiment 2

[0049] Figure 2 Schematic diagram of the structure of a data analysis device provided by the second embodiment of the present invention. Figure 2 As shown, the device comprises:

[0050] Configuration information acquisition module 210, used to acquire configuration information of a data file;

[0051] A parsing script template obtaining module 220 is used to determine a parsing script template according to configuration information;

[0052] A root target table definition script acquisition module is used to acquire a target table definition script according to the parsed script template and configuration information;

[0053] A loading script determining module 230, used to determine a loading script according to a file type of a data file;

[0054] The data file parsing module 240 is used to parse the data file according to the target table definition script and the loading script.

[0055] Optionally, the configuration information includes data format, database type, target table name, field information, data type, and validation rules.

[0056] Optionally, the method further includes: a corresponding relationship establishing module, which is used to:

[0057] Pre-establish the correspondence between database types and parsing script templates;

[0058] Optionally, the script template parsing module 220 is further used to:

[0059] The parsing script template is determined from the corresponding relationship according to the database type in the configuration information.

[0060] Optionally, the loading script determination module 230 is further used to:

[0061] If the file type of the data file is a text file type, then obtaining symbol information in the data file; wherein the symbol information includes separator information and line break information;

[0062] Generates a load script based on symbol information and configuration information.

[0063] Optionally, the loading script determination module 230 is further used to:

[0064] If the file type of the data file is table type, the data file is read line by line to generate a load script.

[0065] Optionally, the data file parsing module 240 is further used to:

[0066] Create a target table in the database according to the target table definition script;

[0067] Load the data file into the target table according to the load script.

[0068] Optionally, it also includes: a verification module, used to:

[0069] Validate the data loaded into the target table according to the validation rules.

[0070] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not described in detail in this embodiment, please refer to the methods provided by all the above embodiments of the present invention.

[0071] Embodiment 3

[0072] Figure 3 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 3 A block diagram of a computer device 312 suitable for use in implementing embodiments of the present invention is shown. Figure 3 The computer device 312 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. The device 312 is a typical computing device for data analysis functions.

[0073] like Figure 3 As shown, the computer device 312 is in the form of a general-purpose computing device. The components of the computer device 312 may include, but are not limited to: one or more processors 316, a storage device 328, and a bus 318 connecting different system components (including the storage device 328 and the processor 316).

[0074] Bus 318 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0075] The computer device 312 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 312, including volatile and non-volatile media, removable and non-removable media.

[0076] The storage device 328 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 330 and / or cache memory 332. The computer device 312 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 334 may be used to read and write non-removable, non-volatile magnetic media ( Figure 3 not shown, usually called a "hard drive"). Although Figure 3 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a read-only optical disk (Compact Disc-Read Only Memory, CD-ROM), a digital video disk (Digital Video Disc-Read Only Memory, DVD-ROM) or other optical media) may be provided. In these cases, each drive may be connected to the bus 318 via one or more data medium interfaces. The storage device 328 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0077] A program 336 having a set (at least one) of program modules 326 may be stored, for example, in a storage device 328, such program modules 326 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 326 generally perform the functions and / or methods of the embodiments described herein.

[0078] The computer device 312 may also communicate with one or more external devices 314 (e.g., keyboard, pointing device, camera, display 324, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 312, and / or communicate with any device that enables the computer device 312 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 322. In addition, the computer device 312 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 320. As shown, the network adapter 320 communicates with other modules of the computer device 312 through a bus 318. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computer device 312, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems.

[0079] The processor 316 executes various functional applications and data processing by running the programs stored in the storage device 328, such as implementing the data parsing method provided in the above embodiment of the present invention.

[0080] Embodiment 4

[0081] The embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processing device, a data parsing method as in the embodiment of the present invention is implemented. The above-mentioned computer-readable medium of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0082] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0083] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0084] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains configuration information of the data file; determines a parsing script template according to the configuration information; obtains a target table definition script according to the parsing script template and the configuration information; determines a loading script according to the file type of the data file; and parses the data file according to the target table definition script and the loading script.

[0085] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0086] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0087] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0088] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0089] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0090] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data analysis method, characterized in that: include: Obtain configuration information of the data file; the configuration information includes data format, database type, target table naming, field information, data type and verification rules; Determine a parsing script template according to the configuration information; Acquire a target table definition script according to the parsing script template and the configuration information; Determine a loading script according to the file type of the data file; Parsing the data file according to the target table definition script and the loading script; The obtaining of the target table definition script according to the parsing script template and the configuration information includes: generating the target table definition script according to the target table naming, field definition, and data type in the parsing script template and the configuration information; The step of determining the loading script according to the file type of the data file comprises: If the file type of the data file is a text file type, then acquiring symbol information in the data file; wherein the symbol information includes separator information and line break information; Generate a loading script according to the symbol information and the configuration information; If the file type of the data file is a table type, the data file is read line by line to generate a loading script; The parsing of the data file according to the target table definition script and the loading script includes: Establish a target table in a database according to the target table definition script; The data file is loaded into the target table according to the loading script.

2. The method according to claim 1, characterized in that Also includes: Pre-establish the correspondence between database types and parsing script templates; Determining a parsing script template according to the configuration information includes: A parsing script template is determined from the corresponding relationship according to the database type in the configuration information.

3. The method according to claim 1, characterized in that After the data file is loaded into the target table according to the loading script, the method further includes: The data loaded into the target table is verified according to the verification rules.

4. A data analysis device, characterized in that: include: A configuration information acquisition module is used to acquire configuration information of a data file; the configuration information includes data format, database type, target table naming, field information, data type and verification rules; A parsing script template acquisition module, used to determine a parsing script template according to the configuration information; A target table definition script acquisition module, used to acquire a target table definition script according to the parsing script template and the configuration information; A loading script determination module, used to determine a loading script according to the file type of the data file; A data file parsing module, used for parsing the data file according to the target table definition script and the loading script; The target table definition script acquisition module is further used to generate a target table definition script according to the target table naming, field definition, and data type in the parsed script template and configuration information; The loading script determination module is also used for: If the file type of the data file is a text file type, then acquiring symbol information in the data file; wherein the symbol information includes separator information and line break information; Generate a loading script according to the symbol information and the configuration information; If the file type of the data file is a table type, the data file is read line by line to generate a loading script; The data file parsing module is further used to establish a target table in the database according to the target table definition script; and load the data file into the target table according to the loading script.

5. A computer device, characterized in that: The device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data parsing method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the data analysis method as described in any one of claims 1 to 3 is implemented.

Images