Data storage method, device, electronic device and storage medium

Abstract

The present invention discloses a data storage method, device, electronic device and storage medium. The method comprises: reading storage data in at least two databases according to a driver configuration file; storing each of the storage data according to a temporary database. The embodiment of the present invention can realize the unified management of different types of databases, improve the security of data storage, facilitate the use of storage data, and improve the management efficiency of storage data.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of data storage, and in particular, to a data storage method, device, electronic device, and storage medium. Background Art

[0002] With the continuous advancement of Internet infrastructure and the further improvement of top-level Internet interconnection, informatization has become an important development factor for current enterprises. At present, the Internet industry has made full use of the advantages of new-generation information technologies and platform services such as cloud computing, big data, and artificial intelligence, and has achieved rapid development. However, in the process of development, due to the influence of the technological background of the times, different business scenarios of enterprises are often distributed on different platforms and systems, and the databases corresponding to each business scenario are often independent of each other. Different database types and service providers are different, which makes it impossible for enterprises to uniformly manage data in different databases, resulting in poor data availability and security. Summary of the invention

[0003] The present invention provides a data storage method, device, electronic device and storage medium to achieve unified management of different types of databases, which can improve the security of data storage, facilitate the use of stored data, and improve the management efficiency of stored data.

[0004] In a first aspect, an embodiment of the present invention provides a data storage method, wherein the method comprises:

[0005] Read the stored data in at least two databases according to the driver configuration file;

[0006] The stored data are stored according to the temporary database.

[0007] Further, the stored data in at least two databases is read according to the driver configuration file, including:

[0008] Extracting the database driver corresponding to each of the databases in the driver configuration file;

[0009] The database driver is called to read the stored data in the corresponding database.

[0010] Further, storing each of the stored data in a temporary database includes:

[0011] Processing each of the stored data according to the data storage format of the temporary database;

[0012] The stored data is stored in the temporary database.

[0013] Furthermore, the temporary database includes: a MySQL database.

[0014] Furthermore, the data storage method provided by the embodiment of the present invention further includes: configuring connection parameters of the temporary database according to user parameters, wherein the connection parameters at least include a data source address and login information.

[0015] Further, processing each of the stored data according to the data storage format of the temporary database includes:

[0016] Convert each of the stored data into a comma separated value file;

[0017] Extracting target content from the comma-separated value file according to field information in the data storage format;

[0018] The stored data is spliced ​​into the data storage format according to the field information corresponding to each of the target contents.

[0019] In a second aspect, an embodiment of the present invention further provides a data storage device, wherein the device comprises:

[0020] A data reading module, used to read the stored data in at least two databases according to the driver configuration file;

[0021] The data temporary storage module is used to store the storage data according to the temporary storage database.

[0022] In a third aspect, an embodiment of the present invention further provides an electronic device, wherein the electronic device includes:

[0023] one or more processors;

[0024] a memory for storing one or more programs,

[0025] When the one or more programs are executed by the one or more processors, the one or more processors implement any method as described in the embodiments of the present invention.

[0026] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein a computer program is stored thereon, and when the computer program is executed by a processor, the method as described in any one of the embodiments of the present invention is implemented.

[0027] The embodiment of the present invention reads the storage data in different databases in the driver configuration file and stores each storage data in a temporary database. The embodiment of the present invention can realize the unified management of different types of databases, improve the security of data storage, facilitate the use of storage data, and improve the management efficiency of storage data. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of a data storage method provided by Embodiment 1 of the present invention;

[0029] Figure 2 is a flow chart of another data storage method provided by Embodiment 2 of the present invention;

[0030] Figure 3 This is a schematic diagram of the structure of a data storage device provided by Embodiment 3 of the present invention;

[0031] Figure 4 It is a structural schematic diagram of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention are shown in the accompanying drawings, rather than all structures. In addition, the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0033] Embodiment 1

[0034] Figure 1 is a flow chart of a data storage method provided in the first embodiment of the present invention. This embodiment is applicable to the case of unified management of storage data of different types of databases. The method can be executed by a data storage device, which can be implemented in hardware and / or software and can generally be integrated in a server or server cluster. Figure 1 The method provided in the embodiment of the present invention specifically includes the following steps:

[0035] Step 110: Read the stored data in at least two databases according to the driver configuration file.

[0036] The driver configuration file may be a configuration file driven by one or more databases, and the database driver may be a software program code for connecting to a database, which may be composed of a programming language and provide a unified access method for the database. A database may be a warehouse that organizes, stores and manages data according to a data structure and may be stored for a long time inside a computer.

[0037] In an embodiment of the present invention, database drivers of one or more databases may be read in a preconfigured driver configuration file, and a connection may be made to a corresponding database through the read database driver, and storage data may be read in the database.

[0038] Step 120: storing each storage data in a temporary database.

[0039] The temporary database may be a database for uniformly managing the storage data in different types of databases, and the type of the temporary database may be the same as or different from the type of the original database storing the data. It is understandable that the type of temporary database may be selected according to the needs of the enterprise business to increase the management efficiency of different storage data.

[0040] Specifically, the storage data read from each database may be stored in a temporary database, and unified management of the data in each database may be achieved by processing the storage data in the supporting database.

[0041] The embodiment of the present invention reads the storage data in different databases in the driver configuration file and stores each storage data in a temporary database. The embodiment of the present invention can realize the unified management of different types of databases, improve the security of data storage, facilitate the use of storage data, and improve the management efficiency of storage data.

[0042] Embodiment 2

[0043] Figure 2 is a flowchart of another data storage method provided by Embodiment 2 of the present invention. This embodiment of the present invention is a specific implementation of the above-mentioned embodiment of the invention. Figure 2 The method provided in the embodiment of the present invention specifically includes the following steps:

[0044] Step 210: Configure connection parameters of the temporary database according to user parameters, wherein the connection parameters at least include a data source address and login information.

[0045] The connection parameters may be information for a user to connect to a temporary database, and may include a data source address and login information, etc. The data source address may be a network address used to connect to the temporary database, and the login information may be verification information used by a user to access the temporary database.

[0046] In an embodiment of the present invention, a user may input the data source address and login information of the temporary database into the server by configuring connection parameters, and the server may connect to the temporary database through the connection parameters and the database driver corresponding to the temporary database to implement the configuration of the temporary database. The user parameters may be input by the user or determined by the server according to the user information.

[0047] Further, based on the above-mentioned embodiment of the invention, the temporary storage database includes: a MySQL database.

[0048] Step 220: extract the database driver corresponding to each database in the driver configuration file.

[0049] In the embodiment of the present invention, the database drivers corresponding to the respective databases can be extracted in sequence in the driver configuration file. Different types of databases in the driver configuration file can have their own corresponding database drivers, for example, a database driver corresponding to a MySQL database, a database driver corresponding to a SQLSERVER database, a database driver corresponding to an ORACLE database, and a database driver corresponding to a HIVE database, etc. Step 230: Call the database driver to read the stored data in the corresponding database.

[0050] Specifically, for each database, the corresponding database driver can be called to connect to each database and read the stored data in each database. It can be understood that a thread can be created separately to execute each database driver program to achieve parallel reading of stored data in different databases.

[0051] Step 240: Process each stored data according to the data storage format of the temporary database.

[0052] The data storage format may be a structure for organizing and storing data in a database.

[0053] In an embodiment of the present invention, the original database of part or all of the stored data may be an unstructured database. Before the stored data is transferred to the temporary database, the read stored data may be formatted so that the data format of the converted stored data is the same as the data storage format of the temporary database.

[0054] Step 250: Store the stored data in a temporary database.

[0055] Specifically, the storage data that can be read is stored in the temporary storage database, and the unified management of the data in each database can be achieved by processing the storage data in the supporting database.

[0056] In an embodiment of the present invention, by using user parameters to configure the connection parameters of the temporary database, extracting the database driver in the driver configuration file, and calling the database driver to read the storage data stored in the database, each storage data is processed according to the data storage format of the temporary database, and each storage data is stored in the temporary database. This can achieve unified management of different types of databases, improve the security of data storage, facilitate the use of storage data, and improve the management efficiency of storage data.

[0057] Further, based on the above-mentioned embodiment of the invention, each of the stored data is processed according to the data storage format of the temporary database, including:

[0058] Convert each of the stored data into a comma-separated value file; extract the target content in the comma-separated value file according to the field information of the data storage format; and splice the stored data into the data storage format according to the field information corresponding to each of the target contents.

[0059] Among them, a comma-separated values ​​(CSV) file can be a plain text file that stores table data. The file is a character sequence consisting of any number of records separated by some line break character; each record consists of fields, and the separators between fields are other characters or strings, the most common of which are commas or tabs. Field information can be column information or row information at a fixed position in the data storage format. The data storage format can be composed of one or more field information, and different field information can be different positions in the data storage format.

[0060] In an embodiment of the present invention, the read storage data can be first stored in a comma-separated value file, different storage data can be separated by a delimiter, and the corresponding target content can be matched in the comma-separated value file according to different field information of the data storage format. For example, the target content can have the same identification number as the field information or the semantics of the target content can be similar to the field information. For each storage data, the extracted target content can be spliced ​​into one or more data storage formats according to their corresponding field information, thereby realizing the format conversion of the storage data.

[0061] Embodiment 3

[0062] Figure 3 301 is a schematic diagram of a data storage device provided in Embodiment 3 of the present invention, which can execute the data storage method provided in any embodiment of the present invention and has the corresponding functional modules and beneficial effects of the execution method. The device can be implemented by software and / or hardware, and specifically includes: a data reading module 301 and a data temporary storage module 302.

[0063] The data reading module 301 is used to read the stored data in at least two databases according to the driver configuration file.

[0064] The data temporary storage module 302 is used to store the storage data in a temporary storage database.

[0065] In the embodiment of the present invention, the storage data in different databases in the driver configuration file is read by the data reading module, and the data temporary storage module stores each storage data in the temporary storage database. The embodiment of the present invention can realize the unified management of different types of databases, improve the security of data storage, facilitate the use of storage data, and improve the management efficiency of storage data.

[0066] Further, based on the above-mentioned embodiment of the invention, the data reading module 301 includes:

[0067] A driver determination unit, used for extracting a database driver corresponding to each of the databases in the driver configuration file;

[0068] The read execution unit is used to call the database driver to read the storage data corresponding to the database.

[0069] Further, based on the above-mentioned embodiment of the invention, the data temporary storage module 302 includes:

[0070] A format conversion unit, used for processing each of the stored data according to the data storage format of the temporary database;

[0071] The data storage unit is used to store the storage data in the temporary storage database.

[0072] Further, based on the above-mentioned embodiment of the invention, the temporary storage database includes: a MySQL database.

[0073] Furthermore, based on the above-mentioned embodiment of the invention, the device also includes: a temporary storage configuration module, which is used to configure the connection parameters of the temporary storage database according to user parameters, wherein the connection parameters at least include a data source address and login information.

[0074] Further, based on the above-mentioned embodiments of the invention, the format conversion unit is specifically used to: convert each of the stored data into a comma separated value file; extract the target content in the comma separated value file according to the field information of the data storage format; and splice the stored data into the data storage format according to the field information corresponding to each of the target contents.

[0075] Embodiment 4

[0076] Figure 4 is a schematic diagram of the structure of an electronic device provided by a fourth embodiment of the present invention, Figure 4 A block diagram of a computer device 312 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The computer device 312 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. The device 312 is a typical computing device that implements the data storage method.

[0077] like Figure 4 As shown, the computer device 312 is in the form of a general-purpose computing device. The components of the computer device 312 may include, but are not limited to: one or more processors 316, a storage device 328, and a bus 318 connecting different system components (including the storage device 328 and the processor 316).

[0078] Bus 318 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0079] The computer device 312 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 312, including volatile and non-volatile media, removable and non-removable media.

[0080] The storage device 328 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 330 and / or cache memory 332. The computer device 312 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 334 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a read-only optical disk (Compact Disc-Read Only Memory, CD-ROM), a digital video disk (Digital Video Disc-Read Only Memory, DVD-ROM) or other optical media) may be provided. In these cases, each drive may be connected to the bus 318 via one or more data medium interfaces. The storage device 328 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0081] A program 336 having a set (at least one) of program modules 326 may be stored, for example, in a storage device 328, such program modules 326 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 326 generally perform the functions and / or methods of the embodiments described herein.

[0082] The computer device 312 may also communicate with one or more external devices 314 (e.g., keyboard, pointing device, camera, display 324, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 312, and / or communicate with any device that enables the computer device 312 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 322. In addition, the computer device 312 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 320. As shown, the network adapter 320 communicates with other modules of the computer device 312 through a bus 318. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computer device 312, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems.

[0083] The processor 316 executes various functional applications and data processing by running the programs stored in the storage device 328, such as implementing the data storage method provided in the above-mentioned embodiment of the present invention.

[0084] Embodiment 5

[0085] The embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processing device, a data storage method as in the embodiment of the present invention is implemented. The above-mentioned computer-readable medium of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0086] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0087] The computer-readable medium may be included in the computer device, or may exist independently without being incorporated into the computer device.

[0088] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the computer device, the computer device: reads the stored data in at least two databases according to the driver configuration file; stores each of the stored data according to the temporary database. The computer program code for performing the operation of the present disclosure can be written in one or more programming languages ​​or a combination thereof, and the programming language includes but is not limited to object-oriented programming languages ​​such as Java, Smalltalk, C++, and also includes conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0089] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0090] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0091] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0092] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0093] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data storage method, characterized in that: The method comprises: Read the stored data in at least two databases according to the driver configuration file; Storing each of the stored data according to the temporary database; Configuring connection parameters of the temporary database according to user parameters, wherein the connection parameters at least include a data source address and login information; The storing of each of the stored data according to the temporary database comprises: Processing each of the stored data according to the data storage format of the temporary database; storing the stored data in the temporary database; The temporary database includes: MySQL database; The processing of each stored data according to the data storage format of the temporary database includes: Convert each of the stored data into a comma separated value file; Extracting target content from the comma-separated value file according to field information in the data storage format; The stored data is spliced ​​into the data storage format according to the field information corresponding to each of the target contents.

2. The method according to claim 1, characterized in that: The step of reading the stored data in at least two databases according to the driver configuration file includes: Extracting the database driver corresponding to each of the databases in the driver configuration file; The database driver is called to read the stored data in the corresponding database.

3. A data storage device, characterized in that: The device comprises: A data reading module, used to read the stored data in at least two databases according to the driver configuration file; A data temporary storage module, used for storing each of the stored data according to a temporary storage database; The device further comprises: a temporary storage configuration module, configured to configure connection parameters of the temporary storage database according to user parameters, wherein the connection parameters at least include a data source address and login information; The data temporary storage module comprises: A format conversion unit for processing each of the stored data in accordance with the data storage format of the temporary database; a data storage unit for storing the stored data in the temporary database; The temporary database includes: MySQL database; The format conversion unit is specifically used to: convert each of the stored data into a comma separated value file; extract the target content in the comma separated value file according to the field information of the data storage format; and splice the stored data into the data storage format according to the field information corresponding to each of the target contents.

4. The device according to claim 3, characterized in that: The data reading module comprises: A driver determination unit, used for extracting a database driver corresponding to each of the databases in the driver configuration file; The read execution unit is used to call the database driver to read the storage data corresponding to the database.

5. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in any one of claims 1 to 2.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images